Difference between Bayesian and frequentist analysis.

Over lunch today I had an in-depth discussion about the difference between the Bayesian and frequentist approaches to probability. We both seemed to agree that most of the disagreements between the two are semantic in nature. However, my friend said that frequentists refuse to accept prior knowledge that makes common sense and will only look at data allowed to be considered. We came up with the following example.

Suppose a US-minted coin is selected at random from a random bank. It is then given a fair flip (let's say dropped from a tall building) 100 times. The results come back 65 tails and 35 heads. The two sides are then asked what is the probability the next flip would be tails.

Would it be true that each would answer roughly as follows?...

Bayesian: Very close to 50%. The experiment was basically a waste of time. The chances of picking a biased coin randomly from a random bank would be microscopic. Like rolling 18 yo's in a row. Common sense tells us a random coin is basically fair and the 65/35 split was just random variation.

Frequentist: About 35%. Given the data we have, 35% is the best estimate we can give. Factoring in other information is strictly not kosher.

I said that while a Frequentist might say that in principle, he wouldn't be willing to lay any odds on heads. In other words, he secretly knows he's wrong and is a closet Bayesian.

Is this a fair comparison of the two? Are there any frequentists on the forum to defend their side. I will admit I'm biased as a Bayesian and I find frequentists very annoying. I'd love to do some kind of challenge between the two.

The question for the poll is which are you?

A Bayesian walks into a bar. What's he say?

Quote: billryan

A Bayesian walks into a bar. What's he say?

Ouch?

I voted "Bad example." I hear pitchforks being sharpened as torches soak in kerosene. This thread may lose focus and devolve into another discussion of the immutable 50/50 probability of a fair coin flip.

Full disclosure: this is the first I've heard about the two schools of thought. Having said that, I think a better point of discussion would be, "given the evidence, what is the chance the coin is biased?"

Of course, there is a statistical probability as an answer. I think that the Bayesian viewpoint would be that slight probability should be further discounted by knowledge of the stringent quality standards used for the manufacture of US coins. The frequentist camp would not allow that, because to do so might prevent identifying a "black swan" occurrence.

Sort of odd frequentist is not also capitalized, I don't think I like the evident decision that the one comes from someone's name so only Bayesian calls for it. However, I confess to wanting to over-capitalize. I just noticed my spell-checker recognizes Bayesian but not frequentist btw.

Seems to me scientists are often bound by the rule to not "factor in other information". However, probability analysis of data is used to 'score' hypotheses.

image from http://noahpinionblog.blogspot.com/2013/01/bayesian-vs-frequentist-is-there-any.html

I'm guessing this one would be type 2. Delete or split if needed. I think with this, any time a right winger creates a crime, they say terrorist.

Quote: odiousgambit

Sort of odd frequentist is not also capitalized, I don't think I like the evident decision that the one comes from someone's name so only Bayesian calls for it.

Yes, because Bayes was a mathematician and the word refers to him. My spell checker doesn't even respect frequentist was a word.

Quote: onenickelmiracle

I'm guessing this one would be type 2. Delete or split if needed. I think with this, any time a right winger creates a crime, they say terrorist.

Warning given for hijacking.

Our human brain is frequentist. That means that our brains are wired to see patterns and to develop conclusions from the patterns we see. Our whole scientific development was based on observational evidence. We originally concluded that the sun was at the center of the universe and without the math and physics to prove otherwise, we went with this. Only after centuries of study did humans discover the vast universe and the place that we are in.

The coin flip is a good example. Is the coin flip biased in some way, or it is an acceptable deviation from fair? Watching the coin flip, you can throw binomdist(35,100,.5,true) into excel to discover that the event of 65 or more occurrences of heads has a 0.09% chance of happening and conclude that you are very lucky or unlucky (if you are betting) or demand that the coin and the event be examined for fair play.

Now i am sure that when early coins came out back in the day that people made determinations in games based on coin flips knowing that likely the odds were even that it would flip one way or the other. Early probability theory without mathematical proof was likely a no-brainer.

So in the case of coin-flipping I think one has to be Baynesian because modern coins are fair and there is no evidence that supports bias. In the case of other games, the more complex the math is to understand I think the more frequentist we become, because it's difficult to overcome the evidence of patterns that you see with the actual results that occur.

Slot machine inventors know this. Casino game manufacturers know that near-misses get people hooked.

Quote: Wizard

Over lunch today I had an in-depth discussion about the difference between the Bayesian and frequentist approaches to probability.

Quora has a good discussion of differences. https://www.quora.com/What-is-the-difference-between-Bayesian-and-frequentist-statisticians

Giri Gopalan has a good answer.
"...Yes, there are undoubtedly conceptual differences but not enough to arouse the amount of animosity between both "camps" that exists, and these differences seem smaller in magnitude compared to the differences between a design vs. model or parametric vs. nonparametric approach. Furthermore, most modern statistical frameworks seem to blend the two in some way, as will be discussed."

IMO a frequentist would also factor in that "100 flips is a very small sample"
35% is the "sample mean"
...but what is the "population mean".

A Bayesian would include "a priori" beliefs:
(a) Likelihood that coin is fair: 99.9-99.999% (or whatever you want).
(b) More sophisticated model of coin manufacture - e.g. coins with defects, and old coins vs. new coins in bank storage, etc...
Things that many people would ignore, or make unstated assumption.

Consider the difference between a "Bayesian credible interval" and a "frequentist confidence interval".
Most people probably misunderstand confidence intervals & use them like credible intervals. The difference is subtle.

---
P.S. Likelihood of a coin being damaged on drops from a high building is quite high. So "i.i.d" is a probable fail.

Fun xkcd comic from Nishaanth Shanmughasundaram (on Quora thread).

Frequentists vs. Bayesians (Did the sun just explode?)

https://xkcd.com/1132/

What if you had flipped the coin 1,000 times and the outcome was 650 heads and 350 tails? Or 100,000 trials and 65,000 heads and 35,000 tails?

How many trials does it take with "unlikely" results before the Bayesian loses confidence in the underlying assumption that the coin is not weighted or the coin flip is not influenced in ways you have not thought of?

There are four sectors of knowledge:
-the things you know you know
- the things you don't know you know
- the things you know you don't know
- the things you don't know you don't know

It is that last category that is so confounding, and intelligent humans try to "get a clue" about what they don't know they don't know by observing and noticing patterns or actions that are not what you would have expected.

If you assume that a coin flip or casino game is unbiased and completely random and reject all information to the contrary, then you are not using the brain that God gave you.

Don't use a newly minted coin. Use a blank before it's minted.

You are the captain of an NFL team. It's time to decide who kicks off. You know this:

Quote: Wizard

Suppose a US-minted coin is selected at random from a random bank. It is then given a fair flip...

by the referee.

Do you call heads or tails?

Prior to the next game, you have your clubhouse attendants do some spy work and determine this about the coin to be used for this afternoon's game:

Quote: Wizard

...(let's say dropped from a tall building) 100 times. The results come back 65 tails and 35 heads.

Do you call heads or tails?

Quote: gordonm888

There are four sectors of knowledge:
- the things you know you know
- the things you don't know you know
- the things you know you don't know
- the things you don't know you don't know.

A professor in college called the last category the "unk unk".
One approach to the "unk unk" in household budgeting is to allocate 10% of the annual budget to "unexpected expenses".
...which of course, won't handle black swan events (e.g. catastrophic accident not fully covered by insurance).

...and almost everyone posting doesn't really seem to understand either the Bayesian or Frequentist philosophies, so it's somewhat weird (imagine people debating Democrat vs. Republican but not realizing what kinds of positions each side takes).

-----
Modeling methods are much more important than the Bayesian/Frequentist difference. (as one commenter on Quora nicely summed up)

(1) How many levels of models, meta-models, etc... do you use?
(2) How much parameterization do you use?

https://en.wikipedia.org/wiki/Parametric_statistics

"Parametric statistics is a branch of statistics which assumes that sample data comes from a population that follows a probability distribution based on a fixed set of parameters. Most well-known elementary statistical methods are parametric. Conversely a non-parametric model differs precisely in that the parameter set (or feature set in machine learning) is not fixed and can increase, or even decrease if new relevant information is collected.

Since a parametric model relies on a fixed parameter set, it assumes more about a given population than non-parametric methods do.[4] When the assumptions are correct, parametric methods will produce more accurate and precise estimates than non-parametric methods, i.e. have more statistical power. As more is assumed when the assumptions are not correct they have a greater chance of failing, and for this reason are not a robust statistical method. On the other hand, parametric formulae are often simpler to write down and faster to compute. For this reason their simplicity can make up for their lack of robustness, especially if care is taken to examine diagnostic statistics."

Quote: IndyJeffrey

You are the captain of an NFL team. It's time to decide who kicks off. You know this:

by the referee.

Do you call heads or tails?

Prior to the next game, you have your clubhouse attendants do some spy work and determine this about the coin to be used for this afternoon's game:



Do you call heads or tails?

Tails of course. You have the best of both worlds and believe that you have a 50% or greater chance of winning!

I think the main difference (having never heard of the two alternatives before) is whether you accept known information or only what you've seen.

In this case you may (or may not) know how many coins in the bank tend to be biassed or you may make an educated guess that it's less than 1 per 10^N (where N could be 6 9 or whatever). With 35 of 100 tosses, being one specific side is actually not that rare (.001758821), so you would ignore the problem as just luck and say 50%/50%.

However suppose you found it was 35000 from 100k tosses, now it's beginning to seem like you have a biassed coin. Eventually as the numbers got larger you would switch sides and say almost certainly you had a biassed coin and say 35%/65%.

Thanks for the responses everybody. No further comment from me. I think I'm going to let this topic go.