Frequentist vs. Bayesian statistics: resources to help you choose (UPDATED)

There are two dominant approaches to statistics. Here, I explain why you need to choose one or the other, and link to resources to help you make your choice.

Most ecologists use the frequentist approach. This approach focuses on P(D|H), the probability of the data, given the hypothesis. That is, this approach treats data as random (if you repeated the study, the data might come out differently), and hypotheses as fixed (the hypothesis is either true or false, and so has a probability of either 1 or 0, you just don’t know for sure which it is). This approach is called frequentist because it’s concerned with the frequency with which one expects to observe the data, given some hypothesis about the world. The P values you see in the “Results” sections of most empirical ecology papers are values of P(D|H), where H is usually some “null” hypothesis.

Bayesian statistical approaches are increasingly common in ecology. Bayesian statistics focuses on P(H|D), the probability of the hypothesis, given the data. That is, this approach treats the data as fixed (these are the only data you have) and hypotheses as random (the hypothesis might be true or false, with some probability between 0 and 1). This approach is called Bayesian because you need to use Bayes’ Theorem to calculate P(H|D).

At a broad-brush verbal level, both these approaches sound eminently reasonable, to the point that differences between them sound subtle to the point of unimportance. A frequentist basically says, “The world is a certain way, but I don’t know how it is. Further, I can’t necessarily tell how the world is just by collecting data, because data are always finite and noisy. So I’ll use statistics to line up the alternative possibilities, and see which ones the data more or less rule out.” A Bayesian basically says, “I don’t know how the world is. All I have to go on is finite data. So I’ll use statistics to infer something from those data about how probable different possible states of the world are.” And indeed, there are contexts in which Bayesian and frequentist statistics easily coexist.

But there are many contexts in which they don’t; frequentist and Bayesian approaches represent deeply conflicting approaches with deeply conflicting goals. Perhaps the deepest and most important conflict has to do with alternative interpretations of what “probability” means. These alternative interpretations arise because it often doesn’t make sense to talk about possible states of the world. For instance, there’s either life on Mars, or there’s not. We don’t know for sure which it is, but we do know for sure that it’s one or the other. So if you insist on trying to put a number on the probability of life on Mars (i.e. the probability that the hypothesis “There is life on Mars” is true), you are forced to drop the frequentist interpretation of probability. A frequentist interprets the word “probability” as meaning “the frequency with which something would happen, in a lengthy series of trials”. The most common alternative interpretation of “probability” (though not the only one) is as “subjective degree of belief”: the probability that you (personally) attach to a hypothesis is a measure of how strongly you (personally) believe that hypothesis. So a frequentist would never say “There’s probably not life on Mars”, unless she was just speaking loosely and using that phrase as shorthand for “The data are inconsistent with the hypothesis of life on Mars”. But the most common sort of Bayesian would say “There’s probably not life on Mars”, not as a loose way of speaking about Mars, but as a literal and precise way of speaking about his beliefs about Mars. A lot of the choice between frequentist and Bayesian statistics comes down to whether you think science should comprise statements about the world, or statements about our beliefs.

I’m a frequentist. But lots of very smart people aren’t. This post isn’t an argument for or against either philosophy. It’s just to alert you that this philosophical conflict exists, that it is very deep, and that you, as a working scientist, need to be familiar with it in order to make an informed choice of statistical approach. One thing frequentists and Bayesians agree on is that it’s a bad idea to do “cookbook statistics”, where you just mindlessly choose and follow some statistical “recipe” without worrying about why the recipe works–or even about what it’s trying to cook! I agree with Ellison and Dennis (2010) that ecologists should be “statistically fluent”, although I disagree with them that taking calculus-based technical courses in statistics is the only way to achieve fluency. Note that “fluency” is not at all the same thing as “technical proficiency”. If anything, I think one unfortunate side effect of the increasing popularity of technically-sophisticated, computationally-intensive statistical approaches in ecology has been to make ecologists even more reluctant to engage with philosophical issues–i.e. less fluent, or else less likely to care about fluency. It seems like there’s a “shut up and calculate the numbers” ethos developing, as if technical proficiency with programming could substitute for thinking about what the numbers mean. Lee Smolin noted a similar trend in fundamental physics.

Unfortunately, even advanced stats textbooks aimed at ecologists mostly don’t bother with more than the most cursory philosophical remarks. For instance, Clark (2007) spends only two pages on philosophy of statistics. And he uses those two pages to argue for the irrelevance of statistical philosophy to the real world scientist, because longstanding philosophical debates show no sign of definitive resolution! As I’ve noted elsewhere, this is a terrible argument for “pragmatism”, analogous to arguing that debates between liberal and conservative political philosophies are longstanding, and therefore irrelevant to the real world voter. Bolker (2008) is an admirable exception to this general reluctance of ecological statistics textbooks to grapple with conceptual issues.

So below is some food for thought, a compilation of some interesting and provocative writings I’ve found really helpful in developing my own philosophy of statistics. I encourage you to dip into them.

Note that most of the items I’ve listed assume some basic familiarity with different statistical philosophies, beyond the very brief sketch I gave above. Unfortunately, I have yet to find a really good, freely available, non-technical introduction to alternative philosophies of statistics, pitched at a level suitable for any professional ecologist or grad student. The discussion in Bolker (2008) is the sort of thing I’m thinking of, but it’s part of a book that costs money. Anyone know of anything good?

Books:

Error and the Growth of Experimental Knowledge by Deborah Mayo. Great defense of frequentist statistics as part of a broader philosophy of science, and a great compilation and debunking of the (often jaw-dropping) criticisms of frequentist statistics by Bayesians. I suspect a lot of scientists who consider themselves Bayesians, or who use Bayesian methods without really worrying about the philosophy most commonly used to justify those methods, may be rather shocked to discover just what sort of philosophy they’ve gotten into bed with. Even if you don’t buy Mayo’s argument for frequentist statistics and against Bayesianism, you ought to engage with her broader argument that one’s choice of statistical philosophy should be dictated by one’s philosophy of science.

The Nature of Scientific Evidence, Mark Taper & Subhash Lele (eds). Great series of chapters (many followed by critiques and rejoinders) covering a range of issues to do with philosophy of statistics and learning from evidence more generally. The author list is an all-star collection of ecologists, statisticians, and philosophers.

Articles:

Population ecologist Brian Dennis’ (1996) polemic on why ecologists shouldn’t be Bayesians is essential reading, and a lot of fun. Think philosophy of statistics is abstruse, technical, or dry? Read this, then think again. Think choice of statistical philosophy has no real-world consequences? Read this, then think again.

My fellow Canadian Subhash Lele has done a lot of very original work in statistical methods, much of it showing how to use frequentist methods to do things that frequentists supposedly can’t do, such as fit complex hierarchical models or incorporate prior knowledge and expert opinion.

Statistician Brad Efron invented bootstrapping. So, he’s a smart guy. So you should probably be interested in what he has to say about things like the challenges modern-day “Big Data” raises for both Bayesian and frequentist approaches, and about some points of commonality between frequentists and Bayesians. He’s one of those people whose off-the-cuff remarks are more incisive and interesting than most people’s most rigorous efforts.

Blogs:

Statistical Modeling, Causal Inference, and Social Science. Statistician and social scientist Andrew Gelman’s blog. He’s an entertaining, breezy writer, and he writes about all kinds of stuff, from highly technical statistical problems sent to him by readers, to plagiarism in science, to US politics, to inferring causality from non-experimental data, to philosophy of statistics. He’s an unusually thoughtful pragmatic “Bayesian” (I put that in quotes because he’s so unorthodox and frequentist in his Bayesianism that one can probably question whether he’s really Bayesian at all). If you insist on being a pragmatist, then you at least ought to be the sort of pragmatist Andrew Gelman is. Be a pragmatist because you have a deep understanding of and appreciation for alternative philosophies, not out of ignorance of alternative philosophies, or because you think philosophy doesn’t matter. For a sampling of his voluminous work on the philosophy of Bayesian statistics, see here, here, here, here, here, here, here, and here (yes, that’s only a sampling!)

Error Statistics Philosophy. Philosopher Deborah Mayo (see above) just started a blog! Unfortunately, it is the ugliest blog in the world (P<0.001), but don’t let that stop you. Mayo pulls no punches, and she grounds her philosophical discussions in practical, real-world examples. Her blog has links to many of her articles, and it helps to read one or two of them before diving into her blog, as she tends to assume more familiarity with the issues than does Gelman.

UPDATE: Deborah Mayo herself has added some comments, in particular on the importance of seeing frequentism as an approach to “error statistics” (briefly, the view that it’s the job of statistics to help us root out, and rule out, sources of error).