Posted by: Jeremy Fox | June 10, 2011

Objections to microcosms in ecology, and their answers

I work in protist microcosms. A typical experimental unit is a glass jar with 80 ml of nutrient medium, which I inoculate with bacteria, protists, and perhaps other microbes. Then I (actually my research assistants) follow the resulting population dynamics. Several features of this system combine to make it an excellent model system for asking fundamental questions about population and community dynamics. Protists have very short generation times (4-48 h), so you can collect hundreds of generations of data in a few months. Microcosms are easy and cheap to replicate highly, giving you a lot of statistical power. You can control and manipulate features of the system that are difficult or impossible to control and manipulate in most other systems. I tend to use the system to ask questions about the consequences of particular processes or combinations of processes that would be difficult or impossible to ask in other systems. For instance, how does productivity, in isolation from other factors, affect the occurrence of alternate stable states and assembly cycles (Fox 2008 Oikos)?

The advantages and drawbacks of protist microcosms, and of artificially-assembled communities more generally, have been debated in various places (see, e.g., papers in the April 1996 issue of Ecology and chapters in Bernardo and Resetarits (eds.) 1998). But over the years I’ve run into various objections that either haven’t been addressed publicly, or haven’t been addressed in quite the way I’d address them. Plus, although the debate over microcosms seems to have died down a while ago, who knows when it will flare up again, so I figured I might as well take a preemptive shot in the next Microcosm Wars.

So below are some objections to microcosms (in bold), and my own answers. I emphasize that I really have encountered all these objections (sometimes in unsigned reviews, so I can’t link to the sources); I’m not setting up straw men here. I also emphasize that, in answering these objections in the way I do, I’m speaking only for myself. Microcosm experiments are conducted for different reasons by different people, who would probably have different answers to these objections. Indeed, one overarching problem I have with these objections is that they’re blanket objections, and those who raise them often don’t seem to pay attention to the various quite distinct reasons why microcosm experiments are conducted.

I should also note that microcosms absolutely have their limitations. There are plenty of questions you just can’t ask using microcosms. For instance, protists lack stage-structure, so you can’t study stage-structured population dynamics with protists. Protists and other small organisms aren’t convenient for collecting individual-level data, and so if you want to scale up from individual-level data to population dynamics you need a different system (I understand that this is why my fellow Oikos editor Andre de Roos switched from working on zooplankton to working on fish). It’s hard to study the bacteria in a typical protist microcosm, which means I (like many ecologists) mostly treat the bacterial portion of the system as a ‘black box’ whose effects I hope I can summarize implicitly (e.g., by treating bacterivorous ciliates as growing logistically, rather than actually modeling their interactions with the bacteria). But those kinds of detailed considerations are of a very different character than the blanket objections to microcosms which I discuss below.

Microcosms just use organisms to solve equations. Yes, but so do non-microcosm systems. Anything can be described with equations, and so all organisms can be said to ‘solve equations’. The challenge in microcosms, just as in any other system, is to figure out what equations the organisms are solving, at least to a sufficiently good approximation to answer the scientific question of interest. (And if you insist that your system is so complex that it can’t be described with equations, how do you know humans can comprehend it?)

Microcosms are rigged; the results can’t help but confirm the theory. Boy, I wish that were true! Believe me, if I could rig my experiments so they always came out the way I wanted them too, I totally would (and my cv would look a lot more impressive!) I can actually see where this objection comes from, because there certainly are microcosm papers which report such clean results, conforming so beautifully to theoretical expectations, that it’s hard for someone who doesn’t work in the system to believe the experiment wasn’t somehow ‘rigged’. Gause’s (1934) competition experiments are famously well-described by the Lotka-Volterra competition model, and Kaunzinger and Morin’s (1998) test of food chain theory is a terrific recent example. But those kinds of theory-confirming papers not only aren’t rigged (more on that in a second), they’re far outnumbered by theory-disconfirming papers. You know how I just said that, in microcosms, the challenge is to figure out what equations the organisms are solving? Well, that’s a big challenge, because it’s often not the equations you thought they were solving. Take it from me, even though protists aren’t animals, they definitely obey the Harvard Law of Animal Behavior (‘under carefully controlled conditions the animal does what it damn well pleases’). For example, see Harrison 1995 and Fox 2007, to pick just two of many possible examples. Hard as it may be to believe for people who don’t work in microcosms, they do have the capacity to surprise. They can reveal new phenomena and unexpected behavior (e.g., effects of species loss on food web connectance in Fox and McGrady-Steed 2002; effects of dispersal on within-patch demography in Fox 2007). That capacity for surprise is what makes them useful. Microcosms are complex enough to surprise, but simple (and controllable, and replicable) enough that the source of the surprise hopefully can be identified. If your system is hugely complex and ‘noisy’ (which is often shorthand for ‘hugely complex’), it can be hard to tell if you’ve been surprised or not, because of the many factors that can obscure the signal you’re looking for. It can be even harder to figure out why you’ve been surprised. By throwing up ‘tractable surprises’, microcosms can suggest new ideas and teach us new things, not just serve a negative role as a first-cut test of theory. ‘If a theory doesn’t work in microcosms, it won’t work anywhere’ is a claim sometimes used to justify microcosm experiments, but it’s not a claim I personally buy (I used to, but not anymore).

And even when microcosms do conform to theoretical predictions, well, that’s generally a surprise too. Because microcosms never conform precisely to theoretical assumptions—that’s pretty much impossible. So even when the theoretical predictions are supported, the experiment demonstrates that those predictions are robust to the ways in which microcosms violate the relevant theoretical assumptions. For instance, Kaunzinger and Morin (1998) is a beautiful confirmation of Oksanen et al.’s (1981) predictions for how equilibrium biomasses on different trophic levels change along a productivity gradient—even though Kaunziner and Morin’s population dynamics were highly non-equilibrial (they averaged over a predator-prey cycle), even though their top predator seriously violated the assumption of a linear functional response, even though their basal trophic level in some treatments comprised a heterogeneous mixture of bacteria rather than the single species assumed by Oksanen et al., and even though their unshaken microcosms violated the assumption of a spatially-homogeneous system. Indeed, theoretical models that explicitly incorporate precisely these complications actually make different predictions than Oksanen et al. So a very interesting question is why the Oksanen et al. predictions are robust to these complications. One way to get at that question would be to manipulate features of the system like the number of species per trophic level and the spatial homogeneity, and indeed such manipulations have been conducted (e.g., Fox 2008 Oikos, Fox and Barreto 2006). I suspect this objection gets raised because my fellow microcosmologists and I often emphasize the ways in which our system conforms to theoretical assumptions while neglecting to emphasize the ways in which it doesn’t.

Microcosms aren’t ‘realistic’ or ‘natural’. I can totally see the point of this concern if (and only if!) the goal of a microcosm experiment is to mimic some specific natural system, or some specific feature of some specific natural system. For instance, if you put an individual predator and some prey in a small container in order to estimate the predator’s feeding rate in nature, you’d better be sure that the container doesn’t introduce any artifacts which bias your measurements (e.g., lacking prey refuges which exist in nature). This is the main reason why Carpenter (1996) objects to microcosms. But if that’s not the goal of the microcosm experiment, then this objection is simply irrelevant and frankly it frustrates and mystifies me when it’s raised. I wonder if those who raise this objection have really thought it through, and ask that they consider the following points:

a. In many respects, microcosms are realistic. For instance, all microcosm experiments use species found in nature, and most use species that coexist in nature. Nobody uses robots, or Jurassic Park-style lab-created species, or children pretending to be animals (but see Bell 2007). And the resulting dynamics are realistic in many respects too. For instance, the average strength of trophic cascades in protist microcosms is almost frighteningly similar to the average strength of trophic cascades in nature (Fox 2007 Oikos). Protist microcosm food webs exhibit pyramids of numbers and biomass, just like many natural food webs. Etc.

b. Sure, microcosms aren’t exactly like any specific natural system. But neither is any other natural system! As every ecologist surely knows, natural systems differ a lot from one another, and change over time. ‘Natural’ isn’t one state of affairs, it’s a massive range of states of affairs. So if you say microcosms aren’t natural, which natural system are you comparing them to? And don’t say ‘they’re unnatural compared to every natural system that’s ever existed or could exist’. No one knows what sort of natural systems could exist, and more importantly protist microcosms actually are reasonable analogues for small, naturally-occurring water bodies such as puddles and rockpools (McGrady-Steed and Morin 1996).

c. Many microcosms aren’t intended to be realistic or natural. Again, the point of my experiments is not to reproduce the behavior of any particular natural system, it’s to obtain information that couldn’t be obtained in any other way. Criticizing this kind of experiment as ‘unnatural’ is like criticizing a car because it can’t fly.

d. No experiment is ‘natural’ or ‘realistic’. That’s the whole point of experiments, including field experiments—to change nature, to create conditions that wouldn’t otherwise exist, i.e. unnatural conditions. We do that because a really good way to obtain information about how realistic systems work is to create unrealistic conditions.

e. You’re effectively insisting that we discard useful information which we could not obtain in any other way. Microcosms, by allowing you to create conditions which would be difficult or impossible to create in nature, allow you to expand your information base. For instance, it’s really useful to know that, under the conditions produced by Kaunzinger and Morin (1998), the Oksanen et al. model works. It mostly doesn’t work under other conditions, so comparing the systems where it does work to the systems where it doesn’t lets you develop testable hypotheses about the circumstances under which it works or doesn’t work. You wouldn’t be able to make such comparisons if all you had were systems where the model doesn’t work. So if you don’t think we should do microcosm experiments because they’re ‘unrealistic’, you’re effectively saying that you want less information, about a smaller range of systems and conditions. Please explain how that could possibly be a good thing! And don’t just explain it to me, explain it to Charles Darwin. Charles Darwin, the greatest naturalist in history, didn’t just rely on observations of unmanipulated nature for evidence for evolution by natural selection. One of his most important lines of evidence, the line of evidence with which he chose to begin the Origin, was the effects of artificial selection in domestic species. Artificial selection provided Darwin with evidence relevant to interpreting nature and testing his hypotheses, evidence that he could not have obtained any other way, even though artificial selection as practiced by breeders is highly unrealistic. For instance, selection coefficients imposed by breeders often are much stronger than those typically observed in nature, and breeders often select for traits that would never be favored in nature. My point here is not proof by authority; Darwin wasn’t infallible. My point is merely to ask those who have a blanket objection to all microcosm experiments as ‘unnatural’ or ‘unrealistic’ (e.g., Carpenter 1996) to carry their arguments to their logical conclusions.

f. If there’s some specific respect in which you think microcosms are unrealistic, it’s probably possible to manipulate that feature of the system and thereby turn your objection into a testable hypothesis. A constant, undisturbed environment is unrealistic? Ok, let’s manipulate environmental variation or disturbance (e.g., Warren 1996 Oikos, McGrady-Steed and Morin 1996 Oikos, Fukami 2001 Oikos). It’s unrealistic to mix species with no coevolutionary history? Ok, let’s manipulate coevolution (see, e.g., the work of Mike Brockhurst, among many others). It’s unrealistic for a community to be closed to immigration and emigration? Ok, let’s open it to dispersal (e.g., Holyoak and Lawler 1996, Warren 1996 Oikos). Etc.

Microcosms are too small-scale. Umm, you are aware that protists are really tiny, right? Population sizes in my microcosms are on the order of 10,000-1,000,000, which is as large or larger than the population sizes in the study areas most ecologists use to study  most macroscopic organisms (e.g., there are only 225,000 trees in the famous 50 hectare plot on BCI). So relative to the size of the organisms, microcosms are actually large-scale (and long-term, relative to the generation times of the organisms). And unshaken microcosms contain substantial spatial heterogeneity at spatial scales relevant to microbes (e.g., Meyer and Kassen 2007). Plus, spatial scale can be manipulated, for instance by manipulating vessel size, and by studying arrays of microcosms linked by dispersal (e.g., Holyoak and Lawler 1996).

Microcosms lack the full range of processes that occur in nature. The objection here, as best I can understand it, is that a system that includes only a subset of whatever goes on in any natural system cannot tell us anything useful about nature. Rather than extending our information base, microcosms just represent irrelevant ‘apples’ that can’t usefully be compared to natural ‘oranges’. Again, if the objection is only meant to apply to microcosms that try to mimic specific natural systems, then I can appreciate it. But if it’s meant to apply more broadly, then I don’t buy it at all, because it amounts to denying that ecologists can build up an understanding of how a complicated ecological whole works by studying its parts. Carpenter (1996) claims that studying parts of a system is informative in molecular biology (‘a molecular biologist who isolates ribosomes is working on ribosomes’), but not in ecology (‘an ecologist who isolates organisms in bottles may not be working on communities and ecosystems in any relevant sense’). Unfortunately, I confess I don’t fully understand his reasons for this claim. One of his reasons is that ‘there is general agreement about human health goals that rationalize most of [molecular biology’s] funding’. What this has to do with the scientific validity of ecological microcosms I have no idea. Another is that ‘relatively rapid replicated study is possible at several levels’ in molecular biology, which you would think would be an argument in favor of microcosms in ecology since rapid, replicated experiments are what microcosms are for. The claim that ecologists cannot build up an understanding of a complex ecological whole by studying its parts just strikes me as so obviously false that I don’t even know what to say in response.

Microcosms should not be studied for their own sake. The ultimate goal should be to understand nature. I don’t know anyone who studies microcosms for their own sake. I sure don’t. I mean, yes, I do play close attention to the results of previous experiments in microcosms, in order to help me design my own experiments. But that’s surely no different than what any field ecologist does—you build on what’s already known in order to learn something new. The mere fact that someone conducts a lot of microcosm experiments, and no field studies, is not evidence that he or she is just interested in microcosms for their own sake. I ride the bus a lot too, but I don’t ride the bus for its own sake. A frequently-employed means to an end is still a means to an end, and contrary to Carpenter (1996) I don’t believe there’s any serious risk that that a frequently-employed means will ever be mistaken for an end. Heck, field experiments are only a means to an end, but you don’t see anyone worrying that, if we focus too much on field experiments, we’ll start treating them as an end in themselves. (Come to think of it, maybe we should start worrying about that…) (just kidding)

As an aside, while the ultimate goal of microcosm work, including my own, often is to understand nature, worthwhile work in ecology can have other goals (e.g., Caswell 1988 talks about how theoretical ecology has ‘a life of its own’, independent of empirical data).

It’s irresponsible to train people to do microcosms; to solve real-world ecological problems we need field scientists with a deep appreciation of natural history and a ‘sense of the ecosystem’. Carpenter (1996) says this (and yes, he actually uses the word ‘irresponsible’). I’m glad he says it, because I suspect this gets to the heart of the matter. All the above objections, I suspect, really spring from the feeling that, if you’re not doing it in the field, and if your starting point is not natural history, you’re not doing proper ecology.

If you buy this objection, I doubt there’s anything I or anyone can say to change your mind. But a few remarks are in order.

First, if solving real-world ecological problems is your overriding goal, you should seriously consider going into law, politics, or economics rather than field ecology. The ultimate causes of anthropogenic impacts on the environment are not ecological. And while ecological knowledge is essential for addressing those impacts,  lack of knowledge typically is not among the biggest impediments to addressing those impacts.  But that’s a discussion well beyond the scope of this blog entry.

Second, I personally am a crap natural historian. So the fact that I work in microcosms is no loss to natural history-driven field ecology. 😉 (By the way, this is emphatically not true of everyone who works in microcosms. Peter Morin for instance is an amazing natural historian)

Third, no one I know trains students to be ‘microcosmologists’. Personally, I like to think that I train students to be scientists. For instance, one of my current graduate students entered my lab with a strong interest in working in microcosms, and I steered him towards alpine plant-pollinator interactions because that seemed like a better system in which to address the questions he wanted to address.

Fourth, I can tell you, from my own personal experience, that training students to work in microcosms does not significantly reduce the pool of students who want to train as field ecologists. I struggle to attract graduate students (and undergraduate honors students), and the ones I do attract mostly don’t end up working in microcosms. The vast majority of undergraduates interested in ecology get into ecology because they love the outdoors and wild nature, and they choose graduate supervisors and careers that reflect that love (which I think is great, by the way). Let me tell you, it’s very hard to convince such students to do ecology indoors. If we are indeed, as Carpenter (1996) claims, short on students with the natural history background to do field ecology, it’s not because students who have that background are getting sucked into doing microcosms instead. The students I mostly attract are the rare ones who, although they often love wild nature, also love big, general questions that are relevant to many systems. These students feel, as I do, that if a question could in principle be asked in any system, we may as well ask it in a model system that has features that make it easier to get a clear-cut answer.

Fifth, I respectfully disagree that, in order to solve the world’s ecological problems, and train the next generation of ecologists who will help solve those problems, that we should only train field ecologists. Natural history and a sense of the ecosystem are hugely important, but they cannot answer all the questions we ought to be asking, and indeed can’t even identify all the questions we ought to be asking. For instance, the ideas of alternate stable states, hysteresis, and critical slowing down, which have tremendous management relevance and on which Steve Carpenter himself is currently working, derive from dynamical systems theory and weren’t originally discovered by field ecologists. The competitive exclusion principle, which is implicitly or explicitly part of every field study of competition (e.g., between invasive species and natives) was not discovered by a field ecologist. The many surprising and counterintuitive behaviors of nonlinear, stochastic dynamical systems (e.g., chaos), which are a major plank in the argument for ‘adaptive management’, were not discovered by field ecologists (for instance, think of ‘stochastic resonance’, which may explain outbreaks of many pest insects). The fact that ‘diversity’ does not necessarily promote, and can even inhibit, ‘stability’, was not discovered by a field ecologist (indeed, field ecologists pretty much thought the opposite until Bob May came along). Modeling global climate change and its ecological consequences requires lots of mathematical and computer programming skills that could not realistically be part of the standard training for field ecologists. I could go on, but you get the idea. Good ecology, even good conservation ecology, and even good system-specific conservation ecology, often depends on ideas and skills drawn from other sources than natural history. A ‘sense of the ecosystem’ is necessary but not sufficient to conserve the ecosystem, because ecosystems are far too complex for that.



  1. […] Steve Ellner. Great stuff. Steve’s thoughts on why microcosms are useful very much mirror my own. And he has a very trenchant question at the end about whether much of the putatively […]

  2. […] once the global economy recovers. Opportunity costs are ever-present. A dollar given to me to grow bugs in jars is always going to be a dollar not spent on something else. Plus, the world is always going to have […]

  3. […] paper on which I am a co-author just got rejected. Which seems as good an excuse as any to repost this, in which I shoot down all the commonly-voiced blanket objections to microcosms in […]

  4. I agree with you Jeremy, that microcosm experiments have their place in Ecology, just like many other approaches. As long as we accept their limitations, we should not ignore their utility either.

  5. Jeremy, your post forced me to re-examine my own mild skepticism about micro-organism laboratory experiments (although I want to emphasize that the skepticism is mild – I think that Rees Kassen, Graham Bell, Peter Morin, Owen Petchey, yourself, write papers that I am always eager to read). And so where does the skepticism come from? It turns out it comes from the same place that my general skepticism about ecology comes from – that ecologists often make claims to understand the world but are rarely able to demonstrate that understanding (and I put myself solidly in the camp of scientists that have provided no evidence that they have contributed, even incrementally, to understanding the natural world better). That’s a fairly sweeping claim (although not a new one – the late Rob Peters made this point explicitly in A Critique for Ecology) but, I believe, a true one.
    I make the claim based on two assertions, one logical and the other semi-empirical. The first assertion is that there is only one way to demonstrate understanding and that is through prediction. An ecologist, a natural historian, a farmer, a fisherman, a hunter can all make claims to understand how the natural world works but the only way they can demonstrate that they truly have understanding is to make predictions about the natural world that are more accurate than we would expect by chance. So, while understanding and prediction are not the same, understanding has one diagnostic and it is prediction (we could argue about whether prediction is sufficient to demonstrate understanding but it is absolutely necessary). The second assertion is that ecologists rarely make predictions (at least, relevant ones) and when they do, the predictions are usually poor. (This is going to get back to microcosm experiments – I swear.) That probably means I have to define what a relevant prediction is but I think you made the point in your post about microcosms – when push comes to shove we all have to acknowledge that our work usually only makes sense if we are trying to use it to understand what happens in natural systems. By extension, I would say that to demonstrate understanding about the natural world we have to make predictions about the natural world (not predictions about what happens in aquaria or beakers or field mesocosms or perhaps, even manipulated’natural’ systems).
    So, the skepticism I have about all experimental ecological research is proportional to the degree to which I think it is likely that the experimental units are representative of nature. More particularly, the degree to which those results will help me make predictions about the natural world. So, for micro-organism laboratory experiments that means there are two major sources of skepticism for me – one, if you are hoping to make general inferences to natural systems made up of microorganisms, are the ‘lab’ communities representative of natural communities? Now, I am less skeptical of microcosm lab experiments than I am many lab experiments because, as you point out, ‘lab’ communities contain large numbers of organisms and it is at least plausible that the spatial and temporal scales that these ‘lab’ communities are constrained to is not dramatically different that the ‘natural’ spatial and temporal scales that the processes of interest operate at in nature. The other source of skepticism is only relevant if researchers choose to make inferences beyond micro-organisms because now there is the additional concern that, even if the patterns observed in the lab also occur in nature, do they scale up? Are the relationships somewhat scale-invariant so that large differences in body size don’t result in qualitative differences in the processes being studied?
    But the proof of the pudding is in the eating, just as with any experiment (lab, field or ecosystem; big or small body) is anybody checking to see if the results from these experiments allow us to make predictions about natural systems that are more accurate than we would expect by chance? There can’t be any claim to better understanding of the natural world until that is demonstrated. And I would say understanding is not strictly a qualitative characteristic it can be quantitative – that understanding can be measured by the precision and accuracy of our predictions.
    So, I think you make a great case for the utility of microcosm experiments – if every experiment was held to the same standard ‘how likely are these experiments to contribute to improved predictions about the natural world’ I suspect that microcosm experiments would come out ahead of many other kinds of experiments. Having said that, microcosm experiments have the same failing that the vast majority of ecological research has – few tests to see if we understand the natural world better than we did before the experiments.
    I’ll finish with an example of how pervasive I think this problem is. Species-area relationships are as well-studied as any pattern in ecology. But could I use that century (or more) of know-how to predict how many fish species are in a randomly chosen lake in New Brunswick? What slope should I use? What intercept should I use? Are there any general principles that would allow me to select the appropriate slope and intercept and make a reasonably good prediction of the number of fish species in the lake (and by reasonably good, I only mean as good as could reasonably be expected given that we are using only a single variable to predict richness). I suspect we might be able to agree on the slope (0.25? 0.20? 0.30?) but what about the intercept? Would it depend on body size? Probably but I’m not sure how. Would it be different in aquatic than terrestrial systems? Not sure. What about homeotherms versus heterotherms? I don’t have a clear answer. My guess is that most ecologists would conclude that a century of research has led us to a place where we could say “a big lake will have more species than a small lake”. And we’re not even getting at mechanism here.

  6. Hi Jeff,

    I consider it the highest compliment to be told that I’ve prompted someone to re-examine their beliefs, so thank you very much for that.

    Your comment is a lengthy one that probably deserves at least an equally lengthy reply. By the way, nothing that I say below is intended as criticism of your very thoughtful comment.

    (i) Re: Peters’ work, it’s true that he emphasized the importance of prediction, but that came with a lot of other philosophical baggage with which I very strongly disagree. I’m not an instrumentalist (hardly any practicing scientists are). That’s just an aside. I take that, in mentioning Peters in passing, you didn’t mean to imply that your comment springs from an instrumentalist philosophy of science. (If you did, that would be a whole ‘nother conversation!). Certainly, plenty of non-instrumentalists have also attached great importance to prediction (e.g., the notion in philosophy of science of “inference to the best explanation”)

    (ii) Another passing remark about Peters, which is only tangentially relevant to your comment: the regression models of the sort Peters favored are infamous for not doing very well at predicting new data, especially when any sort of extrapolation is required. So if successful prediction is the one and only hallmark of successful science, Peters-style instrumentalist ecology isn’t very successful.

    (iii) Given the importance you attach to successful prediction, I take it you’re particularly impressed by those cases in which our theoretical models *do* successfully predict the outcome of microcosm experiments (e.g., Kaunzinger et al. 1998)? Or does that somehow not count because it’s “only” predictions about microcosms, not predictions about nature? This kind of gets back to the issue of the “realism” of microcosms. I kind of get the sense that you still feel like microcosms somehow aren’t “realistic”, but that you’re having trouble articulating this sense in a way that isn’t vulnerable to the counterarguments in my post.

    (iv) Are you worried about lack of successful predictions in ecology, or lack of successful *general* predictions? For instance, if you want to predict the species richness of a lake in New Brunswick at some point in the near future, you could perhaps do pretty well if you studied that particular lake for a while. But if you want to predict its richness just based on its area and the average richness-area relationship for all lakes in the world, yeah, that prediction probably won’t be as accurate. But I’m not sure why that should worry us. That just says that different ecological systems are different. Our failure to make precise-yet-generally-applicable predictions in ecology is surely a sign of *success* if such predictions are actually impossible! I mean, we’d be fooling ourselves if we thought otherwise, wouldn’t we?

    (v) Can you give a few examples of what you consider to be successful predictions in ecology? In particular, I’m curious about how precise you think successful predictions ought to be, or how you think the appropriate level of precision varies from case to case. From your example at the end, I get the impression that you think only very precise predictions (or at least quantitative predictions with precisely-specified confidence intervals), with very precisely-specified domains of applicability, will do.

    (vi) Re: prediction as the *only* way to demonstrate understanding, that is quite a strong claim. I’d only agree if “prediction” were interpreted in a much broader way than I suspect you intend (although I’m not sure–you don’t elaborate on what you mean by “prediction”). For instance, consider what’s known in philosophy of science as the “old evidence” problem. Here’s a standard example: the precession of Mercury was known before Einstein developed relativity theory. Should the fact that relativity theory implies the precession of Mercury count as a successful “prediction”? If so (and I, like the physicists of the time, do think it counts), that indicates that “prediction” doesn’t just mean “correct statements about data that have yet to be collected”. As an example that’s closer to home, in the Origin of Species Darwin famously makes no predictions about data yet to be collected (and certainly no quantitative predictions, or predictions to which confidence intervals could be attached). Rather, he shows how many already-known, apparently-unrelated features of the living world (including not just “nature” but also, e.g., domestic animals) are actually related: they can all be explained by the hypothesis of evolution by natural selection from a common ancestor, together with various ancillary facts and hypotheses (e.g., regarding geology). Did the Origin improve our understanding of nature?

    What I’d say is that good science can have various epistemological and cognitive goals, which are distinct albeit often intertwined. I agree that successful prediction (where “prediction” is interpreted suitably broadly) is one important goal, but it’s only one. I also agree that successful prediction is a powerful way to demonstrate understanding, but I don’t think it’s either necessary or sufficient for demonstrating understanding. For instance, it’s perfectly possible to make good predictions without any understanding of why they work. Think of the purely statistical predictions of the sort Peters loved. If I sample from a population of lakes and find such-and-such a regression relationship between primary productivity and zooplankton biomass, I can use that relationship to predict zooplankton biomass in a newly-sampled lake from that population, without having a clue about why the regression relationship holds. This is basically a slightly fancy version of what philosophers of science call the “straight rule” of induction–all you’re saying is “I predict the future will be like the past”. All the swans I’ve ever seen have been white, so I predict the next one I see will be white. Even though I have no idea why swans are white. As a second example of successful prediction without understanding, think of the use of an incredibly complicated, detailed, and realistic simulation model, like a GCM, to make accurate predictions. The model is so complicated that it’s a black box to us. It’s as complicated as nature itself (or at least close enough to make good predictions), and so we don’t understand how it works any better than we understand nature itself.

    Conversely, I think there are ways to improve understanding without making any successful predictions, or even any predictions at all. Think again of the Origin of Species. Or, as an ecological example, consider the discrete time logistic model. This model illustrates how time-lagged density dependence can destabilize population dynamics (many other models with time lags of course illustrate the same thing). Knowing that time-lagged density dependence is destabilizing improves my understanding of how population dynamics work, even if in nature there are no time lags, or there are time lags but their destabilizing effects are not observed for whatever reason (e.g., because they are overridden by stronger stabilizing forces). If you don’t know about the destabilizing effects of time lags, there is a gaping hole in your understanding of population dynamics. In other words, the discrete logistic, and other models with time lags, improve my understanding of how population dynamics work even if they don’t make any successful predictions.

    All of which is to say that I think microcosms can help us achieve various epistemological and cognitive goals without necessarily making successful predictions about nature. I get the impression that you and I might disagree, at least a bit, on the relative importance we attach to those various goals. I don’t think failure to make precise, quantitative predictions about data yet to be collected indicates that ecologists don’t understand nature, or that ecological science is weak science.

    (vii) Re: your question about whether microcosms are “representative” of nature, I’m not sure what else to say besides what I said in my post.

    (viiii) The view that “microcosm experiments are just as useful as field experiments–i.e. not very useful at all!” isn’t one I’ve encountered before. That’s a rather depressing point of view! It sounds like what you need isn’t so much a defense of microcosms as a defense of ecology! 😉 Or else a beer or something to cheer you up! 😉

    (ix) One of my grad students insists that successful prediction is the one and only legitimate goal of science, the only thing that matters. Responding to your comment has been good practice for arguing with him. 😉

    (x) Have you read Everything is Obvious (Once You Know the Answer) by Duncan Watts? A very good book on how rare truly successful prediction is, and on the many cognitive biases and other obstacles that prevent successful predictions. If you think ecologists are bad at prediction, well, they can join the club! Watts argues powerfully that, except in certain branches of the physical sciences and engineering, or in contexts in which the “straight rule of induction” works (“I predict the sun will rise tomorrow”), successful prediction is basically impossible. Not sure if that conclusion will cheer you up (“At least ecology’s no worse at prediction than most other sciences!”) or depress you further (“Good god, most other sciences are as terrible at prediction as ecology is!”).

  7. […] reading a very interesting string of comments on the Oikos blog, I had to look something up: instrumentalism. Wikipedia describes it as a philosophy that places […]

  8. […] fake it). I don’t pass the “eye test” of what a good ecologist should look like: I mostly work in a lab, don’t even own any boots to get muddy, and work on fundamental topics that aren’t […]

  9. Hi Jeremy, all this talk of prediction brought me back to this post – and it seems that end of term is giving me the chance to think a little.
    (i) I actually don’t know if I’m an instrumentalist. What I believe is that there is no way to demonstrate understanding without prediction. I will admit that you may be able to have understanding without prediction but you won’t be able to demonstrate it and so, in practical terms, not being able to demonstrate it is indistinguishable from not having it. So, can we understand things we can’t measure? I think we can but only to the extent that we can make persuasive arguments that there is a strong (and perhaps deductive) relationship between what we measured and the thing we believe it represents. So, for example, we can talk about diversity when we measure species richness but only to the extent that I can convince you that species richness is a good surrogate for diversity. Do you have an example of some phenomenon that we could not measure and that has no measurable surrogate that we could make claims to understanding and on what grounds we could make the claims?
    (ii) The fact that we haven’t been able to make good predictions (and I would put empirical regression models up against just about any other model you might have in mind) about the ecological world doesn’t say anything about whether prediction is the appropriate diagnostic for understanding. If we discover that it is impossible to make good predictions about ecological communities I believe that the unfortunate conclusion is that we will never be able to demonstrate understanding and any claims to understanding are unfounded. It’s fine to say that chaotic dynamics result in unpredictable systems but unless we can distinguish between chaotic systems and systems we just don’t understand then we can make no claims. And the only way that I can imagine distinguishing between chaotic dynamics and systems we don’t understand is with predictions.
    (iii) I would hold all experiments to the same standard – can they predict patterns in nature? So, I have no more problem with microcosm experiments than I have with field mesocosm experiments or whole-system experiments for that matter. Ultimately, they all have to be put to the test – can they explain observed patterns in nature. Even Schindler’s whole-lake experiment wasn’t enough to convince folks that phosphorus was the problem – it took a large-scale observational study by Dillon and Rigler (I think some Japanese scientists did a similar thing) to drive the last nail in the coffin. So, given that microcosms containing very small organisms are more likely to be representative of natural systems because the experimental scale is more likely to match up with the scale at which natural processes occur, I have more faith that they wil predict natural patterns. But I still want to see reasonably accurate/precise explanations.
    (iv) I want good predictions however we get them. But that means a predictive science that can combine general rules that some large subset of systems follow with specific rules about drivers in different KINDS of systems. So, I distinguish between needing different models to predict species richness in different kinds of lakes and poor predictions for lakes that we explain away because ‘all systems are different’. Look, if we can never develop models that will do a good job of predicting species richness in a new lake – that is, we have to develop separate models for each lake without understanding what drives the differences among lakes, that’s fine. But let’s admit that this implies we will never understand species richness in lakes very well.
    (v) I think there is no doubt that very precise, accurate quantitative predictions are better than biased, imprecise, or qualitative predictions. I suspect you agree with that. My guess is that those kind of predictions are very unlikely in ecology. But, I think by not setting accurate, precise quantitative predictions as our goal we make it easy to settle for less than we are capable. When was the last time you read a paper where somebody developed an empirical model using one set of data and then tested it on another? I know you will be able to give me examples but, is it common? I think because prediction in ecological systems is difficult in ecology we have chosen to look elsewhere for evidence of understanding. I believe that’s a mistake. So, I can give you no examples because (1) we are bad at it (perhaps inevitably so) and (2) a small fraction of scientists actually try to use their experimental, theoretical or observational studies to make predictions about new data.
    (vi) I think old data works for me (although I’m only giving it thought now). I don’t think I have any problem with old data being interpreted as evidence for a new explanation although it would be weaker evidence if the explanation was devised specifically to explain an old piece of evidence. If a new explanation is developed independently of the old evidence and then is shown to predict the old evidence I see it as strong evidence. To me it’s not about whether the data has already been collected but whether it was used to shape the explanation. And , of course, if an explanation can explain lots of old evidence without a continuous series of ad hoc adjustments to the explanation so the data fits, then I think we have strong evidence. And Darwin is held in such high esteem because his theory has continued to result in good predictions about new data. If the last 150 years had passed with little confirmation of his theory on new data his theories wouldn’t hold their central place in biology.
    I think understanding is demonstrated anytime somebody makes a better prediction that you or I could make by chance. That prediction may be coarse and qualitative or precise and quantitative. But the more accurate and precise the prediction the better the demonstration of understanding.
    Good science may have many objectives but one is to increase our understanding of the natural world. I would assert that there is only one way to demonstrate understanding and that is with prediction. I swear that I am open to hearing other ways of demonstrating understanding but so far I haven’t heard a convincing one.
    If primary productivity is a good predictor of zooplankton biomass can we really say we don’t understand zooplankton biomass? We may not understand why primary productivity predicts zooplankton biomass but we know it depends on primary productivity. That implies increased understanding. Now, I admit that understanding is hierarchical. We have to dig deeper to understand why primary productivity drives abundance. Here’s how I imagine this working. Let’s imagine that we can perfectly predict zooplankton biomass with productivity but we don’t understand the mechanism. That implies that primary productivity will predict some intermediate phenomenon that will then predict zooplankton abundance. And if we can find that intermediate phenomenon we will have increased our understanding further. And perhaps there will be several intermediate phenomena. And the more intermediate phenomena we identify and the more precisely we predict them, the better our understanding.
    And I don’t see GCM’s as being necessarily a good example because my guess is the people that designed the model do not see it as a black box at all. They could walk us through every equation, variable, functional relationship and parameter estimate. Now admittedly, they may not even be able to intuit very well what outputs would result from specific inputs but I’m sure they could track inputs through the model and explain why they led to particular outputs.
    And I don’t believe that understanding the properties of a model imply understanding of the natural world. The fact that you observed that lagged DD destabilizes populations in the model only means that you have a better understanding of the model, not of nature. If there are no time lags then we have modelled a phenomenon that doesn’t exist in nature and so can’t make claims to increased understanding of natural population dynamics. If we can’t detect the lags then we have no evidence that lagged DD destabilizes natural populations and so still can’t make any claims to increased understanding. If the model makes great predictions in every other context but lagged DD it is either that the model is faulty in some way or we don’t have the ability to detect a real effect. We have no way of knowing which is true and so can’t make any claims to understanding. There is no gaping hole in understanding the natural world if there are no lags or if there is no evidence that lags cause destabilization- perhaps there is a gaping hole in understanding the model.
    I think by this point it’s pretty clear that I agree with your graduate student and I go along pretty happily even though I am convinced it’s possible that ecologists have not done very much useful work (and I am one so if that’s an indictment it’s one of myself as well). I’m actually not absolutely sure because I have almost no idea of what we can predict because so few people do it. It’s one of my goals for the next decade or two – to figure out what we can predict.
    This has been a blast, as usual. And I’ve been incredibly long-winded, as usual. You have to take some of the blame… for creating this terrific forum for discussion and taking such an active part in it.

    Jeff Houlahan

    PS I haven;t read Everything is Obvious but I’ve put it on my list.

    • Thanks as always for your lengthy comments Jeff. At this point I think we’ve probably said to each other about as much as we can; we’ve started going in circles a bit (I’ve just raised the example of the Origin again to you on another thread over on Dynamic Ecology). Don’t know that I have something new to say in response to every point you make, so I’ll go for a broader reply.

      I don’t think we disagree too much on the value of testing predictions. I’m perfectly happy with the notion of prediction as a powerful way to demonstrate understanding, and in some contexts as a goal in its own right. I’m not ok with prediction as the only goal of science, as down that road is Robert Peters-style instrumentalism and I think the history of science demonstrates that that is an unsuccessful way to do science even as judged by its own goals. If all you care about is good predictions, and to hell with explanation and understanding, then paradoxically the history of science says that you’d better care about explanation and understanding! But since you emphasize prediction as a powerful (or even the sole) demonstration of our understanding, rather than as the only legitimate goal of science (with understanding and explanation being either impossible or valueless, which is what an instrumentalist would claim), my beef with instrumentalism isn’t really a beef with you.

      At least, it’s not a beef with you in principle; in practice, it may be a bit of one. I think there are practical risks to making too much of a fetish of the prediction of empirical data. It’s a very short step from your very strong emphasis on the importance of prediction to doing prediction badly. My concern here isn’t hypothetical. For instance, a lot of what went wrong with the whole history of research on the intermediate disturbance hypothesis is that there was a very strong emphasis on tests of one particular prediction (is diversity a humped function of disturbance?). With no emphasis on where those predictions came from (e.g., do the predictions even follow logically from the model assumptions?), no emphasis on tests of other predictions of the models, and no emphasis on tests of model assumptions (so as to discover if the models made the right prediction for the right reasons or if they were merely “getting lucky”). So we ended up with a lot of tests of predictions that actually have no basis in logic (and so no implications for how the world actually is), and no tests of other predictions or of model assumptions. Tests of other predictions, and tests of assumptions, might have revealed that testing the predicted diversity-disturbance relationship is, on its own, an incredibly weak test of any model of how disturbance affects diversity.

      Now, you’re perfectly entitled to respond by saying that weak or otherwise faulty tests of predictions merely highlight the importance of testing predictions well. To which I’d reply that “testing predictions well” often means we need to do much else besides test predictions. We need to also have logical models from which to derive predictions, we need to test assumptions as well as predictions, etc. I think subsuming all of that under the heading “testing predictions” is unhelpfully question-begging. It sweeps under the rug, or at least de-emphasizes, how to test predictions well. So if I don’t follow you by pushing for the overriding importance of making more precise and accurate predictions, well, one reason is because I’m afraid that such a push might drive unintended and counterproductive behavior. Analogously, one might say that student performance on exams is the best way for students to demonstrate their grasp of the material, and that might well be true. But even if that’s true, if you emphasize it to students and teachers too much, what you end up with is teachers “teaching to the test”, uncurious students capable only of rote memorization and whose first and only question about the material is always “will this be on the exam?”, etc.

      Happy to try to continue to move the discussion forward as best I can, but at this point I think I’m running out of new things to say about prediction. So I think I’ll hand you off to Brian over on Dynamic Ecology now–I think I can promise that you’ll like his forthcoming posts. 🙂

      p.s. I don’t write for Oikos Blog anymore, and comment threads on my old posts here aren’t really “live” anymore. All of my old Oikos Blog posts, and the comments made prior to my switch to Dynamic Ecology, are archived over at Dynamic Ecology. The URL’s are all the same, except just replace “oikosjournal” with “dynamicecology”. If you comment on old posts over at Dynamic Ecology, your comment will show up in our “recent comments” sidebar on the homepage, so that others will know there’s an active conversation going on and can join in.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


%d bloggers like this: