Posted by: Jeremy Fox | February 22, 2012

Instrumental variables: the key to analyzing “natural experiments”?

Inferring causality is hard. Especially in a world where lots of factors, some of them unknown, causally affect the response variable of interest (and each other), and where there are causal feedbacks (mutual causation) between variables. It’s even harder when, for whatever reason, you can’t do a properly controlled, replicated experiment. What do you do then?

One standard answer is to rely on what Jared Diamond (and probably others) have called “natural experiments”.  The basic idea is as follows. If you think that variation in variable A causes variation in variable B, compare the level of B across systems that vary in their level of A. So instead of manipulating A yourself, you’re relying on the “manipulations” (variations) in the level of A that nature happens to provide.

Unfortunately, natural experiments are infamously unreliable, not just compared to “real” experiments but in an absolute sense. As my PhD supervisor Peter Morin liked to say, “The problem with natural experiments is that there’s no such thing as a natural control.” That is, systems that vary in their level of A often vary in lots of other ways as well, some of which probably also affect the level of B. You can of course try to address this by statistically controlling for the levels of those other variables, assuming you can identify them. And you can try to simply collect lots of data from a large range of systems in the hopes that surely some of the among-system variation in variable A will be independent of all confounding variables. And you can try to get rid of any causal feedbacks from B to A by praying to the god of your choice…

Or maybe there’s a better way. Economists have to deal with all the same challenges in inferring causality that ecologists do. If anything, economists have it even worse because doing relevant experiments often is harder in economics than it is in ecology. In response, economists have come up with an interesting and potentially-powerful approach to inferring causality from natural experiments, the method of “instrumental variables” (IV).

Here’s the basic idea (for details, click the link above, which goes to the very good Wikipedia page on IV). An instrumental variable, call it X, is a variable that causally affects B only via its effect on A, and that is not itself causally affected (directly or indirectly) by B or A. Economists summarize the latter assumption by saying that X is “exogenous”. So you can estimate the causal effect of A on B by using, not just any natural variation in A, but only that natural variation in A that can be attributed to natural variation in X. Changes in X are perturbations that propagate to B via only one causal path, that running from A to B, so variation in the instrumental variable X allows you to estimate that strength of that causal path. The approach can be generalized to multiple causal paths, as long as you have multiple instrumental variables.

One thing I find interesting about IV is that they highlight how “more data” is not always helpful. Tempting as it is to think that, if only you had enough data on A from enough different systems, you could reliably infer the causal effect of A on B, it’s not true. What you need is not more data on the variability of A, you need the right sort of data on the variability of A (namely, that generated by an instrumental variable). Indeed, more of the wrong sort of data on variability in A can actually be harmful to inferring the effect of A on B.

The nice thing about the IV method is that it doesn’t require you to know anything about the rest of the system, such as other variables that might affect B while also covarying with A. All you have to know (and this is the hard part) is that X is what economists call a “good instrument”–that it satisfies the assumptions that make it an instrumental variable.

Which may limit the applicability of IV in ecology. In economics, IV are often policy changes. For instance, an increase in cigarette taxes should affect health only via its effect on how much people smoke. So you can use changes in cigarette taxes to estimate the effect of smoking on health, thereby getting around the fact that lots of factors may affect both health and smoking, and that people’s health may affect their inclination to smoke. Weather events like droughts also tend to make good instruments in economics.

I’m unsure whether ecologists will often have good instruments available to them. Weather is exogenous to ecological systems as well as to economic systems. But the problem is that weather changes typically affect any variable of interest via multiple causal pathways. And many policy changes certainly have ecological as well as economic effects. But the problem with many policy changes affecting ecological variables is that they’re not exogenous–the policy changes are made in response to observed changes in the variable which the policy change is intended to affect. So if ecologists want to use policy changes as instrumental variables, they may want to focus on policies with unintended ecological consequences. And even there you still might have the problem of unintended consequences propagated via multiple causal paths.  But we won’t know if IV can be useful in ecology if we don’t try them out.

And if you do try out IV and get them to work, I hope you’ll submit the paper to Oikos. 😉

Advertisements

Responses

  1. Perhaps you are aware that there’s already an interesting ecological application of regression with IV, published a few years ago:

    Creel S. & Creel M (2009). Density dependence and climate effects in Rocky Mountain elk: an application of regression with instrumental variables for population time series with sampling error. J Anim. Ecol., 78, 1291-1297.

    This papers exploit the fact that an instrumental variable is correlated with the regressors, but not with the errors, to compare the performance of IV regression with respect to state space modeling; obviously, there’s a lot of subtleties in both the IV and state space modeling approaches that are no dealt with in the above paper (for example, the Kalman filter can indeed be easily extended to accommodate strongly non-linear processes and non-Gaussian errors although the authors intriguingly suggest this is not possible, and it would be interesting to compare IV vs. state space approaches in these settings); but it is definitely a good starting point!

    • I was not aware of this, thank you for the link. Indeed, the main reason I wrote the post is so that readers could alert me, and each other, to applications of this approach in ecology.

  2. Great write-up of IV! And from IV-regression, it’s just a hop, skip, and a jump to full Structural Equation Modeling with observed variables. What I enjoy about the comparison between the two is that for each, you need to think carefully about 1) causality and 2) the whole system, not just a single bivariate relationship. Something I would love to see more of in Ecology!

    • Thanks Jarrett. But don’t get too excited–there’s going to be a follow-up post on structural equation models at some point, that’s going to be rather more skeptical. Maybe I’ll try to time it to coincide with the SEM course that I hear you teach, so that you can assign all your students to comment on my post. 😉

      • I look forward to it! FYI, I can’t remember if I have mentioned this before, but, I just started in on reading Judea Pearl’s Causality. You may find it of interest. In particular, it covers a few issues that SEM folk have been not as solid on until now.

      • I’d say if you have reason to be skeptical of SEM approaches you really should be skeptical of IV approaches*. I say this because IV approaches make all the assumptions of SEM, and a boatload more. Essentially, from what I understand, IV approaches heavily rely on linearity of the relationship between the instrument and the predictor that your interested in. In addition to that, if the strength of the instrument is weak (which a lot of instruments are), then even a very small relationship between the suggested IV and the outcome variable can lead to wildly inaccurate estimates of causal effects. The final nail in the coffin for me for IV approaches is that there is no way to actually test for that second problem; If it turns out your instrument actually does influence your final outcome through a path not through the treatment, you won’t be able to tell.

        *I am not an expert on IV methods, but I’ve got this from people who have far more experience with IV, and have seen them massively misused. The one post I was trying to find was by Daniel Davies on D-Square Digest, but I see he’s set his blog to a friend-only view, so I’ll link to Cosma Shalizi’s a href=”http://www.stat.cmu.edu/~cshalizi/402/lectures/23-causal-estimation/lecture-23.pdf”> course notes talking about this issue .

      • Thanks Eric. As I said in response to another comment, it was my hope that readers who know more than I do would chime in. I agree that the assumptions required to make IV analysis work are pretty strong, and in particular having an instrument with a strong effect is crucial.

        Whether IV or SEM merits more skepticism, I’m not sure. It’s not just a matter of what assumptions each approach makes, but our ability to assess whether those assumptions are satisfied. But it’s not an issue I’ve thought about a lot yet.

        I suspect IV proponents might respond to these concerns by admitting to them, but would also argue that the naive comparative approach–just ask how variable B varies as a function of variable A–is even worse.

        Frankly, neither IV nor SEM is an approach that I myself would be inclined to try, for reasons I’ll be elaborating on in a future post (briefly, my biggest concerns are actually more philosophical than statistical). But that’s mostly because I choose my questions, and the systems in which I address those questions, so that I can use much more powerful and reliable approaches. In writing posts on IV and (soon) SEM, I’m consciously taking a break from arguing for my own approach to science and trying to put myself into somebody else’s shoes.

  3. Interesting, Eric. And this is where SEM does come in – the test of mediation is fairly straightforward and can easily test that second assumption. And/or a good examination of semi-partial correlations. As for linearity, one can conduct an IV analysis with nonlinear responses. The underlying idea is the same, but the statistical machinery is just a wee bit different. Same for SEM, although my current understanding is that this is easier to deal with in IV analysis. And, honestly, if you’re assuming linearity everywhere you look, that’s going to be a problem in any analysis. Linearity is something we all too often blithely assume, even when the problem of interest cries out that linearity is the incorrect assumption.

    Thanks for the fantastic link!

    • One of the things I like about Judea Pearl and collaborators approach to causal modelling is that it not only lets us make statements about what causal relationships the data can support, it also lets us say that “given a set of predictors, no amount of evidence will allow us to distinguish between some sets of causal models”. I think that’s my biggest problem with IV… it seems very easy to disregard alternate supported models, since it doesn’t force you to write out your causal model explicitly. This might very well be my personal prejudices speaking though. 🙂

  4. This sort of discussion is why the Oikos blog is my first biology-blog stop of the day.

    • Thanks!

  5. Being a relative newbie to SEMs, I’m glad to see I’m not the only one who was thinking “this all sounds a bit like path analysis” as I was reading the post.

    The introductory stuff I’ve read about SEMs in Ecology has been very clear about its assumptions and limitations. However, I’ve also seen the method thoroughly abused in some manuscripts – e.g., basically as a data trawling tool with too few a priori hypotheses for the proposed directional, causal links.

    Looking forward to your actual SEM post, Jeremy!

    • Yes, SEMs are just a (fairly modest) generalization of path analysis.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: