Posted by: Jeremy Fox | August 18, 2011

Changes to NSF submission process: another reason to move to Canada?

NSF has made radical changes to their submission process, in order to cut down on the number of full proposals requiring external review. For summary and discussion of the changes see Jabberwocky Ecology and The Spandrel Shop.

Early response to the changes seems to be largely negative, but of course we won’t know the full consequences for several years. But that won’t stop me from asking again: At what point does the Canadian NSERC Discovery Grant system start to look good to US investigators tired of having their careers depend on a lottery ticket? See here and here for previous Oikos blog posts on the strengths of the Canadian system relative to the US system. And see here and here for discussion of an alternative way to ensure that there are enough reviewers for all the grant applications NSF receives (short version: oblige submitters to review).

Advertisements

Responses

  1. I would never compare the NSF system to a lottery system. Well, let me take that back. NSF is a lottery system if you write a mediocre grant proposal (which most proposals I have read are). If you write a clear proposal that answers important questions and moves your field forward, using appropriate and doable methods, chances are you will get funded – or at the very least you will get good ideas on how to revise the grant to get it funded the next time around. If you spit out 6 grant proposals a year all of which you wrote in a few days, that ask a bunch of questions with no clear focus, then yes – a lottery.

    The hardest part, for any funding agency, is to get great science funded – not fund the most people. Our tax money should fund the BEST science. For that only great proposals should be funded. Mediocre proposals should be relegated to a lottery system or the trash can.

    I know it may be harsh, but I come from this as someone who has reviewed many NSF proposals, most of which have been pretty poorly devised, designed and written. In many cases, it is not even worth the reviewer’s (read – my) time. I applaud NSF for reducing the number of proposals sent out to review – saving the best of the best – the ones most likely to push the field forward, and allowing other researchers to spend their time doing good science rather than reviewing poor science. NSF has the most FAIR system around – the great science is funded. That is how it should be.

    • To be clear, I’m not arguing that the science NSF does fund is mediocre, or that all or even a majority of the proposals they receive are of essentially equal quality. What I do question is their ability to reliably distinguish great from good, which is problematic given that they receive a large volume of good stuff. Is there really a big difference between the X% of applications they fund (where X is some small number), and the X% immediately below the cutoff, or even the X% below that? I doubt it, although I admit I have no hard data to back it up.

      Nor am I arguing in favor of funding the most people. Obviously that’s an inappropriate goal. What the Canadian system does is try to fund excellent *long-term research programs* (as opposed to individual, short-term projects, the way NSF does) Please see previous posts for more discussion of this. I do think there is room for debate on whether NSERC funds an optimal percentage of applicants (although I also think it’s possible that NSERC’s current renewal rates actually are close to optimal, given NSERC’s goals). As I asked in a previous post, would NSERC ultimately be funding better science if it gave more money to fewer research programs, a la HHMI? Maybe, although of course at some point you run into an NSF-type problem at the level of research programs rather than projects. Are the best X% of people, where X is a small number, really so much better than the next X% that it’s (a) clear who the best X% are and (b) clear that the best X% should get all the money?

      It’s also worth noting that, while renewal rates at NSERC are high, there is a lot of variance in how much money you get. If you have a really productive 5 years, and propose a really good plan for the next 5 years, your funding level can really jump. Conversely, if you don’t propose a really good plan (perhaps hoping to coast on your track record), you may not get zeroed but you will get cut substantially. This approach (which is rather new, previously there used to be a lot of inertia in funding levels) seems to me a nice way to fund good people sustainably for the long term (thereby allowing them to take risks they wouldn’t otherwise take), while also holding them to account and ensuring deadwood gets weeded out.

      I entirely agree that the “best” science should be funded, but what’s at issue here is what the “best” is and how to fund it, given that we can never know for certain how any proposed research will turn out (e.g., high-risk, high-reward science vs. low-risk, low-reward science). Again, see previous posts for discussion of this.

      Not clear why NSF has the most “fair” system around–can you clarify? I’m not sure why a strategic decision to fund people rather than projects, or to aim to provide long-term rather than short-term support, would reduce fairness.

      BTW, it’s not just Canadians who say that NSERC’s model works; an independent international review panel recently said so (happy to dig up a link if anyone’s interested). One could also look at comparative international data on scientific productivity per dollar spent. I should perhaps dig up some numbers and post them at some point.

  2. The issue with the Canadian system is that it CAN’T work in the US without massive changes to how universities operate. US universities use the overhead from grants as a major portion of their budget, whereas this is not the case in Canada. If NSF started funding at $30-50K/year, not only would labs who have to support students (+ tuition) off that get little to no work done, but universities would lose a huge amount in revenue (not to mention that the Uni would take 1/3 of that money anyway).

    On the surface you can just say that the universities will have to compensate in some way, but it also means that most larger universities will focus their hiring on NIH-fundable PIs, even more than they already do. Basic science in the US would then suffer BOTH from being underfunded and phased out of university hiring priorities. The end result would likely be a situation where smaller universities do basic science and larger ones do NIH stuff, which is not an ideal situation.

    Changes at the funding source would have to be matched with changes at the university level, which are unlikely if NIH is unaffected.

    • Very good points. My thoughts on this issue haven’t incorporated these broader issues, which absolutely would need consideration if NSF were ever to seriously consider moving to an NSERC-type system.

      A bit of context, for readers who may be unfamiliar with the details of the Canadian system (I know you know this already, Prof-like Substance). Overhead in the Canadian system isn’t budgeted as part of individual grants. Instead, NSERC basically totals up the value of all the NSERC funding a university receives in a given year, and then pays that university 20% (I think; the number may be higher but it’s definitely not anything like 50%) of that total value as overhead. So there is overhead in the Canadian system, but it’s not something individual PIs need to incorporate into their budgets, and the rate is fixed nationally. Canadian academic positions are all 12-month appointments, so there’s no summer salary for PIs in NSERC Discovery Grants. And Canadian grad students are typically funded either as TAs, or by scholarships, assistantships, or fellowships from the dept., province, or NSERC, so most PIs only pay summer salary for their grad students. Grad student tuition is typically paid by other sources, not the PI’s grant funds or the student (well, at Calgary the students pay a small fraction of the tuition themselves). So from the perspective of a PI, a 40K/year NSERC Discovery Grant (a roughly average, and modal, value) actually isn’t all *that* small compared to an NSF grant, since so much of an NSF grant goes to overhead and PI summer salary.

      But yes, from the university’s perspective an NSERC-type system would be a big financial change from the NSF system, and no, it’s not obvious how US universities would compensate. Which isn’t to say that there might not be hybrid systems that could work. For instance, presumably the choice of overhead rate is at least somewhat independent of the choice of mean grant size (and thus success rate), both of which are presumably at least somewhat independent of whether one wants to fund people (=research programs) or short-term projects. I don’t pretend to have all the answers here, but I do think there are issues here that are worth discussing. When it comes to funding agency policies, I have a sense that folks who work in a certain system get so used to it (or at least are so focused on trying to succeed within that system) that they rarely step back and think about alternatives.

  3. […] Low funding rates and the recent application changes have people once again invoking the tired "Why can't we do it like Canada?" Look, I love our neighbors to the North as much as the next person, but there are things you can […]


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: