The NSF Proposal Revolution: The DEB Data
Over the past year, you can’t get two scientists together who submit to the BIO Directorate at NSF without the conversation drifting to the radical changes in the proposal process. If you have no idea what I’m talking about, I’ve added some links at the bottom of the post for you to check out. For everyone else, suffice it to say that there has been immense speculation about the possible impacts of these changes on the scientific process. Well, over the winter break, DEB released its data to date (IOS did this a little earlier and comparisons between IOS and DEB are discussed here). So let’s see what happened!
Table 1. Basic Stats on Funding Rates
|Preproposal Invites for Full Submission||380|
|Full proposals recommended for funding||259|
|*^Number of proposals to be funded||83.6|
|Preproposal Invitation Rate||23.4%|
|New Investigator Preproposal Invitation Rate||20.4%|
|Full Proposal Panel Recommendation Rate||68%|
|Early Career Investigator Full Proposal Panel Recommendation Rate||35%|
|*Anticipated Overall Fund Rate on Full Proposal Panel||22%|
|*^Overall fund rate from preproposal pool||5.1%|
^ numbers I’ve estimated given the statistics provided by NSF *value complicated by uncertain fund rate
You’ll notice some of the items in the table are starred. That’s because things get a little…complicated in the full proposal funding data. When DEB released the data, funding decisions weren’t finalized, so they only had an estimate of funding rates. Also some full proposals didn’t need to submit preproposals to DEB (e.g. CAREER, OPUS, RCN, co-reviewed proposals from other divisions), so the starred items have two possible sources of fuzziness: non-preproposal proposals and uncertain fund rates. The NSF info doesn’t make some of this transparent. For example, NSF reports a full proposal ‘success rate’ of 35% of the 82 full proposals submitted by early career investigators through the pre proposal process. However, the accompanying table (see below) on success rates over the past 5 years shows the 2012 data as 16% out of 181 proposals. I assume the numbers don’t match due to the proposals submitted outside the preproposal process (i.e. CAREERs). It’s also unclear to me whether ‘success rate’ is ‘recommended for funding’ or actual funding.
Table 2. Statistics for Early Career Investigators over past 5 years:
|Fiscal Year||Success Rate||# proposals||% total submissions|
Interesting Stats to Chew on:
1) Preproposal Funding rates: Let’s assume funding rates for full proposals did not differ between CAREERS, RCNs, or invited preproposals (an assumption that is probably wrong). If that’s the case, then in table 1 I estimated the funding rate of the preproposals at 5.1% (i.e. 5.1% of preproposals eventually got funded as a full proposal). It’s important to note that 5.1% is probably wrong, but how wrong is unclear as it hinges on how different the fund rate is for CAREERS, etc. My guess is that the preproposal fund rate is a little higher because things like CAREERs have a lower fund rate and thus bring down the overall average. However, I’d be surprised if the difference takes preproposals above 10%.
2) Quality of Full Proposals: 68% of proposals made a funding category (i.e. not allocated to the ‘Do Not Fund’ category). I’d be interested in seeing the data from previous years, but 68% seems high from my limited experience.
3) Early Career vs overall funding rates: Focusing on the preproposal process data only (i.e. not Table 2 data), my interpretation is that the young people fared well through the preproposal process but took a serious hit in the full proposal process (35% of young investigators recommended for funding vs. an overall recommendation rate of 68%). As a disclaimer, the preproposal data is post-portfolio balancing while the full proposal data seems to be pre-portfolio balancing, so it’s possible that the preproposal panels were equally hard on the youngsters but that the Program Directors corrected for it.
4) Early Career funding rates: I’ve been studiously ignoring Table 2 (as you might have noticed). The truth is even if funding rates were equal between established and early career scientists, 5.1% success rates (or even 10%) mean that anyone who needs a grant to have a job (whether to get tenure or because they are on soft money) or to keep research going (labs that needs techs or uninterrupted data collection) is in a tough spot right now. Additional biases in funding rates clearly exasperate the situation for our young scientists and this is something we should all be aware of when our young colleagues ask for our help or come up for tenure.
There’s enough nebulous stuff here that I’m going to hold off on any grandiose statements until NSF releases its full report in early 2013. But the following are things that the data made me start thinking about:
1) There’s nothing in the NSF data thus far that changes my opinion about the preproposal revolution: until NSF has more money to fund science, 5.1% funding rates are the real enemy of science. What NSF is doing is more akin to shuffling the deck chairs than to blowing a hole in the hull.
2) The preproposals per se don’t seem to be filtering out the young people. It’s the full proposal process that seems to be the big hurdle to funding. I suspect that is not a novel result of the new system, but has been true all along. The interesting insight that the preproposal data might suggest is that the lower funding rates have nothing to do with the ideas of the young scientists but more to do about either the methodologies or with how those methodologies are being communicated.
3) There’s clearly two things that will help our younger scientists: a) increasing funding rates overall (not a solution in NSF’s power) and b) figuring out why the bias in the full proposal rates exists and figuring out how to fix it (something we can all try to work on). Assuming that the lower recommendation rate for full proposals of young career scientists is due to their proposal and not a bias against young scientists (i.e. lower name recognition), then this might be a legitimate argument for how the new system could hurt young people: young people may need more submissions of full proposals (and more panel feedback) before managing to get a proposal recommended for funding.
Additional Links about the Changes at NSF:
Contemplative Mammoth: Inside NSF-DEB’s New Pre-Proposals: A Panelist’s Perspective
Jabberwocky Ecology: Changes in NSF process for submissions to DEB and IOS*, NSF Proposal Changes – Follow-up