Category Archives: science

It’s that time of year again when we’re all busy working on preproposals for the National Science Foundation, and just like last year it’s more difficult than you would think to track down the official guidelines using Google. So, for all of you who don’t have a minute to spare, here they are:

• DEB
• IOS

Also, remember that Biographical Sketches are different than for full proposals:

Biographical Sketches (2-page limit for each) should be included for each person listed on the Personnel page. It should include the individual’s expertise as related to the proposed research, professional preparation, professional appointments, five relevant publications, five additional publications, and up to five synergistic activities. Advisors, advisees, and collaborators should not be listed on this document, but in a separate table (see below).

And that there is a big stack of things that should not be included at this stage in the process:

Budget, Budget Justification, Facilities, Equipment and Other Resources, Current and Pending Support, Letters of Collaboration, Data Management Plan, Postdoctoral Mentoring Plan, RUI Impact Statement, Certification of RUI Eligibility, or any other Supplementary Documents.

Good luck!

UPDATE: Included separate links for DEB and IOS.

Over the past year, you can’t get two scientists together who submit to the BIO Directorate at NSF without the conversation drifting to the radical changes in the proposal process. If you have no idea what I’m talking about, I’ve added some links at the bottom of the post for you to check out.  For everyone else, suffice it to say that there has been immense speculation about the possible impacts of these changes on the scientific process.  Well, over the winter break, DEB released its data to date (IOS did this a little earlier and comparisons between IOS and DEB are discussed here). So let’s see what happened!

Table 1. Basic Stats on Funding Rates

^ numbers I’ve estimated given the statistics provided by NSF *value complicated by uncertain fund rate

You’ll notice some of the items in the table are starred.  That’s because things get a little…complicated in the full proposal funding data. When DEB released the data, funding decisions weren’t finalized, so they only had an estimate of funding rates. Also some full proposals didn’t need to submit preproposals to DEB (e.g. CAREER, OPUS, RCN, co-reviewed proposals from other divisions), so the starred items have two possible sources of fuzziness: non-preproposal proposals and uncertain fund rates. The NSF info doesn’t make some of this transparent. For example, NSF reports a full proposal  ‘success rate’ of 35% of the 82 full proposals submitted by early career investigators through the pre proposal process.  However, the accompanying table (see below) on success rates over the past 5 years shows the 2012 data as 16% out of 181 proposals. I assume the numbers don’t match due to the proposals submitted outside the preproposal process (i.e. CAREERs). It’s also unclear to me whether ‘success rate’ is ‘recommended for funding’ or actual funding.

Table 2. Statistics for Early Career Investigators over past 5 years:

Interesting Stats to Chew on:

1)      Preproposal Funding rates: Let’s assume funding rates for full proposals did not differ between CAREERS, RCNs, or invited preproposals (an assumption that is probably wrong). If that’s the case, then in table 1 I estimated the funding rate of the preproposals at 5.1% (i.e. 5.1% of preproposals eventually got funded as a full proposal). It’s important to note that 5.1% is probably wrong, but how wrong is unclear as it hinges on how different the fund rate is for CAREERS, etc. My guess is that the preproposal fund rate is a little higher because things like CAREERs have a lower fund rate and thus bring down the overall average. However, I’d be surprised if the difference takes preproposals above 10%.

2)      Quality of Full Proposals: 68% of proposals made a funding category (i.e. not allocated to the ‘Do Not Fund’ category). I’d be interested in seeing the data from previous years, but 68% seems high from my limited experience.

3)      Early Career vs overall funding rates: Focusing on the preproposal process data only (i.e. not Table 2 data), my interpretation is that the young people fared well through the preproposal process but took a serious hit in the full proposal process (35% of young investigators recommended for funding vs. an overall recommendation rate of 68%). As a disclaimer, the preproposal data is post-portfolio balancing while the full proposal data seems to be pre-portfolio balancing, so it’s possible that the preproposal panels were equally hard on the youngsters but that the Program Directors corrected for it.

4)      Early Career funding rates: I’ve been studiously ignoring Table 2 (as you might have noticed). The truth is even if funding rates were equal between established and early career scientists, 5.1% success rates (or even 10%) mean that anyone who needs a grant to have a job (whether to get tenure or because they are on soft money) or to keep research going (labs that needs techs or uninterrupted data collection) is in a tough spot right now. Additional biases in funding rates clearly exasperate the situation for our young scientists and this is something we should all be aware of when our young colleagues ask for our help or come up for tenure.

Summary:

There’s enough nebulous stuff here that I’m going to hold off on any grandiose statements until NSF releases its full report in early 2013. But the following are things that the data made me start thinking about:

1)      There’s nothing in the NSF data thus far that changes my opinion about the preproposal revolution: until NSF has more money to fund science, 5.1% funding rates are the real enemy of science. What NSF is doing is more akin to shuffling the deck chairs than to blowing a hole in the hull.

2)      The preproposals per se don’t seem to be filtering out the young people. It’s the full proposal process that seems to be the big hurdle to funding. I suspect that is not a novel result of the new system, but has been true all along. The interesting insight that the preproposal data might suggest is that the lower funding rates have nothing to do with the ideas of the young scientists but more to do about either the methodologies or with how those methodologies are being communicated.

3)      There’s clearly two things that will help our younger scientists: a) increasing funding rates overall (not a solution in NSF’s power) and b) figuring out why the bias in the full proposal rates exists and figuring out how to fix it (something we can all try to work on). Assuming that the lower recommendation rate for full proposals of young career scientists is due to their proposal and not a bias against young scientists (i.e. lower name recognition), then this might be a legitimate argument for how the new system could hurt young people: young people may need more submissions of full proposals (and more panel feedback) before managing to get a proposal recommended for funding.

Additional Links about the Changes at NSF:

Prof Like Substance: NSF BIO decides to screw new investigators, What I learned at an NSF BIO preproposal panel

Contemplative Mammoth: Inside NSF-DEB’s New Pre-Proposals: A Panelist’s Perspective

Jabberwocky Ecology: Changes in NSF process for submissions to DEB and IOS*, NSF Proposal Changes – Follow-up

As some may be aware, ESA has launched a new journal: Ecosphere. ESA describes Ecosphere as “… the newest addition to the ESA family of journals, is an online-only, open-access alternative with a scope as broad as the science of ecology itself. “

The description is vague  - is it a new incarnation of Ecology? Or is it an ecologically focused equivalent of PLoS One? I’m not the only one who is confused, as illustrated by a comment by Jeremy Fox from Dynamic Ecology.  I recently had an interesting experience with Ecosphere that both clarified Ecosphere for me, and also what I think its potential is. I’ve been meaning to post on this for a while, but seeing that Jeremy and I are having similar thoughts finally encouraged me to get off my butt and write the post.

What is Ecosphere? If you’ve ever reviewed a paper, you know that part of your decision is based on the paper and part is based on the journal itself. For journals like Ecology, Am Nat, and Ecology Letters, you are judging both the rigor of the science and its potential impact. For PLoS One, the potential impact is not supposed to be part of the review decision, just whether the science is sound. I recently reviewed a paper for Ecosphere that was sound but not broadly interesting. What to recommend? The editor made it clear to me that Ecosphere is an ecological PLoS One and that what mattered was the scientific soundness. Often I hear these components of the review process conflated – but interesting and rigorous are not actually the same thing. So when Ecosphere talks about maintaining the same ‘rigorous peer-review standards’ as Ecology, it means that it is focusing on the soundness, not the interest component.

Future of Ecosphere? I have no insights into whether Ecosphere is performing as ESA had hoped but I think Jeremy’s view on Ecosphere is probably common. I suspect public relations outreach to clarify the role of Ecosphere in the journal pantheon would help. I also suspect that it could greatly benefit from an incentive to publish there. But what is an easy incentive that doesn’t undercut the economic benefits of Ecosphere for ESA? Jeremy nails it in his comment and it’s the same thought I had after I reviewed for them: make it easy to have solid but rejected papers from Ecology be rapidly accepted in Ecosphere. You see, the paper I reviewed for Ecosphere was also a paper I had just reviewed for Ecology and recommended rejecting based not on any issues with the science, but based on its importance. How much easier would it be if there was a button on the Ecology reviewer form that says “Is this paper suitable for Ecosphere? If yes, is it acceptable as is, with minor revision, with major revision?” Essentially, reviewers can review for both journals at the same time. Then if a paper is rejected from Ecology but recommended for acceptance at Ecosphere, the author can get a letter saying – so sorry about your Ecology rejection, but (if you would like) congratulations on your acceptance to Ecosphere!

I think this is a good idea for Ecosphere because it provides a mechanism whereby really good papers can still end up inthat journal, thus helping improve its impact (used generically, though I suppose it might also help its impact factor). Let’s be honest, when only 3 people are judging whether a paper will be ‘interesting’ to the broader field, bad things can happen to good papers. The direct tie between Ecology and Ecosphere increases the probability of getting those papers into Ecosphere because a guaranteed acceptance can be hard to turn down. If a paper has been ‘making the circuit’, it can be tempting to just take that acceptance, even if it’s not the “quality” of journal you might have been hoping for.

I also think this is a good thing for science. Perhaps you’re review process experience is always smooth sailing, but many of us are spending a lot of time revising and resubmitting papers that are technically sound but that reviewers dislike because they don’t like the topic or are uncomfortable with the take home message, or (my favorite) this isn’t the paper that they would have written themselves. Science slows down when sound science is rejected based on ‘interest’ and not on technical reasons, because papers may take an additional year or more to be published as they are repeatedly submitted to multiple journals. The big journals have the right to judge on interest, and there is some value to this in that they can help serve as filters for the deluge of new papers, but I think having quick avenues for publication of sound science is good for us all. Tying the big ESA journals to Ecosphere provides the benefits of both – the time cost of taking a shot at Ecology is minimized because if judged sound it would still get an acceptance into Ecosphere, even if rejected from Ecology because it ‘wasn’t interesting enough’.

Finally, it’s good for ESA to have Ecosphere capture more of the ecological literature through its open access model. Right now, if rejected from Ecology, the next step for most papers is probably not Ecosphere but some Elsevier or Wiley-Blackwell journal (or maybe PLoS One). Each scientifically sound paper that does not end up at Ecosphere is $1,250 that ESA doesn’t get (based on page charge cost for members). The more papers that end up at Ecosphere, the more $$ goes to ESA, which can then use that money to do all the great things it does both for its members and for ecology in general.

UPDATE: Click here to read discussion re: Ecosphere on ECOLOG back in March (thanks to @JJVenky for pointing this out)