The NSF Pre-Proposal Process: Pt 1. Judging Preproposals

Before we start, this post refers to posts already written on this topic. To make sure no one gets lost, please follow the sequence of operations below:

Step 1: Do you know about the new pre-proposal process at NSF?

  • If Yes: Continue to Step 2.
  • If No: please read one of these posts and then proceed to Step 2.

Step 2: Have you read Jack William’s most excellent post (posted on Jacquelyn Gill’s most excellent blog) about a preproposal panelist’s perspective on the new process?

Step 3: Have you read Prof-like Substance’s post about his experience on a pre-proposal panel? (What? You haven’t read Prof-Like Substance’s blog before?! Go check him out.)

  • If Yes, continue to Step 4
  • If No, go to The Spandrel Shop and read Prof-like Substance’s post and return.

Step 4: Read our post! Like Jack and Prof-Like Substance, we also have experience with the new pre-proposal panels. The nuts and bolts of our experiences were similar to theirs (i.e., number of proposals read, assigning pre-proposals to one of three categories, etc). The main differences are really in our perceptions of the experience and the implications for the broader field. Please remember, there were a TON of pre-proposal panels this spring in both IOS and DEB. Differences from other panelists may reflect idiosyncratic differences in panels or differences in disciplines or just different takes on the same thing – because of NSF confidentiality rules, we can’t identify anything specific about our experiences – so don’t ask. And, speaking of rules: [start legalese] all opinions expressed within this post (including our comments, but not the comments of others) reflect only the aggregated opinions of Ethan & Morgan – henceforth referred to as Weecology – and do not represent official opinions by any entity other than Morgan & Ethan (even our daughter does not claim affiliation with our opinion…though to be honest, she’s two and she disagrees with everything we say anyway). [end legalese]

1) The Importance of Big Ideas. Our perspective on what made for a successful pre-proposal jives largely with Jack’s. The scope of the question being asked was really important. The panelists had to believe that the research would be a strong and important contribution to the field as a whole – not just to a specific system or taxon. Not only did the question being proposed need to be one that would have broad relevance to the program’s mission, it needed a logical framework for accomplishing that goal. In our experience, disconnects between what you propose to address and what you’re actually doing become glaringly obvious in 4 pages.

2) Judging Methods. The limited space for methods was tricky for both reviewers and writers. Sometimes the methods are just bad – if a design is flawed in 4 pages, it’ll still be flawed in 40 pages. The challenge was how to judge proposals where nothing was obviously wrong, but important details were missing. After reviewing full-proposals where you are trying to decide whether a proposal should be funded as is, this was a rough transition to make because all the details can’t reasonably be fit into 4 pages. While the panel was cognizant of this, it is still hard to jettison old habits. Sometimes proposals were nixed because of those missing details and sometimes not. We honestly don’t have a good feel for why, but it might reflect a complex algorithm involving: a) how cool the idea was, b) the abilities of the research team – i.e. is there a PI with demonstrated experience related to the unclear area, and c) just how important did those missing details really seem to a panelist.

3) Methods vs. Ideas. Our impression is that the 4-page format seems to alter the focus of the reviewer. In 15-pages, so much of the proposal is the methods – the details of questions, designs, data collection, analyses. It’s only natural for the reader to focus on what takes up most of the proposal. In contrast, the structure of the pre-proposal really shifts the focus of the reviewer to the idea. Discussions with our fellow panelists suggest we weren’t the only ones to perceive this though it’s important to note that not everyone feels this way – Prof-Like Substance’s post and comments flesh out an alternative to our experience.

4) Reviewers spend more time thinking about your proposal. This was an interesting and unexpected outcome of the short proposals. We both spent more time reading the literature to better understand the relevance of a pre-proposal for the field, looking up techniques, cited literature, etc. There was also a general feeling that panelists were more likely to reread pre-proposals. In our experience, most panelists felt like they spent about as much time reviewing each preproposal as they would a 15-pager, but more of this time was spent reading the literature and thinking about the proposal.

In general, like Jack, we came away with a positive feeling about the ability of the panel to assess the pre-proposals. A common refrain among panelists is that we were generally surprised how well assessing a 4-page proposal actually worked. However, the differences in how a 4-pager is evaluated could have some interesting implications for the type of science funded – something we will speculate on in our next blog post (yes, this is as close as an academic blog gets to a cliff-hanger….).

10 Comments on “The NSF Pre-Proposal Process: Pt 1. Judging Preproposals

  1. NSERC Discovery Grant proposals are 5 pages, for 5 years of work. Always have been. So maybe NSF should farm out pre-proposal reviews to the Canadians, since we’re pre-adapted to do it.😉

  2. Morgan and I both tried to bet each other that you would leave a comment about the Candadian funding system, but even at 100:1 neither of us was willing to bet against it.

  3. Pingback: The NSF Preproposal Process: Pt 2. A promising start. « Jabberwocky Ecology | Weecology's Blog

  4. Pingback: Mixed reports on the NSF preproposal process | Sociobiology

  5. But the REAL test of whether you and Morgan know me is whether you ALSO predicted that I would leave a self-deprecating response to your comment mocking me for my predictability! Knowing that you will have predicted this, I clearly can’t leave a self-deprecating response. Of course, since you know me, and know that I know you, you will have anticipated that I would anticipate your anticipation, and so will have predicted that I WON’T leave a self-deprecating response. So clearly I can’t NOT leave a self-deprecating response either! Further, I come from Canada, as everyone knows, and Canada is entirely peopled with modest, self-deprecating people, so clearly I must leave a self-deprecating response. And since you know I come from Canada, you will have predicted that I will leave a self-deprecating response, so clearly I cannot do so. [Morgan: “You’re just stalling now.”] You’d like to THINK that, wouldn’t you?! You and Morgan are professors, which means you must have studied, and in studying you must have learned that Man is predictable, and so–what in the world can that be?! [runs away when your backs are turned]

  6. Jeremy – that definitely wins the contest for the funniest comment we’ve ever received. Nicely done.

  7. Even funnier than the whole “old school internet”, my-homepage-is-still-on-Geocities-and-I-still-have-a-Prodigy-email-address thing?😉

  8. That one was good too, but I definitely think the laugh-o-meter as I was reading it to Morgan has higher on this one.

  9. Pingback: Give the NSF a piece of your mind « Nothing in biology makes sense!

  10. Pingback: Give the NSF a piece of your mind | The Molecular Ecologist

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: