Jabberwocky Ecology

Definitely not the meaning of “non-significant”

Andrew Gelman over at Statistical Modeling, Causal Inference, and Social Science posted a hilariously awful story about the interpretation of a non-significant result he saw at a recent talk (I particularly love the Grrrrrrr at the end).

I’m always yammering on about the difference between significant and non-significant, etc. But the other day I heard a talk where somebody made an even more basic error: He showed a pattern that was not statistically significantly different from zero and he said it was zero. I raised my hand and said something like: It’s not _really_ zero, right? The data you show are consistent with zero but they’re consistent with all sorts of other patterns too. He replied, no, it really is zero: look at the confidence interval.

Grrrrrrr.

This and related misinterpretations crop up all the time in ecology. I’ve witnessed some particularly problematic cases where the scientist is interested in attempting to determine if some data are consistent with a theoretically predicted parameter and the confidence intervals are relatively wide. The CIs sometimes contain both 0 and the theoretically predicted value and yet it is concluded that the data are not consistent with the model because the parameter is “not significant”. This is obviously problematic given that the goal of the analysis in the first place had nothing to do with demonstrating a difference from 0.

Macroecologists to rule the world… for a while

I saw the first two paragraphs of this quote from an interview of Hal Varian by The McKinsey Quarterly over at Statistical Modeling, Causal Inference, and Social Science. The whole interview is great, but this section suggests the promise of macroecology (and… OK… statistics as well) for training people in the broadly important area of acquiring, manipulating, analyzing and understanding large quantities of data:

I keep saying the sexy job in the next ten years will be statisticians. People think I’m joking, but who would’ve guessed that computer engineers would’ve been the sexy job of the 1990s? The ability to take data—to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it—that’s going to be a hugely important skill in the next decades, not only at the professional level but even at the educational level for elementary school kids, for high school kids, for college kids. Because now we really do have essentially free and ubiquitous data. So the complimentary scarce factor is the ability to understand that data and extract value from it.

I think statisticians are part of it, but it’s just a part. You also want to be able to visualize the data, communicate the data, and utilize it effectively. But I do think those skills—of being able to access, understand, and communicate the insights you get from data analysis—are going to be extremely important. Managers need to be able to access and understand the data themselves.

The third paragraph (the part not discussed at SMCISS) speaks to a challenge that we are really still grappling with in ecology:

You always have this problem of being surrounded by “yes men” and people who want to predigest everything for you. In the old organization, you had to have this whole army of people digesting information to be able to feed it to the decision maker at the top. But that’s not the way it works anymore: the information can be available across the ranks, to everyone in the organization. And what you need to ensure is that people have access to the data they need to make their day-to-day decisions. And this can be done much more easily than it could be done in the past. And it really empowers the knowledge workers to work more effectively.

This challenge is the easy access to data by those doing the science, those evaluating the science, and those attempting to apply science to address major socioecological issues. Data is increasingly available, but it is in a wide variety of formats, hosted by different providers, and much of it comes with strings attached. This, combined with a general lack of appropriate technical skills among practicing ecologists, puts us at a disadvantage for tackling important problems and doing so in the most general and useful ways. To be sure, these problems are being addressed by a variety of groups, but we still have a lot of work to do. Perhaps the promise of ruling the world (and the fear of missing our 10 year window) will help keep us moving forward.

Making choices when writing

Yesterday’s post made me think of this great scene between Michael Douglas and Katie Holmes’ characters in the movie Wonder Boys:

Hannah Green: Grady, you know how in class you’re always telling us that writers make choices?
Grady Tripp: Yeah.
Hannah Green: And even though you’re book is really beautiful, I mean, amazingly beautiful, it’s… it’s at times… it’s… very detailed. You know, with the genealogies of everyone’s horses, and the dental records, and so on. And… I could be wrong, but it sort of reads in places like you didn’t make any choices. At all.

A catch 22 of length limits in publication

It is increasingly common for journals to employ fairly strict length limits on submissions. I’m actually a big fan of this. I feel that the most important points of most manuscripts will fit into 6-8 published pages and details that only a small fraction of an already small readership will be interested in can easily be placed in online supplements. Keeping papers short forces authors to write succinctly and makes it easier to manage the overwhelming amount of literature being published every year.

That said, I’ve recently been running into a catch 22 with respect to these limits. The issue stems from the fact that in order to make a full blown research paper short, you have to make some tough decisions. In the course of conducting your research you’ve almost certainly explored more ideas and checked out more details than will fit in 6-8 printed pages, so you can’t include it all. Making these decisions is a good thing. It forces the authors to focus on what’s really critical to the paper. This can be a difficult task to learn but it is important for effective scientific communication in the modern era.

The catch 22 occurs because reviewers almost always request that additional information be added to manuscripts. Sometimes this information is important, but often it is not necessary to the manuscript and includes such things as analyses that are largely tangential, more discussion of the reviewers area of research, more citations, etc. The reviewers almost never consider the length issue (which is OK) so they don’t typically recommend anything to remove, and the the AE sends the paper back with a form letter asking the authors to incorporate the reviewer comments. The authors do this because they want the paper to be accepted and send it back but… oops… now the paper is too long. So, for the paper to be submitted to and published in the journal it has to be shorter than the limit, but in order for it to be accepted it has to be longer than the limit.

In a recent experience at Ecology satisfying the reviewer comments caused the manuscript to go from being the length of a Report to being the length of an Article. This would have meant that that the article was no longer open access and would have cost us an additional $1000 for the color figures (color is free for Reports). Fortunately the Managing Editor and I got it worked out with some last minute trimming (David Baldwin is a really nice guy btw). In an even more classic catch 22 (with a journal whose name I will leave out to protect the guilty) we were asked by a reviewer to move material from the supplement to the body of the ms. We made this change, which caused us to go over the figure limit, and then the journal wouldn’t even allow us to resubmit. They unsubmitted our ms and told us to cut down to the appropriate number of figures. It was awesome.

So if we’re going to keep these strict length limits in place for revisions as well as original submissions, and I think we should, then the burden falls on the AEs to consider the length limits of the submission when requesting revisions. If the AEs could provide guidence to the authors when a revision is requested as to whether they deem particular additions of sufficient importance to cut something else, and if so, provide some guidence as to what should be cut, I think it would take care of this problem. But, that is probably a lot to ask, so until then I guess I’ll just keep combining two figures into a single two-panel figure and tightening my writing (which always seems to have space to give).

The benefits of having a lab wiki

A few months ago Mike Kaspari over at Getting Things Done in Academia emailed me to ask “how do you use your Lab Wiki? Has it been worth it?”. I sent back much of what is included below and Mike asked if he could use some of my thoughts for a post on wikis. I of course said yes, but since GTDA has been silent for quite a while now I figured I’d go ahead and post on it myself.

First, for those of you whose only exposure to wikis is wikipedia, a wiki is basically just a group of web pages that can be edited online (i.e., inside a browser). These pages might be editable by anyone (like wikipedia) or only by users with permission. They also might be viewable by anyone, or only by a small set of people. For a lab wiki I’d recommend that most of the wiki only be accessible and editable by members of the lab with some areas accessible by specific external collaborators.

So, why have a lab wiki?

1. Collaboration – The wiki provides a central place for idea development, conversation, posting of results and interpretation, etc. It is particularly valuable for remote collaboration in that it switches the paradigm from one or a few people taking the lead and dealing with keeping everyone else informed to a system where it is the participants responsibility to keep an eye on what is going on and everyone decides their own level of involvement.

2. Idea generation, motivation and storage – Have you ever had a cool idea but didn’t have time to think about it at the moment. I jot down 2 sentences on the research ideas page of my wiki. The idea of this page is to put interesting ideas out in front of larger audiences and see if anyone is interesting in working on them. This has already lead progress being made on one project that I wouldn’t have gotten around to otherwise. At the very least the idea is there for later if I want to go back to it myself.

3. Progress in small steps – If I have an idea about something, even a small one, I just login and jot it down on that projects page. If I have 20 minutes of spare time I can do one simple analysis, or learn how to do a single step in an statistical or computational problem and post the answer. A small step might allow one of my collaborators to make their next step on a project, or seeing progress may motivate them to do something themselves. In general I find that this medium encourages and allows progress to be accomplished with more smaller steps, thus lowering the activation energy for some projects.

4. Information aggregation – When someone in the lab finds a useful tool, good advice on some academic task (e.g., writing papers), etc. this information can be stored on the wiki. This sort of thing can be done via email or via blog, but I think that permanent archiving in well structured pages combined with the inherently improved potential for participation by members of the lab makes the wiki superior for this task. For example, a third year student who is starting to write papers in 2012 can just go to the wiki page on advice for writing and get going. If I’d just emailed the relevent link to my current students then it won’t reach future students and it won’t even reach many current students at a time when it’s useful to them.

5. Lab openness – Most of the pages on our wiki are open to anyone in the lab (though we do have the ability to restrict access and do for outside guests and in a few other circumstances). My hope is that this will help with a variety of minor issues I’ve observed over the years (not in my labs, but in labs I’ve been a member of and/or known people in) including territorial disputes resulting from a lack of communication and the classic graduate student mistake of thinking that because it took your advisor three weeks to read your paper that it’s because they’re at home playing Xbox all evening. I don’t know if this will work but my feeling is that openness is in general a good solution to a lot of problems and the wiki makes such openness happen almost by default and provides a historical record for cases where disputes still occur.

So, that’s my plug for having a lab wiki. There are a variety of ways to setup a wiki, with the easiest being to use one of several online hosts. We use PBwiki, which I can’t say enough great things about, but there are a variety of options. Many of these wiki sites have full WYSIWYG editors so editing a page is just as easy as editing a Word document. So, go find a wiki provider, set up a free wiki, and try it out. Maybe you’ll find it as essential a tool for a modern academic lab as we do.

Why senior authorship is bad for ecology (and probably science in general)

I’ve been giving a fair bit of thought recently to the concept of “senior authorship”. Senior authorship is the practice whereby the last position on an author line is occupied by the leader of the lab in which the project was conducted (i.e., the P.I., the advisor, whatever terminology you prefer). Being the senior author on a paper is considered a sign of leadership on the project and is arguably at least as prestigious as being the first author. The importance of this position on the line is illustrated by the fact that Nature in its RSS feed lists the senior author, not the first author, on the ‘by line’ for the abstract. This practice is commonplace (i.e., practically required) in the cellular, molecular & biomedical fields, and is becoming increasingly prevalent in ecology.

This practice might make a certain amount of sense in traditional lab environments where it is practically impossible to do research without grant money and where most work is conducted primarily by members of a single lab, but it makes a lot less sense in ecology. For starters, many graduate student projects don’t depend on grant support from the advisor: field projects are done on the cheap with only small dollar support directly to the graduate student, increasing amounts of research are based on already collected data, and theory plays a prevalent role. Certainly advisors still play important roles in these projects (well, some of them anyway), but not necessarily in some way that is inherently different than that of other contributors.

But whether or not the advisor/PI “deserves” special recognition for projects conducted entirely by members of their labs isn’t the real issue. The real issue is that ecology is increasingly a collaborative science. Ecology is increasingly so interdisciplinary that it is difficult or impossible for a single lab to conduct the most interesting research on its own. Numerous projects combine field work and genetics, field work and theory or advanced statistical analysis, work on multiple major taxonomic groups, etc. The best way to conduct this type of research is for there to be collaboration among labs with different areas of expertise and this practice is increasingly common. But if several labs and therefore several faculty members are involved in a project then who should be senior author?

I have been involved in this type of collaboration and this issue can, in some cases, lead to substantial tension regarding who should be the senior author. I’ve had friends who have had similar experiences as well. These always get sorted out, and if you’re working with the right people there are no hard feelings in the end, but does it even make any sense to elevate one faculty member who has done just a little bit more than another to a position that conotes to the wider world a completely different level of contribution? The logical answer is simply no. In fact this is what causes the resulting tension in the first place. If the issue was who should be second or third author it wouldn’t be such a big deal because there is a level of gradation to the contributions, but senior authorship is completely distinct from all other positions. This doesn’t reflect the reality of cross-laboratory collaboration (except in some very specific circumstances) and it shouldn’t be reflected in the author line.

Now, I don’t care much about author order personally (unless someone else is trying to take a position they clearly don’t deserve), but I’m actually really concerned about this issue. The reason is that I suspect that the increasing emphasis on senior authorship that I’ve been seeing in ecology (an increasing prevalence in its practice, distinction of senior authorship for promotion and tenure, etc.) is actually going to decrease the number of truly collaborative cross-lab projects, just when we need them the most. Increasing pressure for faculty to be the senior author on papers can only lead them to spend less time working on projects where they will not (or risk not) being senior author. This means both not starting collaborative projects and also investing less in those collaborative projects when they do start them (I’ve heard a disturbing number of young faculty say things that support this possibility recently).

I suspect that these kinds of problems have impeded cross-disciplinary research in other fields, but I fear that the concept of senior authorship may be so ingrained in those fields that it may be too late to change it. In contrast, in ecology we still have a chance to insist that our discipline maintains its traditional approach to authorship where the author line is ordered from start to finish with respect to contribution. I believe that this will foster the cross-lab interdisciplinary collaborations that are so critical to understanding ecology and ecological challenges. Or, we can let the tail wag the dog and accept measures of personal success that impede scientific progress. The choice is up to you.

Lazy reviewers and whiny AEs

Those of you who attend conferences will be familiar with the standard – walk up to someone you know, who is talking to someone you don’t, get introduced to new person – conference interaction. Following the introduction at one of these the other introducee looked knowingly at me and said something like, “oh yes, you just turned down a request to review from me.” He then proceeded to complain to the assembled group about how hard it was to find people to review manuscripts. At this point another young AE who was part of the conversation jumped in and expressed similar frustrations. Both of them proceeded to make it clear that in their opinion the problem stemmed primarily from one cause – lazy reviewers. In other words, if everyone would just agree to do a reasonable number of reviews (defined in the conversation as doing enough reviews to match or exceed the number of reviews that are performed on your own papers) then there wouldn’t be any problems.

Now, both these people seemed like they were very nice individuals, and I’m sure that difficulty in finding reviewers is a common frustration among AEs, but as an active member of the reviewing community who not only pulls their own weight, numbers wise, but also spends the time to produce detailed high quality reviews, it really bothers me when either in person, or in print, editors and associate editors whine about lazy reviewers. It’s not that I don’t believe that there aren’t self-centered scientists out there who aren’t in a big rush to do something that can be quite time consuming, that they get effectively no credit for, for free (but that’s a subject for another time). It’s that it is hard to imagine serious people who are paying attention to the state of scientific publishing believing that this is the primary source of the problem.

The real issue is that the number of papers being published on an annual basis has been exploding. Here is a plot of the number of papers with the keyword ‘ecology’ indexed in ISI (note that the growth in this figure is conservative due to the way in which ISI chooses to include journals for indexing).

Now I don’t have any data on the number of regularly utilized reviewers, but I’m pretty sure that if we could get it that the numbers wouldn’t be increasing anywhere nearly as quickly. So the problem is simple – too many papers, too few reviewers.

Who is to blame for this situation? I don’t really know. We could blame publish or perish systems (I don’t, but you could), we could blame corporate publishing houses for the rapid proliferation of academic journals in pursuit of profit, or for that matter we could blame the whiny AEs for not rejecting enough of the poorer submissions without sending them out for review. Whatever the cause/solution this is clearly something that we need to address, and we need to address it now before we break the system.

Funnily enough, in preparing to write this post I went back through my email from the last six months and I couldn’t find a single review request from a journal on which the offending individual served as an AE. Maybe he had me confused with another lazy reviewer…