Definitely not the meaning of “non-significant”

Andrew Gelman over at Statistical Modeling, Causal Inference, and Social Science posted a hilariously awful story about the interpretation of a non-significant result he saw at a recent talk (I particularly love the Grrrrrrr at the end).

I’m always yammering on about the difference between significant and non-significant, etc. But the other day I heard a talk where somebody made an even more basic error: He showed a pattern that was not statistically significantly different from zero and he said it was zero. I raised my hand and said something like: It’s not _really_ zero, right? The data you show are consistent with zero but they’re consistent with all sorts of other patterns too. He replied, no, it really is zero: look at the confidence interval.

Grrrrrrr.

This and related misinterpretations crop up all the time in ecology. I’ve witnessed some particularly problematic cases where the scientist is interested in attempting to determine if some data are consistent with a theoretically predicted parameter and the confidence intervals are relatively wide. The CIs sometimes contain both 0 and the theoretically predicted value and yet it is concluded that the data are not consistent with the model because the parameter is “not significant”. This is obviously problematic given that the goal of the analysis in the first place had nothing to do with demonstrating a difference from 0.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: