Definitely not the meaning of “non-significant”

Andrew Gelman over at Statistical Modeling, Causal Inference, and Social Science posted a hilariously awful story about the interpretation of a non-significant result he saw at a recent talk (I particularly love the Grrrrrrr at the end).

I’m always yammering on about the difference between significant and non-significant, etc. But the other day I heard a talk where somebody made an even more basic error: He showed a pattern that was not statistically significantly different from zero and he said it was zero. I raised my hand and said something like: It’s not _really_ zero, right? The data you show are consistent with zero but they’re consistent with all sorts of other patterns too. He replied, no, it really is zero: look at the confidence interval.

Grrrrrrr.

This and related misinterpretations crop up all the time in ecology. I’ve witnessed some particularly problematic cases where the scientist is interested in attempting to determine if some data are consistent with a theoretically predicted parameter and the confidence intervals are relatively wide. The CIs sometimes contain both 0 and the theoretically predicted value and yet it is concluded that the data are not consistent with the model because the parameter is “not significant”. This is obviously problematic given that the goal of the analysis in the first place had nothing to do with demonstrating a difference from 0.