The Art of Statistical Scamming in Experiments

John Bohannon writes,

Here’s a dirty little science secret: If you measure a large number of things about a small number of people, you are almost guaranteed to get a “statistically significant” result. Our study included 18 different measurements—weight, cholesterol, sodium, blood protein levels, sleep quality, well-being, etc.—from 15 people. (One subject was dropped.) That study design is a recipe for false positives.

Usually, I think of health studies as bad because they are non-experimental. But this is a way to scam experimental studies.

This entry was posted in statistical methods. Bookmark the permalink.

2 Responses to The Art of Statistical Scamming in Experiments

  1. Bryan Willman says:

    See also Slate Star Codex commentary on same:

  2. 3rdMoment says:

    Even with a large sample you still have the multiple comparisons problem. No matter how many subjects you have, you’ll still get a false positive about 5% of the time.

    What small sample does is exacerbate the statistical significance filter…the false positives you do get will look like really big effects.

Comments are closed.