@cyrilpedia The fact that "null hypothesis significance" leads to junk science has been pointed out for a very long time. Unfortunately the belief, leading to distinct Kuhnian paradigms, is deeply ingrained in the social sciences, and too many careers are at stake to rescind. I fear that little will grow on the cinders of much of psychological publications of the last 70 years. Who knows how long this will continue until young researchers finally re-examine the foundations of their science.

@cyrilpedia @nadege “Researchers often don’t realize that statistically significant results don’t prove that a true effect has been discovered.”

I try to teach my students that we use statistics as a tool to help us make good guesses, but that stats simply can not tell us the truth of the world no matter how small the p-value is. Stats is not math in that sense—it does not lead to conclusive results and natural truths. It’s more like a coin flipping game and we just want to know whether we think it’s a fair coin (null) or not. If we think it isn’t a fair coin, we don’t stop. We have to figure out why (theory and mechanism) to win the game. But I feel like we’ve so indoctrinated students that low p-values mean that the null isn’t true that it’s like a deprogramming exercise when I go over this stuff.

I would love to see more history on how we got here. Thanks for sharing!

We develop theory and just keep running experiments priding them, refine and repeat.