"Psychologists in particular wanted a statistical skeleton key to unlock true experimental insights. It was an unrealistic burden to place on statistics, but the longing for a mathematical seal of approval burned hot."
https://www.sciencenews.org/article/statistical-significance-p-value-null-hypothesis-origins
@cyrilpedia @nadege “Researchers often don’t realize that statistically significant results don’t prove that a true effect has been discovered.”
I try to teach my students that we use statistics as a tool to help us make good guesses, but that stats simply can not tell us the truth of the world no matter how small the p-value is. Stats is not math in that sense—it does not lead to conclusive results and natural truths. It’s more like a coin flipping game and we just want to know whether we think it’s a fair coin (null) or not. If we think it isn’t a fair coin, we don’t stop. We have to figure out why (theory and mechanism) to win the game. But I feel like we’ve so indoctrinated students that low p-values mean that the null isn’t true that it’s like a deprogramming exercise when I go over this stuff.
I would love to see more history on how we got here. Thanks for sharing!
We develop theory and just keep running experiments priding them, refine and repeat.
@AllenNeuroLab @cyrilpedia My favorite quote is "a picture is worth a thousand reckonings of statistical significance". This is something an old teacher in grad school also said to me once. Actually his exact words were "if the graph does not hit you between the eyes, don't bother with statistics" and that has been the most useful advice. Since then I have been caring for effect size. I keep telling my students about this, and I can see that they don't immediately believe me and they keep looking at statistics before looking at graphs and I keep fighting over this.
@cyrilpedia The fact that "null hypothesis significance" leads to junk science has been pointed out for a very long time. Unfortunately the belief, leading to distinct Kuhnian paradigms, is deeply ingrained in the social sciences, and too many careers are at stake to rescind. I fear that little will grow on the cinders of much of psychological publications of the last 70 years. Who knows how long this will continue until young researchers finally re-examine the foundations of their science.