Paweł Lenartowicz @plenartowicz@qoto.org

@lakens
Assuming of normal distribution under h0 (simply because of the CLT), can be perfectly valid, so t-tests for h0 also. But at the same time, equivalence test could be not!

@lakens @lakens
Yeah, I know your article about it. Bahrens Fisher problem is heavily discussed for years :)
But both tests have assumption of normal distribution of means. And the same problem of ignoring heavy tailed distribution /vviolation of normality / skewed distribution / heteroscedascity/ mediation / moderation. However called situation, where sample is to small to be efficiently affected by CLT.

Let me repeat, equivalence testing can’t provide conclusion, that effect is small, when it’s relatively rare comparing to sample, regardless significance of results.

@lakens @JorisMeys

Estimation effect size and CI via Welsh’s t-test assumes normal distribution of effect :) I mentioned that :)

@JorisMeys
But we never know if sample is big enough to detect rare (but strong) effect. EqTesting is easy way to underestimate sample size (it is why probably EqTesting is so popular in pharmaceutical studies).

@lakens

1) Of course, you can assume any distribution. (And that procedure is called “Neyman-Pearson theory of statistical testing”.)
‘Equivalence testing’ is procedure almost always connected to t-test. Like in your textbook (photo 1) or TOST procedure (Schuirmann, D. J. 1987) .

2) “Violations of normality mostly have very little impact on error rates”, violation of normality have biggest impact on estimation of variance, so also on error rates and effect estimation. (It’s why heteroscedasticity is so important.)

1) It’ll be easy to show how easily ‘equivalence tests’ can be very wrong, if assumptions ignores non-normality of effect (by using t-test).
I think I can make some simulation after 22:00 GMT. For now, I can show what happens to p-distribution, when effect is (very) not normal. (photo 2 - no effect, non normal distibution when h1=true, 3&4 valid use of t-test, effect big but moderated).

Schuirmann, D. J. (1987). A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of pharmacokinetics and biopharmaceutics, 15, 657-680.
https://link.springer.com/article/10.1007/BF01068419

@lakens

The solution for such problem is already known for 90 years.
1) specify your model
2) test your model against probable alternatives

@lakens
Oh, my god, NO!
If we make strong assumptions about normality (Welch’s) or uniformity (Student’s t-test) of effect, as we do in equivalence testing, we can only conclude that that certain model is unlikely.

In other words, if the real effect is moderated or mediated, this procedure fails. Frequency-based statistics is very sensitive to model misspecification. It is a problem, It’s not an advantage to use it. We can’t conclude h0 because data is unlikely in specific h1.