Going to #SIPS2023, but unsure what unconference to visit? Go ahead and give our very brief (5 min!) ramp-up video on standardisation practices in Psychology a watch! @improvingpsych
@JFuenderich, @epizyklen@nerdculture.de and I would love to discuss standardisation practices with you in Padua (or remote). The video is on osf: https://osf.io/eab9u or, including subtitles, on YouTube: https://youtu.be/tRveDihxtfM
@LBnhr @improvingpsych @JFuenderich @epizyklen Important issues! Not attending SIPS unfortunately but I'm a big fan of no standardization (absolute effect sizes) myself :)
@alexh @improvingpsych @JFuenderich @epizyklen@nerdculture.de
Too bad that you can't make it, would have loved to hear more about your experience with this!
What's your reason to prefer absolute effect sizes?
@LBnhr @improvingpsych @JFuenderich @epizyklen they mean a lot more, if the measure means much. They are the measure without dividing by something that is highly contingent, which as you point out, turns a common metric (the raw measure) into apples and oranges
@alexh @improvingpsych @JFuenderich @epizyklen@nerdculture.de
Agreed! I think "if the measure means much" is the key point here - I don't really have the experience to judge this, but I can get the feeling that quite often, we don't know enough about our measures to interpret them directly in a meaningful way. Resorting to standardized effect sizes can lead to a (false) sense of security in interpretations - I'm excited to hear what people at #SIPS2023 think about this.
@LBnhr @improvingpsych @JFuenderich @epizyklen yes, I think reliance on standardized effect sizes signals that the science of the area has failed. Really need to work on measurement so that results aren't contingent on the variability of the population samples for each study. Then you can have a properly cumulative science. I'm no expert in this area, however, so this may be a somewhat unhinged rant.
@LBnhr @improvingpsych @JFuenderich @epizyklen I like that in some cases soft sciences have devised meaningful absolute metrics, such as "equivalent additional days of learning" in education research. Still some interpretation problems but better than dividing by standard deviation. That's the kind of thing psychology should be aiming for.
Oh I wasn't aware of this yet! That sounds like a great step in the right direction - I've just started looking into this, and it seems they specifically communicate "growth in sd" (standardised effect size) and "additional days of learning" (unstandardized/intuitive effect size). I haven't really grasped how these additional days are estimated though. I'd assume with some kind of (linear) predictive model?
I remember a discussion on Twitter from a while ago - I think there's some movement in clinical Psychology as well, to move away from standardized effect sizes and instead discussing change in measures like the BDI. I'm totally out on a whim here though :D
Additionally, discussing something like therapy success sounds promising to me as well - it may be hard to define, but once defined, interpretation and modelling should be more transparent
@LBnhr yes, on a depression scale for example, e.g. BDI, surely clinicians have gained a sense of how many points corresponds to going from depressed to not at all depressed. Why don't they use that amount as a benchmark and relate everything to it? I suppose different cultures give different absolute scores and spreads of scores, but within a large fairly well-understood population like the U.S. shouldn't they be doing that?
The more I think about it, the more certain I am, that this is actually happening (to some extent) - at the very least in judging the effect of novel treatments & interventions. Would be great to talk to some people dealing with such studies - to
a) get confirmation that this is indeed what's happening
b) get an idea of how such benchmarks or baselines are developed
c) understand how we can make such processes our own in the more experimental, ad hoc side of Psychology
