Suppose there is a profession where there are an equal sized pool of qualified candidates in two groups - let's say "right-handed" and "left-handed" to keep the example relatively neutral. Some recruiters are unbiased and would pick right-handed candidates 50% of the time and left-handed candidates 50% of the time, but other recruiters are quite biased against left-handers and would only pick them 10% of the time, choosing right-handers 90% of the time. 1/
In this simple problem let us assume that these are the only two types of recruiters that are common. Suppose one observes a recruiter selecting a right-handed person. Is this evidence of bias? Conversely, if they select a left-handed person, is this evidence of being (literally) even-handed? 2/
One can analyze this problem using Bayesian probability. If one starts with some prior odds 𝑃(𝑏𝑖𝑎𝑠𝑒𝑑)/𝑃(𝑢𝑛𝑏𝑖𝑎𝑠𝑒𝑑) that the recruiter is biased, then observes them recruit a right-handed person, the posterior odds are equal to the prior odds multiplied by 𝑃(𝑟𝑖𝑔h𝑡−h𝑎𝑛𝑑𝑒𝑑|𝑏𝑖𝑎𝑠𝑒𝑑)/𝑃(𝑟𝑖𝑔h𝑡−h𝑎𝑛𝑑𝑒𝑑|𝑢𝑛𝑏𝑖𝑎𝑠𝑒𝑑)=1.8 . In other words, with each right-handed hire, the odds of being biased go up by 80%.
3/
If instead one observes a left-handed person being recruited, the posterior odds are equal to the prior odds multiplied by 𝑃(𝑙𝑒𝑓𝑡−h𝑎𝑛𝑑𝑒𝑑|𝑏𝑖𝑎𝑠𝑒𝑑)/𝑃(𝑙𝑒𝑓𝑡−h𝑎𝑛𝑑𝑒𝑑|𝑢𝑛𝑏𝑖𝑎𝑠𝑒𝑑)=0.2 . In other words, with each left-handed hire, the odds of being biased go down by 80%.
An individual hire only moves the needle a little bit. If one had assigned a 50% prior probability to the recruiter being biased (so the prior odds were 1:1), then a right-handed hire would increase the odds to 1.8:1 (so that the probabilty of bias increases to about 64%) while a left-handed hire decreases it to 0.2:1 (so the probability of bias decreases to about 17%). But neither is conclusive evidence, although the left-handed hire is somewhat more convincing. 5/
If the recruiter was actually unbiased, then over time the odds of bias would go up by 80% half of of the time and down by 80% of the time. Since (1.8)^(0.5) * (0.2)^(0.5) ~ 0.6 < 1, the observed odds of bias would decrease exponentially to zero as one observes more and more hiring data. 6/
Conversely, if the recruiter was actually biased, then since (1.8)^(0.9) * (0.2)^(0.1) ~ 1.45, the observed odds of bias would increase exponentially to infinity as one observes more and more hiring data. 7/
But one should caution against actually implementing this type of analysis as a metric for detecting bias, as it becomes subject to Goodhart's law. If a biased recruiter wants to do the bare minimum to avoid being definitively accused of bias, he or she would only need to increase their hiring of left-handed people to about 27% (and still hire right-handed people 73% of the time), since \( 1.8^{0.73} * 0.2^{0.27} \approx 1\). \) 8/
The more useful conclusion from this sort of analysis is that one cannot extrapolate too strong evidence of bias (or lack thereof) from any single data point. It's only after the accumulated weight of many observations that the odds of bias become either exponentially high or exponentially small. END
@tao
This logic, applied to geographic concentration (e.g. is the auto industry concentrated in Detroit?) is explored in:
https://www.jstor.org/stable/10.1086/262098
and since that 1997 paper, economists tend to call it the dartboard approach.
