In this simple problem let us assume that these are the only two types of recruiters that are common. Suppose one observes a recruiter selecting a right-handed person. Is this evidence of bias? Conversely, if they select a left-handed person, is this evidence of being (literally) even-handed? 2/

One can analyze this problem using Bayesian probability. If one starts with some prior odds π(ππππ ππ)/π(π’πππππ ππ) that the recruiter is biased, then observes them recruit a right-handed person, the posterior odds are equal to the prior odds multiplied by π(πππhπ‘βhπππππ|ππππ ππ)/π(πππhπ‘βhπππππ|π’πππππ ππ)=1.8 . In other words, with each right-handed hire, the odds of being biased go up by 80%.

3/

If instead one observes a left-handed person being recruited, the posterior odds are equal to the prior odds multiplied by π(ππππ‘βhπππππ|ππππ ππ)/π(ππππ‘βhπππππ|π’πππππ ππ)=0.2 . In other words, with each left-handed hire, the odds of being biased go down by 80%.

An individual hire only moves the needle a little bit. If one had assigned a 50% prior probability to the recruiter being biased (so the prior odds were 1:1), then a right-handed hire would increase the odds to 1.8:1 (so that the probabilty of bias increases to about 64%) while a left-handed hire decreases it to 0.2:1 (so the probability of bias decreases to about 17%). But neither is conclusive evidence, although the left-handed hire is somewhat more convincing. 5/

If the recruiter was actually unbiased, then over time the odds of bias would go up by 80% half of of the time and down by 80% of the time. Since (1.8)^(0.5) * (0.2)^(0.5) ~ 0.6 < 1, the observed odds of bias would decrease exponentially to zero as one observes more and more hiring data. 6/

Conversely, if the recruiter was actually biased, then since (1.8)^(0.9) * (0.2)^(0.1) ~ 1.45, the observed odds of bias would increase exponentially to infinity as one observes more and more hiring data. 7/

But one should caution against actually implementing this type of analysis as a metric for detecting bias, as it becomes subject to Goodhart's law. If a biased recruiter wants to do the bare minimum to avoid being definitively accused of bias, he or she would only need to increase their hiring of left-handed people to about 27% (and still hire right-handed people 73% of the time), since \( 1.8^{0.73} * 0.2^{0.27} \approx 1\). \) 8/

Follow

@tao

This logic, applied to geographic concentration (e.g. is the auto industry concentrated in Detroit?) is explored in:

https://www.jstor.org/stable/10.1086/262098

and since that 1997 paper, economists tend to call it the dartboard approach.