Algorithmic Fairness
Summary: Mitchell et al. (Annual Review of Statistics, 2021) catalog the proliferating and often mutually incompatible mathematical definitions of fairness in ML/AI decision systems, expose the hidden choices that shape fairness long before the math, and show that no single definition can satisfy all stakeholders when base rates differ across groups.
Sources: Academia/annurev-statistics-042720-125902.pdf
Last updated: 2026-05-06
Why it matters
Prediction-based decision-making is now embedded in lending, hiring, pretrial detention, child maltreatment screening, public health, and welfare eligibility. The stakes are high and the populations affected are often already disadvantaged. A “fair” model sounds like a technical goal but turns out to involve irreducibly normative choices.
The proliferation problem
A “recent wave of research” attempted to define fairness quantitatively. The result was dozens of competing, often incompatible definitions. Chouldechova (2017) and Kleinberg et al. (2016) proved that several of the most widely used criteria are mathematically mutually exclusive except in degenerate cases (equal base rates across groups):
- You cannot simultaneously achieve equalized false positive rates and equalized false negative rates and calibration across groups when base rates differ.
- These incompatibilities are not technical problems to be solved — they reflect genuine tensions in what “fair” means, tensions that involve contested value choices. More data or better models cannot resolve them.
Hidden choices that shape fairness before the math
The paper’s central contribution is exposing three choices usually treated as given:
1. The overarching goal
What is the model for? Different stakeholders (a bank, a judge, a social planner, a community organization) have genuinely different and conflicting goals. A model that successfully advances one goal may be unacceptable to another, regardless of which fairness metric it achieves. If you disagree with the overarching goal, no amount of mathematical fairness in the model makes it acceptable.
2. The population
Models are applied to a specific subpopulation determined by a selection mechanism. That mechanism may itself be systematically unfair — e.g., policing that targets racial minorities generates arrest data that over-represents those groups. A model that appears fair within the population may be built on a population that entered it unfairly.
3. The decision space
What decisions are actually available? Reducing “release or detain” to a binary obscures the many possible intermediate options. A model can only optimize within the decision space it is given, and a narrow decision space may itself be unjust.
Statistical bias vs societal bias
The paper distinguishes two forms of bias that require different remedies:
- Statistical bias: nonrepresentative sampling or measurement error — the data do not accurately reflect reality as it is. Can potentially be addressed technically.
- Societal bias: accurate data that encodes objectionable social structures. Women’s wages may be accurately measured but systematically lower due to discrimination. Using that data accurately perpetuates the injustice. This may have no technical solution.
The two interact: overpolicing of certain communities creates statistical bias in crime data (more police → more arrests, independently of crime rates). Even if crime could be “perfectly measured,” those rates reflect unequal social structures — societal bias arises even in how crime is defined.
Omitted payoff bias
Prediction systems focus on one or a few measurable outcomes. This narrows complex social goals (“public safety,” “rehabilitation,” “opportunity”) to measurable proxies. Optimizing the proxy can actively work against the true goal — particularly when the proxy captures only short-term, easily counted outcomes while ignoring longer-term, harder-to-measure ones.
Causal frameworks
Standard statistical fairness definitions operate on observed associations. Causal frameworks (DAGs, counterfactual reasoning) offer more principled tools — they can ask “what would have happened if this person had been a different race?” But causal frameworks require knowing the true causal structure of the system, which is almost always contested in high-stakes domains. They shift the normative question rather than resolve it.
Connections
The incompatibility theorem echoes uncertainty: some questions cannot be resolved by more information because they embed genuine value conflicts. The societal-bias problem connects to systems-thinking: the behavior of a system reflects its structure, not just its components. You cannot fix biased outputs without addressing the feedback loops that produced the biased data.