Algorithmic Bias and Fairness

A Black American diagnosed with melanoma is far more likely to die of it than a white American with the same disease. The American Cancer Society puts five-year survival at roughly 94 percent for white patients and around 71 percent for Black patients — a gap driven not by biology but by lateness: darker lesions get caught later, at more advanced stages, when the cancer has already spread. For decades, dermatology has been a specialty trained, illustrated, and tested overwhelmingly on white skin. Then came the machines, and they inherited the specialty's blind spot wholesale.

When researchers began building AI systems to spot skin cancer from photographs, they trained them on the image libraries dermatology already had. Those libraries were the problem. In 2021, a team led by Roxana Daneshjou and colleagues at Stanford went looking for how much dark skin those datasets actually contained, and the answer was almost none. A systematic review in The Lancet Digital Health (Wen et al., 2022) examined more than a hundred publicly available skin-image datasets and found that only a small handful recorded patients' skin tone or ethnicity at all — and among those that did, images of the darkest skin types (Fitzpatrick V and VI) could be counted on your fingers. An entire generation of "state-of-the-art" skin-cancer classifiers had been built, benchmarked, and in some cases deployed, having essentially never seen a Black patient.

The consequences showed up the moment anyone tested for them. Daneshjou's group assembled a deliberately diverse benchmark — the Diverse Dermatology Images set — and ran leading algorithms against it (Daneshjou et al., Science Advances, 2022). Models that scored in the low-to-mid nineties on the light-skinned benchmarks they were trained on saw their accuracy fall sharply on darker skin, with the diagnostic performance gap between the lightest and darkest skin types running to double digits. A system marketed as "95 percent accurate" was quietly two systems: one that worked, and one that did not, sorted by melanin.

This is the whole subject in miniature, so it is worth being precise about the mechanism, because "the AI is racist" is both true and useless as an explanation. The algorithm holds no opinion about race. It has no concept of race. It is a pattern-matcher that learned what a malignant lesion looks like from the examples it was given, and it was given light skin. Cancer also presents differently on dark skin — acral melanoma on palms, soles, and nail beds is proportionally more common and looks nothing like the classic textbook mole — so even the features the model did learn transfer poorly. The gap is not a bug someone introduced. It is a faithful reflection of what was in the training data and what was missing from it.

How data becomes destiny

Every one of the failures in this chapter runs on the same engine, and understanding it once means understanding all of them. A machine-learning system is, at bottom, a device for finding statistical patterns in historical data and projecting them forward. Feed it the past, and it returns a compressed, generalized model of the past, ready to apply to the future. When the past encodes discrimination — in who got hired, who got arrested, who got treated — the model encodes it too, not as a stored prejudice but as a learned correlation it will reproduce at industrial scale.

The most counterintuitive part is that you cannot fix this by hiding the sensitive attribute. Engineers often assume that if they simply withhold race, sex, or age from the model, it cannot discriminate on those grounds. It can, effortlessly, because protected characteristics leave fingerprints all over the rest of the data. Zip code stands in for race in a country as residentially segregated as the United States. First name, the gap in a résumé, the name of a women's college, membership in a particular sport — each is a proxy that lets a model reconstruct the very attribute you tried to remove. This is why "we don't collect race, so we can't be biased" is one of the most dangerous sentences in the field. Removing the label does not remove the information; it only removes your ability to check whether the model is using it.

Proxies are not incidental to algorithmic bias. They are the primary route by which it travels. And the single clearest demonstration of how a well-intentioned proxy can devastate an entire population came not from hiring or policing, but from a hospital billing system.

The most expensive proxy in medicine

In 2019, Ziad Obermeyer and colleagues published a study in Science that has become the canonical case of algorithmic bias, precisely because the mechanism is so simple and the harm so large. They examined a commercial risk-prediction algorithm — one variant of a family of tools used by health systems covering, by the authors' estimate, around 200 million Americans a year — designed to identify which patients were sick enough to enroll in extra "high-risk care management" programs. The algorithm did not predict illness directly. It predicted cost: how much a patient was likely to spend on healthcare next year, used as a stand-in for how sick they were.

It sounds reasonable. Sicker people cost more. But the reasoning has a hole you could drive a hospital through. Black patients, for reasons rooted in access, mistrust, and a long history of being taken less seriously by the medical system, generate lower healthcare costs than equally sick white patients — they see doctors less, get fewer procedures, are less aggressively treated. The algorithm read those lower costs as lower need. At any given risk score the model produced, the Black patients assigned that score were in fact considerably sicker than the white patients assigned the same score.

The scale of the distortion is what makes the study unforgettable. The researchers calculated that correcting the bias would have raised the share of Black patients automatically flagged for extra care from about 18 percent to roughly 46 percent — the algorithm was identifying less than half the Black patients it should have. The proxy did not add a small amount of noise. It cut the eligible Black population by more than half, silently, across hundreds of health systems, for years.

This is why bias in healthcare belongs in a different moral category than bias in hiring, and it is worth being blunt about why. A biased hiring tool costs a qualified person a job — a serious, appealable, recoverable harm; they can apply elsewhere, sue, be made whole. A biased risk score costs a diabetic the care-management program that would have caught their kidney decline, and there is no appeals process for an untreated illness that has already progressed. The error is invisible to its victim — no one gets a letter saying "an algorithm decided you were healthier than you are" — and its consequences can be irreversible. The Obermeyer case is the most egregious not because the developers were malicious (told of the finding, they cooperated on a fix) but because the tool was so widely trusted, so plausibly designed, and so wrong about the people who could least afford it.

The résumé filter goes to court

If healthcare shows the stakes, hiring shows the law starting, slowly, to catch up. In May 2025, a federal judge in the Northern District of California conditionally certified a nationwide collective action in Mobley v. Workday — an age-discrimination suit brought under the ADEA on behalf of job applicants aged 40 and older who were rejected after being screened by Workday's AI-based applicant tools. The lead plaintiff, Derek Mobley, said he had applied to more than a hundred jobs routed through Workday's system and been rejected by all of them.

Two things make this case a landmark rather than a curiosity. The first is who is on the hook. Workday is not an employer; it is a software vendor. The court's willingness to treat the vendor as potentially liable — as an "agent" carrying out employment decisions on behalf of its client companies — cracks open the defense that everyone in the chain has relied on. Employers say the algorithm decided; vendors say they only supply a neutral tool. If neither can hide behind the other, the accountability gap that let AI hiring spread unchecked starts to close. The second is the class mechanism itself. Certifying a collective action means the question is no longer whether one applicant was unlucky, but whether a system produced a pattern of disparate outcomes across thousands of people — exactly the kind of statistical claim algorithmic bias is suited to, and exactly the kind an individual plaintiff could never afford to prove alone.

Alongside it, a parallel front opened over accessibility. In 2025 the ACLU brought complaints against HireVue and Intuit on behalf of a deaf, Indigenous worker whose AI-scored video interview — a system rating speech patterns and facial movement — failed to accommodate her and contributed to a denied promotion. The through-line with dermatology is exact: a system optimized on a majority population silently underperforms on everyone the training data underrepresented, whether that group is defined by skin tone, age, disability, or accent.

The feedback loop that manufactures its own evidence

Predictive policing adds a mechanism the other cases lack, and it is the most insidious one: the algorithm does not merely inherit a biased past — it actively generates the biased future that appears to confirm it.

Start with the data. A predictive policing system is trained on historical crime records, but "crime records" is a misnomer. Police databases record arrests and reports, which track where officers were sent, not where crime occurred. Decades of concentrated enforcement in particular neighborhoods — disproportionately Black and Latino — produced dense arrest records there and sparse ones elsewhere. The algorithm reads that density as danger and recommends sending more officers to those same neighborhoods. More officers make more stops and more arrests, which flow back into the database as fresh evidence of high crime, which raises the algorithm's risk estimate, which sends still more officers. The loop closes and tightens.

graph LR
    A[Historical over-policing<br/>of certain neighborhoods] --> B[Dense arrest records<br/>concentrated there]
    B --> C[Algorithm predicts<br/>high crime risk]
    C --> D[More patrols deployed<br/>to same areas]
    D --> E[More stops and arrests<br/>recorded]
    E --> B

Crucially, the crime rate itself need not change for the numbers to climb — increased scrutiny alone manufactures more recorded offenses, because officers who are present find things officers who are absent never would. The system does not discover crime hotspots. It launders a policing decision into a prediction and then treats its own consequences as an objective finding.

For the individual swept into this machinery, the harms do not stay in one place — they compound across every stage of the criminal justice system, because the outputs of one algorithm become the inputs of the next. A person flagged as high-risk by a patrol-allocation tool is more likely to be stopped, and a stop is more likely to become an arrest. That arrest feeds a pretrial risk-assessment algorithm — tools like COMPAS, which ProPublica's 2016 investigation found labeled Black defendants as future criminals at nearly twice the rate of white defendants who did not reoffend — which recommends against release, so bail is denied or set high. Pretrial detention itself makes conviction and a harsher sentence more likely. The conviction feeds recidivism scores that shadow the person through parole. Each stage treats the previous stage's biased output as neutral fact, so the initial thumb on the scale is not diluted as the case progresses; it is amplified, laundered a little cleaner at every handoff.

Fairness theater

Here is the quietly radical claim at the center of this chapter: algorithmic discrimination is, in one crucial respect, worse than the human discrimination it replaces — not because the outcomes are more severe, but because they are harder to see and harder to fight.

When a human manager rejects a candidate because of their race, the discrimination is at least legible in principle. There is a decision-maker with a motive, a paper trail, sometimes a witness. When an algorithm produces the identical outcome, it arrives dressed in the authority of mathematics: objective, data-driven, applied uniformly to everyone. This is what critics call fairness theater — the performance of neutrality by a system whose neutrality is exactly what cannot be verified.

The theater works because of opacity, and the opacity is layered. The training data is proprietary. The model architecture is a trade secret. The decision threshold is a business parameter. A rejected applicant has no way to know whether the score that sank them reflected their qualifications or a proxy the model learned to associate with their demographic group — and neither, often, does the company using it. This produces a brutal asymmetry of power: the deployer holds all the data, tools, and access needed to detect bias, while the affected person holds none. Even the audits meant to check these systems are frequently commissioned and scoped by the same company that deployed the tool, shaped by the commercial incentives that put it in place. The result is that a discriminatory outcome becomes not just hard to reverse but hard to name. Discrimination that cannot be seen cannot be challenged, and discrimination that cannot be challenged tends to persist and spread.

Which fairness? The impossibility at the core

Suppose a company genuinely wants its algorithm to be fair, hires ethicists, and asks them for a definition. It will discover that "fairness" is not one target but several mutually exclusive ones, and that mathematics — not politics — forbids satisfying them all at once.

Definition	What it demands	Who tends to favor it
Equal accuracy	The model is correct at the same rate for every group	Deployers seeking a single performance number
Equal false-positive rate	Groups are wrongly flagged (e.g. as high-risk) at equal rates	Defendants, applicants — those harmed by false alarms
Equal false-negative rate	Groups are wrongly cleared / missed at equal rates	Patients, public-safety advocates — those harmed by misses
Equal opportunity	Among those who truly qualify, all groups have equal selection odds	Anti-discrimination advocates
Equal outcomes (demographic parity)	Selection rates are equal across groups regardless of base rates	Advocates of substantive equality

The catch, proven formally by Kleinberg, Chouldechova, and others around 2016–2017, is that whenever base rates genuinely differ between groups, you cannot simultaneously equalize false-positive rates and false-negative rates while keeping the score equally well-calibrated. The COMPAS fight was, at heart, a collision between two of these definitions: ProPublica accused the tool of unequal false-positive rates, while the vendor defended it as equally calibrated across race. Both were arithmetically correct. They were measuring different fairnesses, and no algorithm could give them both.

This is the deepest and most important point in the chapter, so it should be stated without hedging: the competing definitions of fairness are not a technical problem awaiting a cleverer method. They are a genuine value disagreement wearing mathematical clothing. To choose equal false-positive rates over equal false-negative rates is to decide whose errors matter more — the wrongly-flagged defendant's or the missed victim's. No amount of better data or bigger models resolves that, because it is not a question of fact. It is a question of values, and pretending otherwise — outsourcing a moral choice to an optimization target — is itself a way of hiding the choice from the people it will affect.

How long, and how many?

A sobering pattern runs through every documented case: the systems ran for years before anyone caught them. The Science healthcare algorithm shaped care for millions across hundreds of hospitals before Obermeyer's team went looking. Biased dermatology models were benchmarked and published before anyone assembled a diverse test set to check them. These were not caught by regulators, or by the companies that built them, or by the institutions that bought them. They were caught by outside academics who happened to have the data, the tools, and the motive to look — and who found bias essentially every time they looked.

That last fact should keep us up at night, because it inverts the usual reassurance. The known cases are not a catalog of rare failures that were successfully policed. They are the handful of systems that happened to fall under the gaze of a curious researcher with rare access. For every audited COMPAS or Optum tool, an unknown number of hiring filters, credit models, benefits-eligibility screens, and clinical tools operate with no external scrutiny at all. We do not know the base rate of bias among deployed AI systems, and that ignorance is itself the finding: when researchers have checked, they have found significant disparities more often than not. A credible audit regime — mandatory, independent, with subpoena power over training data and access to real-world outcomes broken down by group, conducted by auditors who are not paid by the deployer — does not yet exist at scale anywhere. Until it does, any confident claim about how many biased systems are running is a guess, and the direction the evidence points is not comforting.

What the law is trying, and what it would take

Accountability is arriving, but the deployment curve is far steeper than the legal one. New York City's Local Law 144 requires employers using automated hiring tools to commission annual bias audits and publish the results — a real step, undercut by the fact that companies choose their own auditors and the metrics remain contestable. Colorado passed the most comprehensive US measure, the Colorado AI Act (SB24-205), imposing a duty of "reasonable care" on deployers of high-risk systems to avoid algorithmic discrimination — though implementation has already slipped past its original early-2026 start as legislators wrestle with how to make it workable. The EU AI Act classifies AI used in employment, credit, and essential services as high-risk and requires conformity assessments including bias testing, phasing in across 2026 and 2027. The Mobley certification suggests the courts may move faster than the statutes: existing anti-discrimination law, applied to algorithms as agents, can bite now, without waiting for a new regime.

But two structural questions will decide whether any of this matters, and both are really about where the burden falls. The first is legal: must a company prove its system is non-discriminatory before deployment, or must a harmed plaintiff prove discrimination after the fact? Today the burden sits almost entirely on the victim — the party with the least data, the least access, and the least ability to see inside the machine that harmed them — which, given the opacity described above, means most bias is never litigated at all. Shifting even part of that burden onto deployers, requiring them to demonstrate fairness as a condition of use the way a drugmaker must demonstrate safety, would do more to change behavior than any single definition of fairness.

The second question is technical but leads somewhere uncomfortable. When a system is biased, how much of the blame lies with the training data versus the architecture, the optimization target, or the deployment context? Sometimes the sources separate cleanly — the Science case was overwhelmingly a choice-of-target problem (cost instead of illness), fixable by changing what the model predicted, not what it was trained on. Often they do not, and for liability it may not matter: a company chose the data, chose the target, and chose to deploy. But it matters enormously for fixing the problem, because it tells you whether the remedy is more representative data, a different objective, or not deploying at all.

And the representation-fix has a hard ceiling, which returns us to where we began. You can build a diverse dermatology dataset, and researchers now are. You can swap a cost proxy for a direct measure of illness. You can audit a hiring tool for disparate impact. Each of these genuinely reduces harm and should be done. But no technical fix reaches the source, because the training data is a mirror. It reflects a world in which Black patients were undertreated, women were underhired, and some neighborhoods were overpoliced. An algorithm trained on an unequal society will, absent deliberate correction, reproduce that inequality — and "deliberate correction" is not a line of code but a series of contested value choices about what a fair distribution would even look like. The honest conclusion is that algorithmic bias sits at a hinge between engineering and politics. Better data, independent audits, and pre-deployment burdens can shrink it. Only a less unequal society can remove its source.

Summary

Algorithmic bias is not a malfunction. It is what happens when a pattern-matching system faithfully learns from data produced by an unequal world.

The dermatology case states the whole problem plainly. AI skin-cancer tools that scored in the nineties on light skin lost double-digit accuracy on dark skin (Daneshjou et al., 2022), because the image datasets they learned from contained almost no dark skin at all (Wen et al., 2022). The mechanism is a representation gap, not a hidden prejudice.
Proxies carry the bias even when protected traits are removed. Zip code encodes race; healthcare cost encoded illness so imperfectly that a widely used algorithm cut the share of Black patients flagged for extra care by more than half (Obermeyer et al., Science, 2019). Deleting the sensitive label does not delete the information — it deletes your ability to check.
Healthcare bias is a graver category than hiring bias. A biased hire is appealable and recoverable; a missed diagnosis is invisible to its victim and often irreversible.
Predictive policing manufactures its own confirming evidence. Enforcement history becomes prediction becomes deployment becomes fresh arrest data — a feedback loop that compounds through bail, sentencing, and parole, each stage treating the last stage's biased output as fact.
Fairness theater makes algorithmic discrimination harder to fight than the human kind. Opacity plus an access asymmetry between deployer and victim means discriminatory outcomes become hard to see, hard to prove, and hard to even name.
The competing definitions of fairness are a values conflict, not a math puzzle. When base rates differ, equal false-positive rates, equal false-negative rates, and calibration cannot all hold at once — so choosing among them means choosing whose errors matter most.
The lag is the warning. Every major case ran for years before an outside researcher caught it, and researchers find bias nearly every time they look. Without mandatory independent audits and a shift of the proof burden onto deployers, most biased systems will keep running unseen.
Technical fixes have a ceiling. Diverse data, better targets, and audits genuinely reduce harm. But the training data is a mirror of an unequal society, and no algorithm can be engineered fairer than the world that produced its data.

Sources

Last updated: 2026-08-06

V2 (in progress) Previous: V1