If a computer helped decide your loan, job application or care, you may wonder what is algorithmic bias. It is when an automated system produces systematically unfair results for some people, usually because of the data or design behind it.
This explainer is part of the site’s section on technology and society.
| Item | Value |
|---|---|
| Families of bias in one academic review | 4 (historical/representational, selection/measurement, algorithmic/optimization, feedback/emergent) |
| Speech-recognition error gap for Black speakers | Roughly twice the word-error rate, across 5 U.S. metropolitan areas |
| ProPublica report on criminal risk assessment tools | 2016 |
| Amazon recruiting tool tested | 2018 |
| Early expert systems found to behave in discriminatory ways | 1980s and 1990s |
The figures come from a peer-reviewed review of bias in AI systems and a New America report. Research in this field moves quickly, so check the sources before you rely on the numbers.
How algorithmic bias works
The philosophical literature uses the term algorithmic fairness to assess whether machine learning algorithms operate fairly, as the Stanford Encyclopedia of Philosophy entry, first published in July 2025, puts it. Algorithmic bias is the failure side of that question: a system that treats people or groups unfairly.
Picture a data scientist handed records of past events, such as successful employees, loan recipients who repay, or released prisoners who reoffend. Their task is to build a model that predicts similar outcomes for new people. The same Stanford entry notes that, even when such a model works well, it is unclear what makes it fair or unfair.
Where these tools are used matters. The Stanford entry separates algorithms that allocate benefits and burdens from those that provide information:
- A bank might use an algorithmic tool to decide who is granted loans.
- A state might use one to help decide which inmates stay in jail and which are released on parole.
- Other algorithms provide information, such as translation services or search engines.
Many decision-makers feel comfortable relying on these systems because of a perception of objectivity. New America reports that there is often a perception that automated systems are less biased than people because they rely solely on data and statistics. When organisations actually test such systems, it adds, they often find unintended, biased results.
Where algorithmic bias comes from
One review of bias in AI systems sorts bias into four interrelated families:
- Historical or representational bias
- Selection or measurement bias
- Algorithmic or optimization bias
- Feedback or emergent bias
The most common source is the training data. New America says it reflects biases and discrimination that exist in the physical world. Because predictive algorithms learn from correlations in past data, historical patterns can be carried forward, for example if only certain people held certain jobs in the past.
Removing a sensitive label does not always fix the problem. Discrimination can occur even when a dataset does not explicitly contain categories such as race or gender but includes highly correlated variables that act as proxies. The Stanford entry makes a related point: machine learning algorithms will identify traits correlated with legally protected traits like race and sex, which leaves scholars asking when those traits count as proxies.
Real cases of automated systems treating people unfairly
Documented examples show how the sources of bias play out in practice:
- Hiring. In 2018 Amazon built and tested an AI recruiting tool, and New America reports that it came to prefer male candidates over female candidates for technical positions. It was built to learn from patterns in previous resumes, and applications for those roles had mostly come from men. Amazon abandoned the tool before it was used to recruit.
- Health care. A study of a widely used health care algorithm found that it failed to identify Black patients at risk at the same rate as white patients, so they were less likely to receive preventative care, according to New America. The academic review describes the same study as finding that a deployed health-risk score underestimated the needs of Black patients relative to White patients with comparable disease burden.
- Speech recognition. Across five U.S. metropolitan areas, commercial systems produced roughly twice the word-error rate for speakers who self-identify as Black, the same review reports.
- Criminal justice and facial recognition. ProPublica’s 2016 report revealed racial bias in criminal risk assessment tools, and the Gender Shades study exposed major disparities in facial-recognition performance for darker-skinned women, as the review recounts.
These findings come mostly from the United States. The review warns that most large-scale bias studies rely on datasets from the United States or Western Europe, so results should not be assumed to hold everywhere. In other places, factors such as accent, dialect or income may be the ones that matter.
The concern is also not new. The same review notes that worries about discriminatory behaviour in early expert systems date back to the 1980s and 1990s, though those warnings were largely overlooked.
Why is fairness so hard to define?
Part of the difficulty is that people mean different things by fair. The Stanford entry describes a comparative view, in which you judge a system by how one person or group is assessed compared with another. All comparative accounts share the principle that like cases should be treated alike, but they differ on which similarities matter morally.
A non-comparative view holds that an algorithm is unfair if it is inaccurate, or if its processes are obscure. The entry also treats data problems as a separate issue, including measurement error and whether training data fairly represents people. It devotes a section to the debate over the COMPAS risk tool and the competing fairness measures used to judge it.
Opacity makes all of this harder to resolve. According to New America, it can be especially difficult to hold companies accountable for discrimination caused by AI tools, because affected people often lack knowledge of how the tools work and the ability to obtain that information.
What can reduce algorithmic bias?
The academic review evaluates several mitigation strategies:
- Dataset diversification
- Fairness-aware modeling
- Post-deployment auditing
- Regulatory frameworks
- Participatory design
The Amazon case shows why testing matters. New America draws the lesson that AI systems need testing and auditing for discriminatory impacts both before and after deployment. It also suggests algorithmic tools suit uses that do not rely on sensitive personal information and do not make consequential decisions about individuals.
Rules are emerging too. The review points to frameworks such as the OECD AI Principles (2019) and the U.S. Blueprint for an AI Bill of Rights (2022), and notes that regulatory and ethical frameworks are emerging globally; check each framework’s own page for its present status.
Bias also connects to wider questions about who has access to technology and whose data gets collected, which the guide on digital divide explained covers. Ranking and recommendation systems raise related fairness questions, which you can follow further in the piece on social media and democracy.
Frequently asked questions
What is algorithmic bias in simple terms?
Show answer
It is a pattern of unfair results from an automated system, where some people or groups are treated worse than others. It typically traces back to training data that reflects real-world discrimination, to design choices, or to proxy variables.
Can a system be biased if it never uses race or gender?
Show answer
Yes. According to New America, discrimination can occur when a dataset includes highly correlated variables that serve as proxies for those characteristics. Amazon’s recruiting tool was not built to consider gender, yet it still preferred male candidates for technical positions.
Are automated decisions more objective than human ones?
Show answer
Not automatically. There is often a perception that they are, but when organisations test AI systems they often find unintended, biased results.
Does algorithmic bias look the same everywhere?
Show answer
Not necessarily. Most large-scale studies use data from the United States or Western Europe, and the review cautions that the size, and sometimes the direction, of bias can differ between places.
