Did you know? An AI model can score 90% on an overall test and still make far more mistakes for some of the people who use it. The average sounds encouraging. The hidden differences may be the most important part of the story.
This is why responsible AI teams examine bias and fairness. The aim is not simply to announce that a model is “unbiased.” It is to identify whose experiences are missing, where errors differ, what harms can follow and how the design can be improved.
Quick answer: check the dataset, examine how labels were assigned, test errors in relevant groups, investigate possible causes, improve the process and keep monitoring outcomes. Fairness is an ongoing evaluation of a system in context, not one number that applies everywhere.
The 30-second guide
- Sampling bias: the available examples do not adequately represent the situations the model will face.
- Measurement bias: the observations or labels systematically misrepresent what they are supposed to measure.
- Historical bias: past inequities or practices are reflected in records used for training.
- Proxy feature: an apparently neutral variable that can correlate with another relevant or sensitive characteristic.
- Slice evaluation: checking model performance separately for meaningful groups of examples.
- Fairness criterion: a clearly defined principle for comparing outcomes and errors in a particular use case.
One word of care: bias can mean different things in statistics and in discussions of social fairness. A model can have a statistical prediction bias without that alone proving discriminatory treatment. You must define what is being measured and who could be affected.
A fictional example: the voice recogniser that misses one classroom
Imagine a hypothetical speech-recognition app tested on 200 short sentences. It transcribes 90 of 100 clear recordings correctly, but only 60 of 100 recordings made in a noisy environment. Across all 200, the headline result is (90 + 60) ÷ 200 = 75%.
The 75% total conceals a major difference: 90% correct in one setting and 60% in another. This is a made-up illustration, not a report about any commercial app. It demonstrates why reading only an aggregate score can hide a real usability problem.
The gap may be caused by microphones, training conditions, noise or other factors. It does not prove anything about the ability of the people speaking. A responsible investigator needs evidence before assigning a cause.
Step 1: Ask who or what appears in the training data
Machine-learning systems learn from available examples. If an image classifier sees mainly clear, bright photographs, it may struggle with pictures captured in lower light. If a speech system is trained on a narrow set of accents, it may make more errors on varieties of speech that were underrepresented.
Look at the planned users and situations. Which languages, devices, lighting conditions or accessibility needs are expected? Which appear rarely in the dataset? Even large datasets can be poorly matched to the people who will rely on the product.
The remedy is not always “collect more data.” Adding thousands of examples that repeat the same imbalance may have little benefit. Better coverage, label quality and task design matter.
Step 2: Check whether the labels themselves are trustworthy
A training label is often treated as the answer the model should learn. But some labels are judgments, administrative decisions or measurements with errors. If people were historically treated differently, the recorded outcomes may reflect that treatment instead of an underlying fair standard.
For example, imagine a fictional educational system that labels learners as “engaged” solely because they log into an app every evening. Students who work offline, share a device or have restricted internet access could receive a misleading low-engagement label despite meaningful learning elsewhere.
Changing the model architecture will not repair a faulty definition of engagement. Before training, ask whether the target truly measures the intended concept and whether a reasonable alternative interpretation exists.
Step 3: Watch for hidden shortcuts and proxy variables
An algorithm may rely on a feature that happens to correlate with its training labels, even if that feature is irrelevant to the real task. This is a shortcut. In consequential applications, a postcode, device type or usage pattern might correlate with circumstances that should not determine access or opportunity.
Removing one obvious sensitive column is therefore not proof of fairness. Other features can carry related information. The full decision process, legitimate purpose, data governance and potential disparate effects require inspection.
Proxy correlations do not automatically prove unlawful discrimination, either. Appropriate analysis considers the use case, relevant legal standards, potential harms and whether different groups receive unjustified treatment.
Step 4: Look beyond the average with slice-based testing
In a classification task, developers can measure false positives, false negatives, precision and recall for relevant slices of their evaluation data. In a speech task, they may compare word error rates by language variety or recording condition, subject to suitable data protection and ethical procedures.
Here is another tiny fictional example. A tool flags incorrect homework answers for review. For Group A it catches 18 of 20 genuinely incorrect answers; for Group B it catches 10 of 20. Recall is therefore 90% for A and 50% for B. Both groups may appear in an overall score, yet one receives far less reliable detection.
Those numbers do not reveal why the difference occurred. Perhaps one group’s answer format was omitted from training, or the reference labels were less reliable. Review the examples, the measurement procedure and how the tool would actually be used.
The question isn’t only “How accurate is the model?” It is “Accurate for whom, for which examples, and with what consequences when it is wrong?”
Step 5: Choose a fairness goal that fits the decision
Different uses require different fairness questions. For a reading aid, comparable accuracy across relevant writing styles or languages may be a practical goal. For a system influencing access to opportunities, the analysis may require additional examination of eligibility, error burdens and how decisions can be challenged.
Fairness criteria can conflict. Equalising one error rate does not guarantee that every other error rate or outcome will also match across groups. Dataset prevalence, thresholds and the costs of mistakes matter. This is why fairness cannot be reduced to a single universally correct equation.
Responsible teams define the criterion openly, investigate trade-offs and involve people with relevant technical, domain and lived expertise. They should not use a convenient metric to conceal harms that the metric does not measure.
Step 6: Improve both the data and the decision workflow
Possible interventions include repairing labels, adding properly obtained examples of neglected conditions, improving accessibility, revising task objectives, adjusting model training or changing the decision threshold with careful evaluation.
Sometimes the better solution is not a more complicated AI model. A poorly designed automated decision may need a clearer policy, a manual review pathway or the removal of an unreliable feature. An organisation should be willing to change the surrounding process, not merely retune the network.
After any intervention, retest for unintended regressions. Improving one subgroup’s outcomes should not silently introduce severe problems elsewhere. Use data not already exploited to select the repair.
Step 7: Keep monitoring after deployment
A fair result on a prepared test set cannot guarantee fair outcomes forever. New users, devices, data sources or operating conditions may alter performance. A model might also change which information it relies on after a version update.
Monitor error patterns over time with privacy-aware, representative evaluation. Examine complaints and appeals, and provide a workable path for human review. When a system affects people, visibility and recourse are important parts of reliability.
Tracking a change in overall accuracy without reviewing group-level outcomes can repeat the very mistake that made the problem hard to notice.
Try it: find the gap hidden by a simple average
Imagine a fictional classifier labels 100 easy photographs and 100 dim photographs. It gets 95 of the easy ones right and 65 of the dim ones right. Calculate the overall accuracy: (95 + 65) ÷ 200 = 80%.
Now write the two slice scores separately: 95% and 65%. Ask yourself which number a person taking photographs in poor lighting would find more useful. The answer shows why average performance is not enough.
Next propose a responsible investigation: examine camera conditions, training coverage and label accuracy before changing the model. This is a paper exercise; you do not need to collect real people’s private images.
Five misleading shortcuts
- “We removed sensitive attributes, so bias is impossible.” Other features can act as proxies.
- “The whole model is 95% accurate, so everyone benefits equally.” Averages can hide gaps.
- “A larger dataset solves fairness.” Larger imbalanced datasets can preserve imbalances.
- “One fairness score proves the job is done.” Criteria and harms differ by task.
- “A machine is neutral because it uses numbers.” Data, labels, objectives and deployment choices all come from human systems.
Frequently asked questions
Can a machine-learning model be biased without being deliberately programmed to discriminate?
Yes. Training examples, label definitions, sampling and deployment practices can produce systematic errors or unfair outcomes without an explicit discriminatory rule.
Does removing personal details guarantee fairness?
No. Correlated features and unequal data quality can remain. Privacy protection and fairness assessment are both necessary, but are not interchangeable.
How can a model be accurate overall but unreliable for one group?
The majority of test cases may come from conditions where the model performs well, obscuring a smaller group’s high error rate.
Is fairness the same as making every prediction identical?
No. Fairness concerns justifiable treatment and outcomes under a specified use case; it does not require treating different inputs as if they were the same.
Who should decide whether a model is fair?
The relevant stakeholders and accountable decision-makers, informed by evidence, domain expertise, affected groups and applicable standards. A technical metric alone cannot settle every question.
Your one-minute takeaway
- Training data and labels can contain gaps that average accuracy hides.
- Test meaningful slices, investigate differences and define the fairness goal in context.
- Responsible improvement involves data, models, human decisions and continuing review.
Trusted reading and connected chapters
- Google Machine Learning Crash Course: Identifying Bias — where imbalance and error gaps begin.
- Google: Mitigating Bias — changes to data and optimisation.
- Google: Fairness in Machine Learning — an accessible guided module.
- NIST AI Risk Management Framework — a broader approach to managing AI risks.
- How AI Learns From Examples — understand training and independent tests.
- How AI Personalises Study and Practice — why learner estimates should not become labels of ability.
- Classification Thresholds, Base Rates and Review Capacity — understanding error rates in decisions.