Did you know an AI classifier can be mathematically accurate on the examples it was given and still make systematically poor decisions for people or situations not represented in those examples? A machine-learning model learns patterns from data; it does not automatically know whether the training records are complete, fair or appropriate. Bias can enter before the model is built, while records are labelled, or when the output is used without suitable human judgement.
Ministry of Education V3.0 Reference 384 is a focused study guide to the 2027 Singapore–Cambridge SEC G3 Computing K349 outcome 5.4.5: explaining how unethical AI use or biased data can lead to negative consequences. It uses fictional classifying systems to show selection bias, measurement error, inaccurate labels, unequal error rates, proxy variables, changing populations and misuse of predictions. The aim is to teach students to question a model’s evidence, purpose and impact rather than assume a high accuracy number settles every ethical issue.
This article is an independent eduKateSG educational reference, not an official MOE, SEAB, Cambridge, IMDA or PDPC publication. All people, datasets and AI decisions here are fictional, and no real person’s identity or characteristics are evaluated. The official K349 syllabus defines the examination requirements. Singapore’s IMDA Model AI Governance Framework and PDPC guidance offer broader governance context, not a universal certification that any particular deployed model is fair.
1. Read the official learning outcome accurately
The 2027 K349 syllabus describes AI as the ability of a computer to perform complex tasks without constant human guidance, gives examples including face and voice recognition, image classification and spam filtering, and introduces machine learning as a technique within AI. Outcome 5.4.5 explicitly asks students to explain negative consequences from unethical use or biased data.
This article complements Reference 360, which explains nearest-neighbour classification, rather than repeating its distance calculations. Here the question is different: even when a classifier follows its mathematical rule correctly, should its inputs, labels, error patterns, use and consequences be trusted? A clear answer identifies the mechanism producing harm.
2. Data are examples, not universal truth
A machine-learning system is often trained or configured using collected examples. Those examples represent a limited history of observations and decisions; they do not necessarily cover every context in which the system may later operate. If the data systematically omit a class of ordinary situations, predictions can be poor when those situations arise.
Suppose a fictional recycling sorter was trained mostly on photographs taken under bright lights. In a dim environment its image classifications might become less reliable. No malicious intention is required for that data mismatch. The correct explanation identifies an unrepresentative source, the change in conditions and the resulting error risk.
3. Sampling bias means who or what was recorded matters
Sampling bias occurs when the examples used for development do not adequately represent the intended population or range of conditions. The model may learn a pattern that works well for a selected subset while failing elsewhere. Collecting more records of the same narrow type does not necessarily correct that representation gap.
A classroom model trained only on short audio recordings in a quiet room may work poorly on longer recordings in noisy surroundings. The relevant remedy begins by defining its real intended users and environments, then testing representative conditions. It is not enough to announce that the model had many training examples.
4. Measurement bias changes what the variables mean
Features can be measured inconsistently or with different accuracy across groups or situations. One camera may record colours differently from another, or a sensor may perform poorly in particular temperatures. A model using those measurements can learn patterns that reflect the measurement process rather than the intended object.
Consider a fictional sports activity tracker that records steps more reliably for one type of movement than another. A difference in recorded numbers might reflect the sensor’s limitations, not actual activity. A careful evaluation separates measurement error from genuine differences and avoids drawing conclusions beyond the instrument’s validity.
5. Labels can contain human mistakes
Supervised learning often uses labels that describe the supposed correct categories. Those labels may contain errors, inconsistent criteria or subjective judgements. A perfectly implemented algorithm can reproduce a mistaken label when a new input resembles that incorrectly categorised example.
For a fictional recycling dataset, one plastic object is mistakenly labelled metal. An adjacent example might inherit the wrong category in a nearest-neighbour method. The mathematical distance remains correct; the training record does not. Checking labels is therefore a different task from verifying arithmetic.
6. Historical decisions may encode past unfairness
An organisation might use historical administrative decisions as training labels without considering whether those decisions were themselves appropriate. If earlier choices disadvantaged some people or conditions, a system can reproduce the pattern when the labels are treated as a standard to imitate.
An invented application-screening dataset might contain past decisions made under inconsistent criteria. The student should not infer that copying those outcomes automatically makes the system objective. Historical frequency does not establish ethical legitimacy. The governance question concerns how labels were created and whether the present purpose justifies using them.
7. Class imbalance can hide poor performance
A dataset may contain many examples from one category and very few from another. A classifier can achieve a high overall accuracy by predicting the common class frequently while missing important rare cases. The numerical result must be interpreted against the distribution of actual outcomes.
If 95 of 100 fictional messages are legitimate, a system that always declares ‘legitimate’ obtains 95 per cent accuracy but catches none of the five unwanted messages. This is an illustrative mathematical warning, not evidence of the performance of any real spam filter. Students should look beyond the headline accuracy figure.
8. False positives and false negatives have different consequences
A false positive happens when a system identifies a condition that is not actually present; a false negative happens when it misses a condition that is present. The cost of the two mistakes can differ substantially according to the application. Choosing one threshold therefore involves more than maximising an average score.
A fictional spam filter wrongly blocks an important legitimate message, which is a false positive for spam. Another harmful message passes through, a false negative. A balanced discussion states both risks and what recovery or review path the user has, instead of simply calling the system accurate or inaccurate.
9. Proxies can reproduce sensitive patterns
A model may use a seemingly ordinary feature that correlates with another attribute of concern. For example, a fictional service might rely on location categories that correlate strongly with access to resources. Removing one explicitly sensitive field does not automatically remove every indirect route through which similar patterns can affect decisions.
This does not mean that every geographic feature is inherently discriminatory or that a correlation proves intentional unfairness. The appropriate question is whether a feature is relevant to the legitimate task and whether its use creates unjustified differences in outcomes. A credible evaluation needs data and context.
10. Context determines what fairness means
Different applications involve different decisions and possible harms. A harmless fictional object-sorting exercise and a system influencing access to essential opportunities do not have the same stakes. Claims of fairness require criteria: which outcomes, error patterns or treatment differences are being examined, and for whose benefit?
There is no one universally sufficient fairness number. A learner can nevertheless explain why unequal error rates across relevant conditions warrant further investigation, especially where a person could be harmed. The goal is to ask evidence-based questions without assuming every numerical difference proves wrongdoing.
11. Accuracy on training data is not deployment accuracy
A system can learn patterns that fit its training records but perform worse on new examples. Testing on appropriate separate data helps estimate how it behaves beyond familiar cases. The evidence must reflect intended operating conditions rather than only repetitions of material the model already knows.
Suppose a classifier recognises product images from one photo studio but fails on images submitted by ordinary users. A good evaluation includes different lighting, angles and cameras. Students should name the unseen condition and the effect it has rather than assuming a training success guarantees future reliability.
12. Data can change after deployment
Populations, language habits, devices and environments can change over time. A model trained on old data may become less suitable when the patterns relevant to its task shift. Such change is often discussed as data or concept drift, depending on what changes.
A fictional voice-recognition system trained only on a limited range of recording equipment may perform poorly after widespread adoption of a new input device. Monitoring and appropriate retesting can identify problems. This does not establish that every model needs the same retraining schedule; the risk depends on its use.
13. Privacy and lawful data collection still matter
Collecting more data to improve an AI model can create privacy and security questions, particularly when records identify individuals. A developer should consider whether the information is necessary, what purposes apply, how it is protected and whether applicable rules allow its collection and use.
The PDPC has published guidance on using personal data in AI recommendation and decision systems. It does not follow that every dataset must be made public for transparency or that a model’s statistical value overrides individual data rights. Responsible data practices and useful model development must be balanced under the relevant legal and ethical framework.
14. Human oversight must be meaningful
A person reviewing an AI result should have sufficient information, authority and time to challenge it when necessary. Merely placing a human’s name on a decision does not guarantee effective oversight if the system’s output is treated as automatically correct.
For a fictional service that recommends how to sort applications, a reviewer could examine relevant evidence, correct errors and explain why a recommendation was or was not followed. The important distinction is between human judgement supported by computing and a person rubber-stamping an automated decision without understanding its limits.
15. Explainability and transparency support accountability
Explainability concerns helping relevant people understand why an output was reached, within the system’s actual capabilities. Transparency concerns giving appropriate information about how the AI is used, its role in decisions and its limitations. These are related but not identical requirements.
An organisation could tell users that AI assists triage while providing no meaningful route to question an inaccurate result. A more accountable process states the AI’s role, identifies responsible decision-makers and allows feedback or correction when appropriate. IMDA’s AI governance guidance emphasises explainable, transparent, fair and human-centric practice.
16. Testing should reflect potential harm
A system used for low-stakes content sorting can tolerate different error consequences from one making important decisions about people. Testing should therefore consider whom the output affects, which mistakes matter most, and whether users can recover from errors.
A fictional photo app incorrectly labels a flower; the inconvenience may be small. A system incorrectly flagging a personal record for exclusion can have much larger consequences. The ethical explanation should identify the downstream effect rather than treat every classifier’s error as equally important.
17. Unethical use can be separate from biased training
A system can be trained on reasonably representative data yet deployed for an inappropriate purpose, such as surveillance or manipulation without suitable permission. Another system can be intended for a legitimate task but rely on biased data. The syllabus explicitly mentions unethical use and biased data as separate causes of negative consequences.
A strong student provides an example of both: misuse through an unjustified application and poor outcomes due to unrepresentative or incorrect examples. The two mechanisms can overlap. Neither is resolved by merely increasing the number of training samples without examining purpose and governance.
18. A complete answer focuses on cause, effect and safeguard
The most useful SEC answer names a particular source of bias or unethical use, explains how it changes the model’s output, identifies who may be affected and states an appropriate mitigation. A description such as ‘AI might be unfair’ is too vague without a mechanism or consequence.
For example, training a spam filter only on one language variety may cause inappropriate classification of messages written differently. Testing on a wider relevant set, reviewing labels and providing a way to recover incorrectly blocked messages address different parts of the problem. A responsible conclusion still recognises that no test proves universal fairness.
19. Twelve-stage Bias–Evidence–Impact protocol
This is an independent eduKateSG teaching method, not a formal AI audit or an official examination marking scheme. It helps students explain an AI risk through a real mechanism and an identifiable consequence rather than a vague claim that a computer is biased. All records and people in the exercises are fictional.
Stage 1 — State the AI task
Identify exactly what the system classifies, predicts or recommends. A model’s task must be clear before its fairness can be discussed.
Stage 2 — Name the intended users or situations
Describe the population and environment where the model is supposed to work. A sample can only be judged representative relative to a purpose.
Stage 3 — Inspect how examples were selected
Ask who or what appears in the training data and which relevant conditions or experiences may be missing.
Stage 4 — Inspect how features were measured
Identify sensors, forms and recording practices that could produce systematically different measurements.
Stage 5 — Inspect the labels
Ask who decided the correct categories and whether errors, inconsistency or earlier unfair decisions entered those records.
Stage 6 — Check for proxy variables
A feature may indirectly correlate with another characteristic of concern; its legitimate role and consequences require evaluation.
Stage 7 — Distinguish common and rare cases
Do not let a high overall accuracy number conceal poor performance for a less common but important class.
Stage 8 — Compare error types
Identify false positives and false negatives and what happens to the person or task when either occurs.
Stage 9 — Check unfamiliar or changed conditions
Testing only the training examples gives limited evidence; real deployment can involve different users, devices and environments.
Stage 10 — Identify privacy and permission questions
Consider whether collection and use of personal data are appropriate and protected under applicable rules.
Stage 11 — Name meaningful oversight
Explain how a responsible person can understand, challenge and correct consequential AI results.
Stage 12 — Conclude with mitigations and limits
State the bias or misuse mechanism, a plausible harm, a suitable test or safeguard, and the fact that no single metric proves universal safety.
20. Twenty-four original ethical AI worked cases
Case 1 — Only bright photographs
A fictional recycling classifier trains exclusively on objects photographed under bright studio lights. Its dim-room mistakes can follow from missing environmental examples rather than a programming syntax error.
Case 2 — Only one recording condition
A voice classifier learns from quiet microphones but is used with noisy phones. Differences in measurement conditions can harm reliability even when the program executes as written.
Case 3 — One mislabeled object
A training record calls a plastic item metal. A nearest-neighbour result influenced by that record can be mathematically consistent but factually incorrect because the label is wrong.
Case 4 — Inconsistent annotators
Two reviewers apply different criteria when assigning categories. The model may learn inconsistency rather than a stable, meaningful decision rule. Clear labelling instructions matter.
Case 5 — Class imbalance
Of 100 invented messages, 95 are legitimate and five are spam. A model predicting legitimate for all achieves 95% accuracy but detects none of the actual spam.
Case 6 — A blocked genuine message
A spam filter labels a legitimate document as spam. This is a false positive for spam, which may prevent the user seeing important information.
Case 7 — A missed unwanted message
A spam filter allows a genuine spam message through. This is a false negative for spam, with a different practical consequence from blocking an authorised message.
Case 8 — A misleading proxy
A fictional selection model uses residential distance as a proxy for another relevant characteristic. The input may create unjustified outcome differences and needs task-specific scrutiny.
Case 9 — A missing class of products
A sorter contains almost no examples of transparent containers. Poor predictions for that class can reflect an underrepresented training set, not a universal inability of AI.
Case 10 — Many examples from one source
Adding thousands of nearly identical records from the same studio can increase dataset size without addressing missing lighting, angle or device conditions.
Case 11 — Historical labels treated as correct
A model learns from old decisions made under inconsistent criteria. Reproducing those decisions is not automatic evidence that the new system is fair.
Case 12 — High training accuracy
A classifier achieves an impressive score on records it already learned. That result does not establish performance on suitable unseen inputs.
Case 13 — Different deployment population
A tool is trained on one narrow user population and deployed elsewhere without evaluation. The mismatch can cause uneven performance, requiring appropriate testing.
Case 14 — A changed device
A new camera changes how images appear. A deployed model may need monitoring and testing because its input distribution changed.
Case 15 — An invented personal-data profile
An operator proposes collecting extensive identifiable information without a clearly justified purpose. Model improvement alone does not remove data-protection concerns.
Case 16 — Unnecessary sensitive features
A classifier includes personal attributes unrelated to the legitimate task. The designer should question necessity and possible inappropriate use before building the model.
Case 17 — Removing one column
A team deletes one explicitly sensitive field but leaves a closely correlated proxy. The effect of the proxy must still be assessed rather than assuming automatic fairness.
Case 18 — Two error rates differ
A model makes substantially more mistakes under one relevant condition. The observed difference warrants investigation, although it does not by itself identify its cause or prove intent.
Case 19 — An automatic rejection
A fictional service excludes a user based only on a model output, without a practical review process. The consequence can be serious despite the software producing a valid score.
Case 20 — A human rubber-stamp
A human appears in the approval workflow but cannot meaningfully question the result. A nominal human presence is not the same as effective oversight.
Case 21 — A transparent explanation
A service tells users that AI recommends a category, explains the key evidence and offers appropriate correction channels. These are useful accountability measures, not a guarantee of perfect prediction.
Case 22 — A harmless low-stakes classification
An app incorrectly labels a fictional flower. The inconvenience differs from an error that would affect an individual’s access to an essential opportunity.
Case 23 — An unethical intended use
A system with good test performance is deployed for an unjustified surveillance purpose. The ethical problem comes from how it is used, not necessarily from biased training labels.
Case 24 — A responsible improvement
A team checks labels, broadens relevant training conditions, evaluates errors by subgroup or setting, protects personal data and preserves a meaningful review process. Each safeguard addresses a different risk.
21. The fictional recycling classifier audit
Imagine a class creates an image classifier for paper, plastic and metal. The input photographs are from one classroom desk under the same lighting. In a public exhibition space, the lighting and camera differ, and the model becomes less reliable. The learner should identify the data limitation and propose a test using varied, relevant photographs without pretending that more identical desk images automatically solve the problem.
Next insert an incorrectly labelled training object. Ask why an accurate Euclidean distance or another classification calculation cannot repair a wrong category supplied during training. Separate computational implementation from label quality. A complete response should explain both selection bias and labelling mistakes, then state what testing would reveal.
22. A high score with a hidden weakness
Take 100 synthetic messages: 95 legitimate and five unwanted spam messages. A classifier that calls every message legitimate obtains 95 correct predictions, but it catches none of the unwanted class. The student should distinguish overall accuracy from the actual objective. If the application is meant to filter spam, this high percentage disguises a major failure.
Now imagine a stricter filter catches more spam but also blocks legitimate course notices. Those are false positives for spam and create another cost. The best threshold depends on the intended task and the relative impact of errors. An ethical explanation identifies the users affected and a suitable recovery or review mechanism.
23. A personal-data collection decision
A fictional online tutoring service proposes using identifiable pupil records to improve an AI recommendation engine. The service must first consider the specific purpose, relevant permission or legal basis, protection, data minimisation and retention under applicable law. Technical usefulness does not make collection automatically appropriate, and publishing the records as an open dataset would introduce different risks.
The PDPC’s guidance on AI recommendation and decision systems is relevant for a full legal assessment. In a classroom exercise, students should name the obligations without declaring a real operator compliant or in breach from an imaginary example. Protecting personal information and maintaining useful, representative data are complementary responsibilities.
24. A meaningful human review scenario
Suppose an AI system recommends categories for fictional requests, and one request is mistakenly rejected. A human reviewer who has access to relevant evidence and genuine authority to correct the result may mitigate harm. Merely placing a person’s approval signature beside an opaque recommendation is not necessarily meaningful oversight.
IMDA’s governance guidance emphasises human-centric and accountable systems, with explainability, transparency and fairness. Students should explain how users could receive an understandable notice and seek correction, not claim that a label saying ‘human reviewed’ proves every error was addressed.
25. Frequently asked questions
What makes training data biased?
A systematic selection, measurement or labelling problem that makes the examples unsuitable or unfair for the intended task.
Can a high accuracy score hide poor performance?
Yes. A dominant class can make the overall percentage impressive while rare but important cases are missed.
What is a false positive?
A positive class is predicted when the condition is not actually present, such as a legitimate message wrongly marked as spam.
What is a false negative?
The model misses a real positive case, such as a spam message incorrectly allowed through.
Does correct arithmetic guarantee correct AI classifications?
No. Unrepresentative examples, incorrect labels or unsuitable features can produce bad decisions despite correct calculations.
Does removing one sensitive column guarantee fairness?
No. Other variables can act as proxies or the data can retain relevant historical and measurement biases.
Do more data always solve underrepresentation?
No. Adding similar observations from the same narrow source may leave the important missing conditions unrepresented.
Can AI be used unethically without biased training?
Yes. Deployment for an inappropriate or harmful purpose can be unethical even if a model’s tests are technically strong.
Does a human reviewer guarantee fairness?
No. Oversight must be meaningful, informed and able to challenge and correct outcomes.
Why does deployment context matter?
The intended users, equipment and conditions may differ from the training records, changing error rates or other behaviours.
What does the PDPA have to do with AI?
Using identifiable personal data to develop or deploy systems can raise legal obligations about collection, use, protection and retention.
What should a strong SEC answer contain?
Identify one bias or misuse mechanism, its negative consequence and a relevant mitigation while acknowledging uncertainty.
26. Official syllabus and Singapore AI governance sources
The authoritative examination source is the 2027 SEC G3 Computing K349 syllabus, especially 5.4.5. The IMDA AI governance overview describes explainable, transparent, fair and human-centric AI, and the PDPC guidelines for personal data in AI recommendation and decision systems cover relevant data-use principles. Continue with Reference 360 — AI Nearest Neighbour, Reference 383 — Security Safeguards, the MOE V3.0 reference library and the immutable Clementi teaching standard.
Reviewed 2 November 2026 in Singapore. All people and datasets are fictional examples. This is educational analysis, not a certification of any real AI system or legal advice. The lasting habit is to connect data bias and unethical use to concrete errors, affected people and meaningful safeguards.