eduKate Singapore

VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Next useful routeCanonical hubHELP

How AI Works | Active Learning — Which Examples Are Worth Labelling Next?

—

by

eduKate Asia

Did you know that the next hundred examples used to train a model may matter more than a thousand examples selected carelessly? Active learning is the practice of choosing which currently unlabelled observations should receive trustworthy labels next. A model can identify records it finds uncertain, a selection policy can protect diversity and rare cases, and qualified people can supply or verify the answers. The aim is to improve the model’s useful performance per unit of annotation effort, not merely increase a dataset’s size.

The short answer is a loop: start with a small valid labelled set, train a baseline, score a separate unlabelled pool for possible information value, choose a constrained batch, obtain labels under a written annotation contract, retrain, and measure on independent evaluation data. That is the mechanism. It is not the same as automatically accepting the model’s guesses as truth. Nor does it guarantee that uncertain examples are more valuable in every setting.

Imagine a team classifying support requests into appropriate help routes. It has 1,200 unlabelled records and can fund only 150 new human reviews. Should it review the most confusing messages, the rarest situations, a broad cross-section, or a mixture? We will build a selection plan, check its arithmetic, and explain when apparently clever sample selection actually makes a system worse.

Contents and reading routes

1. What active learning solves—and what it does not

Ordinary supervised learning begins with labelled examples, such as messages and the category each message belongs to. Preparing reliable labels can demand expert time, contextual interpretation, privacy controls, and disagreement resolution. When an unlabelled pool is large but annotation is expensive, indiscriminate collection may waste effort on many almost identical easy examples while unusual cases remain neglected. Active learning asks a narrower, practical question: given a limited budget and an existing model, which observations deserve new human attention?

The mechanism is selective acquisition of training information. A model estimates something about candidate examples; a query strategy ranks or samples them; a human or trusted measurement process supplies their ground truth as far as that concept is meaningful; and the model is retrained. Many active-learning algorithms select one item at a time, but operational systems commonly select batches because experts need queues, staffing, and stable contracts. A useful label is not simply an assertion from a powerful language model; it depends on a clear target definition and appropriate evidence.

Active learning cannot rescue an undefined target, a faulty label specification, discriminatory source data, or an evaluation set contaminated by training. It may fail when the model is confidently wrong about a new population. Keep this chapter distinct from training-label quality, which explains whether a supplied label is sound; here we study which records should be sent for labelling in the first place.

2. Distinguish the unlabelled pool, training set and locked evaluation set

Begin with three boundaries. The training set contains records and labels the model may learn from. The unlabelled pool contains candidates from which reviewers can acquire new labels. The evaluation set is a separate, representative collection with independently checked outcomes; it is not a quarry to mine when the model struggles. If the same person’s messages or near-duplicate documents straddle those boundaries, reported gains may partly reflect recognition rather than generalisation.

Give every record a durable ID, source timestamp, purpose, consent or other valid processing basis, and provenance. This lets the selection algorithm exclude duplicates, records already labelled, records legally or ethically unsuitable for additional review, and records whose features would not exist at prediction time. Where examples arrive over time, freeze or version the candidate pool so later analysts know precisely what was eligible for the batch.

An active-learning experiment is especially vulnerable to leakage because its selection mechanism naturally looks at a model’s current errors or uncertainties. Looking at an ordinary unlabelled candidate’s predicted uncertainty is permitted within the experimental design. Looking repeatedly at the final evaluation labels to choose candidates or tune the strategy is not. The data-leakage guide provides the related split and time-boundary tests.

3. Set the learning objective before choosing a query score

“Improve accuracy” is underspecified. If the system helps route educational support messages, routing a safeguarding concern to an ordinary administrative queue can matter far more than confusing two harmless catalogue topics. Define target classes, what counts as a correct route, unacceptable errors, review capacity, subgroup coverage, and the cost of collecting labels. State whether improvements are sought on the overall population, a high-risk subset, or both.

A team may care about recall for rare but consequential concerns while keeping the total number of human escalations manageable. Active selection can deliberately sample underrepresented message types, but the final metric still needs to reflect the real operating population. Oversampling rare labels for training is not permission to report a rare-enriched test accuracy as ordinary-world performance. If real-world probabilities matter, measure calibration on appropriately representative independent data.

The choice of objective affects the query policy. For refining a simple categoriser, boundary uncertainty can be useful. For recognising a newly introduced service, coverage of unfamiliar vocabulary may matter more. For interpreting disputed reports, qualified adjudication might dominate model architecture. Do not optimise a mathematical acquisition score before checking whether it measures a user-relevant failure.

4. Uncertainty sampling: ask about cases near the decision boundary

The simplest active-learning strategy sends the model examples for which it appears least certain. In binary classification, a model output near 0.50 often represents more uncertainty than one near 0.99, provided those numbers are interpretable and the model is not miscalibrated. For multiclass tasks, practitioners use least confidence, the gap between the top two class probabilities, or predictive entropy. Each rule emphasises a slightly different notion of uncertainty.

For example, a two-class prediction of 0.50 and 0.50 has Shannon entropy of one bit. A prediction of 0.90 and 0.10 has approximately 0.469 bits, calculated as minus 0.9 times log base two of 0.9 minus 0.1 times log base two of 0.1. The first example looks more uncertain under that metric. Yet a high entropy score can arise from unreadable text, corrupted input, genuinely ambiguous ground truth, or an unfamiliar population—not only from an informative decision boundary.

Uncertainty sampling therefore needs an eligibility filter and diagnostic tags. Reviewers may label an ambiguous item “cannot determine from the available evidence” instead of inventing a category. A system should learn from that fact when its data contract permits, rather than forcing false certainty into the training set. A numerical uncertainty score ranks candidates; it does not decide what the correct human judgment must be.

5. Why confident mistakes are a problem for uncertainty-only selection

Did you know that a model can be most confident precisely where it has not learned the relevant distinction? Suppose a request uses unfamiliar local terminology. The model might confidently assign it to an ordinary category because every familiar neighbouring example had that label. A pure uncertainty strategy never sends it for review, so the mistake persists. This is often called a blind spot of the acquisition policy rather than an annotation mistake.

Keep at least one independent exploration channel that does not rely on the model’s present score. Uniform random samples, time-stratified sampling, new-source coverage, and known high-stakes categories can expose confident errors. The amount of exploration should be set by task risk and available review capacity, not a universal percentage copied from another project. Investigate the stability of acquisition scores across model versions as well.

A quality dashboard should count errors discovered by exploration separately from errors discovered by uncertainty ranking. Otherwise a team can celebrate that its selected batch became easier to classify while remaining ignorant of untouched parts of the population. An acquisition policy needs a way to discover when its own assumptions are wrong.

6. Diversity sampling: avoid buying the same lesson repeatedly

If a hundred nearly identical messages all sit near a decision boundary, ranking them by uncertainty alone can consume an entire human-review budget on one repeated pattern. Diversity criteria try to select candidates that spread across useful regions of a representation space, text clusters, geographic sources, time windows, or other justified dimensions. The goal is not to maximise novelty for entertainment; it is to avoid redundant information purchases.

Embedding distances can help, but their meaning depends on how the representation was trained and what differences it suppresses. Two messages can be numerically close yet differ in a legally or educationally crucial word. Conversely, a message can be far away because of formatting noise rather than new content. Combine numerical diversity with domain checks and an explicit deduplication rule. Document the threshold used for near-duplicate grouping so later comparisons are reproducible.

The 2020 research paper Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds presents BADGE, a batch method designed to combine uncertainty and diversity through gradient representations. That is one studied algorithm, not an instruction that its reported results will transfer unchanged to every problem. The broader principle is robust: information value depends on both what the model does not know and whether the selected examples add genuinely different evidence.

7. Rare-case and risk-weighted selection

Uncertainty and diversity do not automatically protect rare classes. A class that appears in fewer than one percent of records may be absent from the early labelled set and therefore invisible to the model’s class probabilities. A team can reserve a part of its budget for rare-source strata, known failure modes, emergent topics or cases that carry disproportionate real-world consequences. Selection can use metadata or expert screening that does not depend on already knowing the unknown label.

A risk-weighted policy still needs boundaries. Selecting cases because they involve a protected characteristic can create privacy or fairness problems unless the collection has a clear lawful purpose and appropriate safeguards. When demographic or sensitive fields are not necessary, use task-relevant sampling strata rather than collecting them speculatively. When fairness analysis requires group information, restrict access and explain the measurement rationale.

Rare-case enrichment is useful for model improvement but changes the composition of the labelled training batch. Evaluate on a separate representative set and, where relevant, a prespecified rare-case set. Report both results. A one-number score that averages everything together can hide a failing safety-critical category even when the model appears to improve overall.

8. Selection is a constrained optimisation problem

An acquisition policy is not just “sort candidates descending by score.” Each label can take different time, require a different expert, or involve an ambiguous record that will never yield the intended target. The candidate pool may also contain repeated authors, duplicated messages, disallowed private content, or records lacking required context. The best next batch under a strict one-hour review budget may differ from the best batch of a fixed one hundred records.

Represent each candidate with expected information value, review cost, eligibility, source or stratum, and known duplicate group. Select a batch whose total cost fits the budget and whose coverage constraints hold. A simple, transparent sequential rule can be preferable to a sophisticated opaque optimiser: reserve exploration capacity, prioritise high-risk gaps, choose uncertain diverse candidates, and deduplicate after each pass. More complicated ranking should demonstrate its value against that baseline.

All estimates are provisional. Annotation costs may differ from forecasts, and a disagreement may require extra review. Keep a contingency pool and state when work will stop. A plan that spends its whole budget on uncertain examples without allowance for adjudication or unlabelable records is not an honest operational plan.

9. A fictional 1,200-record support-routing laboratory

Consider an invented, privacy-screened pool of 1,200 support-request records. It contains 720 routine requests, 300 less common variations, and 180 records from recently introduced channels. These are candidate-source strata determined without knowledge of the target label. The total is 720 + 300 + 180 = 1,200. A separate locked evaluation set of 180 previously adjudicated records exists outside this pool and is not available for query selection.

The annotation team authorises up to 150 fresh review requests, with each record eligible at most once. It selects 60 by model uncertainty, then 45 additional diverse records not already selected, then 25 additional risk or rare-source coverage records, and finally 20 uniformly sampled from the remaining eligible pool. The batch size is 60 + 45 + 25 + 20 = 150 unique records. The order and exclusions matter: without deduplication, overlapping selection channels could produce 150 nominations but fewer than 150 distinct lessons.

This example deliberately mixes exploitation with exploration. Uncertainty helps refine the boundary, diversity avoids a batch of near-copies, rare-source coverage addresses explicit operational concerns, and uniform sampling checks confident blind spots. These proportions are illustrative, not published optimums. Real allocation requires comparison against alternatives under the same annotation budget and task distribution.

10. Check the annotation time as well as the number of records

Suppose the 60 uncertainty-selected records average five minutes per first-pass review, the 45 diverse records average three, the 25 rare-source records average six, and the 20 uniform-audit records average two. These are fictional measured times for this exercise. The time totals are 60 × 5 = 300 minutes, 45 × 3 = 135 minutes, 25 × 6 = 150 minutes, and 20 × 2 = 40 minutes. Adding them gives 625 minutes, or ten hours and 25 minutes of first-pass annotation.

The average first-pass time is 625 / 150 = approximately 4.17 minutes per selected record. If the team had budgeted 150 records at two minutes each, it would have been short by 325 minutes. This difference matters: active selection may target hard examples whose labels are expensive. A method that obtains greater accuracy per label may still be worse per expert hour if it overlooks review cost.

Adjudication, supervisor checks, privacy review, tooling setup and retraining time are not included in the 625 minutes. An honest cost comparison must add them when relevant. Do not claim a total project cost from a first-pass time figure. Distinguish human time, elapsed queue time and machine compute; one can improve while another deteriorates.

11. Reconcile review outcomes before fitting

Assume the 150 review requests produce 132 usable adjudicated labels, 12 items needing further adjudication and six items excluded because the evidence is inadequate or the records are out of scope. The status accounting is 132 + 12 + 6 = 150. Relative to requests, usable labels are 132 / 150 = 88%, pending adjudication 12 / 150 = 8%, and excluded 6 / 150 = 4%. The proportions sum to 100%.

If the 12 disputed items are not yet resolved, they do not become ordinary supervised labels just because the training deadline is near. The 132 usable records can be considered for the next training version after eligibility and grouping checks; the remaining 18 need distinct dispositions. Pending items may later be adjudicated. Excluded records remain informative about data quality but must not be forced into an arbitrary class for the sake of filling the target count.

The first-pass effort per currently usable label is 625 / 132 = approximately 4.73 minutes, excluding subsequent adjudication time. That cost is higher than 4.17 minutes per requested review because not every request yielded a usable label. This distinction often changes the apparent attractiveness of an acquisition algorithm. Measuring “examples selected” alone confuses an invitation to label with successfully acquired information.

12. Retrain with a versioned acquisition ledger

Store the baseline model ID, selection policy, candidate-pool snapshot, per-record score and rank, selected-channel reason, annotator agreement state, final approved label, and training inclusion flag. This is not bureaucratic decoration: without it a later analyst cannot explain why a rare example entered the dataset, whether a supposed improvement was caused by new labels or changed preprocessing, or whether an unexpected record was selected through a bug.

Retrain from a reproducible training set. All feature transforms that learn statistics—imputers, vocabularies, normalisers or embeddings fitted from task-specific data—must respect the training and evaluation boundaries. If the pool includes several messages from the same case, use appropriate grouping to avoid leaked near-duplicates across train and test. Freeze a model candidate and run the same prespecified evaluation procedure used for the previous baseline.

Run the next query round only after examining the current round’s outcomes. Active learning is sequential: as the model changes, the acquisition scores change, and the best examples may move. Reusing old uncertainty rankings indefinitely defeats part of the mechanism. Nonetheless, retraining after every individual label is not required; batch size is a practical trade-off among freshness, cost and operational stability.

13. Reading a small evaluation improvement without exaggeration

Suppose a previously fixed set of 180 independently labelled evaluation cases yielded 126 correct routes before the new batch and 135 correct routes after the updated model. Baseline accuracy is 126 / 180 = 70%, and updated accuracy is 135 / 180 = 75%. The improvement is nine additional correct classifications on this fixed set, or five percentage points, not “five percent more accuracy” in a relative sense. Relative to 70%, the increase is 5 / 70 = about 7.14%, a different quantity.

Those numbers are fictional illustrations, not empirical claims about active learning or any deployed model. More importantly, the comparison alone does not prove that the acquisition strategy caused the increase. Replacing the model architecture, changing the labels, tuning repeatedly on the same held-out set, or training longer could also shift performance. A causal comparison would hold such factors constant and compare selection policies across appropriate repeated splits or trials.

Also ask how the errors moved. If overall correct routes increased by nine but the most serious misroutes doubled, the system may be worse for its intended job. Track error types, class-specific recall, review load, and relevant subgroups. A single accuracy increase is a clue to investigate, not a universal verdict.

14. Why the test set should never be the query pool

An especially seductive error is to take failed evaluation cases and feed them directly into active learning. That might improve the next score on the same cases, but it turns evaluation into additional training and forfeits the independent test. Similarly, repeated strategy tweaks based on the final benchmark invite overfitting to the benchmark’s peculiarities. The right remedy is to reserve a development set for iterative decisions and keep a final audit set out of reach until the protocol calls for it.

Time and identity leakage also matter. If messages from the same learner appear under different identifiers, treating them as independent records inflates the effective sample size and rewards memorisation. If the model is expected to operate on future requests, use a split that respects prediction time. A record acquired after a policy change cannot be used to simulate an earlier deployment unless the features and labels would actually have been available then.

Preserve the provenance of each label and each evaluation result. The W3C PROV-O standard offers a formal vocabulary for entities, activities and agents in provenance records. Teams can implement a simpler local ledger while retaining the same discipline: do not lose the chain from original observation through selection, label creation, model version and measured outcome.

15. When selected samples distort what the model sees

Active learning intentionally creates a nonrandom labelled training sample. This is often useful for learning a decision boundary, but it complicates estimates of population prevalence and error rates. If a training batch heavily favours ambiguous messages, the percentage of “unclear” labels in the batch tells us little about how common ambiguity is across all user requests. Similarly, a rare-source quota is a coverage mechanism, not a population survey.

One response is to maintain a separate probability-sampled audit set, with inclusion probabilities recorded when inference about the wider pool is needed. In specialised research designs, weighting can help estimate population quantities, but weights only address what the sampling design and its assumptions support. They do not make absent observations magically representative or fix unknown label errors.

Explain selection context whenever sharing an active-learning dataset. A benchmark built from actively selected hard examples can be valuable for stress testing, but its raw error rate should not be mistaken for the production error rate. The denominator must name the population: all incoming requests, all acquired labels, all hard-case reviews, or all locked evaluation records.

16. Distribution shift and an expanding label vocabulary

Active learning is most effective when the pool contains candidates relevant to future work. If the service changes language, channel, region, or policy, an old pool may no longer represent upcoming cases. The model may appear stable because it samples only a familiar archive. Add new-source discovery and timestamp checks so the candidate universe tracks operational reality, while respecting privacy and purpose limitations.

A new class creates a particularly awkward problem. If the model was trained only on existing categories, it may force unfamiliar records into those categories with high confidence. Diversity filters, novelty detectors, metadata strata, and expert “none of the above” routes can surface evidence for a legitimate new class. Do not quietly reinterpret an old category after labels have been written; version the taxonomy and audit its downstream effects.

Distinguish concept change from ordinary hard examples. One confusing note may require human explanation; a sustained increase in a new kind of request may require revising the task definition. Active learning can reveal that a system’s original category map no longer serves users, but it cannot solve the governance choice of what the categories ought to mean.

17. Human review is a source of evidence, not a rubber stamp

An annotator needs the correct contextual material, a written instruction for borderline cases, the right to abstain, and a channel for escalation. Model-suggested labels can accelerate easy verification but can also anchor the reviewer toward the model’s mistake. If the goal is an independent reference label, design the interface so that independent first judgments are possible and measure whether suggested labels alter disagreement.

Review roles may differ. A first-pass annotator can handle ordinary messages; a qualified expert may resolve sensitive or technically ambiguous items; a separate auditor can test whether decisions match the annotation contract. Do not call an agreement percentage “truth.” Two reviewers can share the same mistaken interpretation. Keep specific evidence for reversals and rule changes so future model errors can be traced to their origin.

Sometimes the correct next action is to change the label task, not acquire more data. If two experts cannot consistently decide whether a message is an “academic concern” or an “administrative enquiry” because the organisation never defined the boundary, selecting a thousand more messages only multiplies disagreement. Repair the target ontology before the next acquisition round.

18. Confidence scores and entropy are not an oracle

A least-confidence ranking assumes some relationship between model scores and uncertainty about the true label. But model probabilities can be miscalibrated, especially outside their training distribution. Entropy can respond to correlated logits, unusual text length, noisy inputs or adversarial artifacts. A high score is a diagnostic signal that should be interpreted in context. It is not a statement that the sample is maximally valuable or that the annotator will know its true label.

Uncertainty strategies also depend on what ensemble, dropout approximation or predictive distribution the method uses. Different estimators rank the same pool differently. Re-run sensitivity checks and compare against simple baselines. If a complicated acquisition score requires substantial compute but selects almost the same records as a cheap uncertainty-plus-diversity rule, the added cost may not be justified.

Distinguish uncertainty about a class under the existing category definition from uncertainty about the definition itself. The first may be reduced by targeted examples; the second may require policy revision. That is a human decision, not merely an optimisation setting. The companion guide on classification thresholds addresses downstream decision boundaries and review capacity, which are not the same as choosing a training sample.

19. Batch size, stopping rules and diminishing returns

An acquisition strategy needs a stopping rule that exists before the team spends its entire budget. Possible conditions include a limit on expert hours, a prespecified change in a validation metric, saturation of rare-class coverage, or evidence that candidate labels are overwhelmingly redundant. Use a development metric appropriate to the task, then confirm important conclusions on independent evaluation. Do not stop solely because the model announces that it has learned enough.

Small sequential batches adapt quickly but incur repeated training and coordination costs. Large batches use expert capacity efficiently but risk selecting many correlated examples based on a stale model. The right size depends on model training cost, annotation turnaround, data heterogeneity and the rate at which the underlying system changes. Document why a batch is ten, one hundred or one thousand; there is no domain-independent ideal.

A well-run project can legitimately decide that more labels are not worthwhile. If evaluation improvements have plateaued, the bottleneck may be model design, feature availability, target inconsistency or distribution mismatch. Active learning is a method for allocating annotation effort, not a law that more active rounds must always improve performance.

20. Failure diagnosis: the obvious six traps

Trap one: every selected example is an almost identical ambiguous message. Diagnose low embedding or source diversity; repair with deduplication, coverage quotas and batch diversity. Trap two: rare high-impact mistakes never enter review. Diagnose overreliance on model uncertainty; repair with risk strata and independent exploration. Trap three: active-learning results look excellent on a test set that was repeatedly consulted. Diagnose evaluation leakage; restore a genuinely untouched evaluation process.

Trap four: labels become noisier as selected examples become harder. Review disagreement rates and the annotation contract; do not count unresolved guesses as reliable training labels. Trap five: annotation time doubles and budget is exhausted early. Measure expert minutes per usable label and redesign the cost-aware acquisition allocation. Trap six: a new intake channel is confidently misclassified. Inspect drift, pool construction and “unknown class” handling rather than assuming a high confidence score proves familiarity.

The repairs above address different layers: acquisition, annotation, model training, evaluation and governance. A good incident note identifies which layer actually failed, gives its evidence, and records a regression check. Labelling every failure “needs more data” is the machine-learning equivalent of replacing a thermometer whenever a room becomes too hot.

21. A changed-context transfer: identifying laboratory image anomalies

Transfer the workflow from text routing to a research archive with 3,000 unlabelled instrument images. Most are routine, some show rare physical artifacts, and some are corrupted by the sensor. Qualified scientists can inspect only 100 images this month. A raw uncertainty strategy may mostly select blurred corrupt frames because the model cannot identify anything. That would consume costly expert time without improving detection of meaningful anomalies.

Design a constrained acquisition plan: reserve 20 images for uniform random audit, 30 for diverse new instrument states, 30 for uncertain eligible images, and 20 for scientist-nominated high-impact anomalies. Deduplicate by observation session and near-identical frame. That is 20 + 30 + 30 + 20 = 100 unique requested inspections. Ask scientists to distinguish “valid anomaly,” “ordinary observation,” and “uninterpretable sensor artifact” under a written evidence standard.

The solution is to evaluate anomaly detection on a separately curated set covering both routine and rare cases, report performance by instrument state, and record inspection time. The model that improves hardest-case recall at the cost of a flood of false alarms may not serve the laboratory. The Astronomaly: Protege research is an example of human-machine prioritisation for astronomical inspection, not proof that this invented allocation is optimal.

22. What the research supports and leaves uncertain

The central active-learning idea has a long research history: selecting informative labels can reduce annotation effort under appropriate assumptions. The 2024 survey A Survey on Deep Active Learning: Recent Advances and New Frontiers organises query strategies, architectures and applications, while Deep Active Learning: Unified and Principled Method for Query and Training studies the uncertainty-diversity trade-off. Those papers are research evidence, not a guarantee of savings for a particular school, employer or model.

Performance comparisons depend on dataset, budget, architecture, class balance, label noise, the selected pool and the baseline. Some active strategies underperform simple random selection. A published result on benchmark images does not automatically transfer to confidential messages, school enquiries or changing real-world operations. The cost of selecting and verifying examples can outweigh gains from fewer raw labels.

The NIST AI Risk Management Framework provides a broader voluntary approach to governing and measuring AI risks, published in January 2023. It does not specify a universal best active-learning algorithm. Use it to frame lifecycle accountability, while using task-specific evidence to decide whether active acquisition is helpful.

23. A reusable operational review protocol

Before a new round, document the user decision the model supports; the eligible pool and exclusions; the previous model version; the acquisition rule and random seed; the label instructions and qualified reviewer roles; the maximum human hours; and the separate evaluation plan. After selection, reconcile unique record IDs, selection channels, review outcomes, actual time, training inclusion, and any sensitive-data exceptions. After training, compare against the prespecified baseline and error categories.

On a dashboard, show the number requested for review, number completed, number accepted as training labels, number unresolved, expert hours spent, and quality changes on independent evaluation. Do not silently replace the denominator “150 requests” with “132 usable labels” in a percentage without naming it. Check that the improvement is not merely a feature of an easy changed test sample or an accidentally leaked group.

The operational principle is straightforward: ask humans to examine evidence where it is likely to help, remain able to discover what the model has missed, and prove any claimed improvement on records the model did not use as teaching material. That is a more durable lesson than memorising a fashionable acquisition algorithm.

Frequently asked questions

Does active learning mean AI teaches itself without people? No. It usually chooses candidates for an external labelling source such as a person, laboratory measurement or validated database. Treat model-generated labels as hypotheses unless separately justified.

Why not always label the most uncertain example? A high score can reflect noise, ambiguity or corruption, and consecutive uncertain items may be nearly identical. Balance uncertainty with eligibility, diversity, risk and exploration.

Can active learning improve a small dataset? Sometimes. Results depend on how representative the unlabelled pool is, whether labels are reliable, and the strength of simple baselines. A small pilot can reveal the actual cost and value.

Does it make the test set unnecessary? Quite the opposite. Selective training makes an independent, properly constructed evaluation set more important. Never repeatedly consume final test errors as the acquisition queue.

How many items should a batch contain? There is no universal number. Balance annotation staffing, model update cost, candidate redundancy, review urgency and distribution change; compare alternatives under equal resource budgets.

What should happen to unlabelable records? Record the reason and route to adjudication or exclusion according to the contract. Do not invent a forced class just to make a selection policy look efficient.

Is active learning the same as reinforcement learning? No. Active learning chooses information to label for supervised model improvement. Reinforcement learning optimises behaviour from rewards or feedback within an environment; their mechanisms and evaluation questions differ.

Will an active-learning system automatically become fairer? No. It can oversample existing model blind spots or neglect people it already classifies confidently but wrongly. Coverage planning, privacy safeguards and subgroup evaluation require explicit human decisions.

Primary research and learning pathways

Read Ash et al., Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds (2019 preprint; ICLR 2020), Shui et al., Deep Active Learning: Unified and Principled Method for Query and Training (AISTATS 2020), and Li et al., A Survey on Deep Active Learning (2024). For an applied scientific example see Astronomaly: Protege (2024). Use the original papers to distinguish tested findings from the invented teaching calculations above.

For the wider series continue to How AI Works, then Missing Data for observation problems, Training Label Quality for annotation rules, and Data Leakage for honest evaluation boundaries.

Students and parents can connect these ideas to learning that transfers into life: the central habit is deciding what evidence to gather next, checking its reliability, and comparing the result against a question defined in advance.

Discover more from eduKate Singapore

Subscribe now to keep reading and get access to the full archive.

Continue reading