eduKate Singapore

VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Next useful routeCanonical hubHELP

How AI Works | Positive–Unlabeled Learning — Hidden Positives, Selection Bias and Honest Evaluation

—

by

eduKate Asia

A dataset with confirmed positives and everything else marked ‘unknown’ is not an ordinary positive-versus-negative classification dataset. Positive–unlabeled learning, often shortened to PU learning, addresses this exact missing-negative problem: it tries to distinguish a target class when only some positives have been verified and the unlabeled pool contains both undiscovered positives and genuine negatives. The first rule is simple: absence of a positive label is not proof of a negative case.

PU learning can work, but its success depends on how examples were selected for labeling, how many true positives may be hiding among the unknown cases, and whether evaluation uses independently established ground truth. If the identified positives are systematically easier to find than the unidentified ones, naïve correction formulas can be misleading. The useful question is not ‘which model handles blank labels?’ It is ‘what evidence and assumptions make this model’s claims identifiable?’

Contents

Why this problem appears in real datasets

Consider a research database containing compounds already confirmed to interact with a protein. The remaining compounds are not automatically non-interacting; many have never been tested. A fraud investigation database can have verified fraudulent claims and a very large collection of claims not conclusively reviewed. A school may know which learners were formally referred for a particular support service, but the non-referred learners are not necessarily confirmed free of that need. Each dataset combines a verified detection process with an unknown remainder.

Traditional binary classification expects the observed label to describe the target. A positive case is y=1 and a negative case is y=0. Positive–unlabeled data instead contain an observed selection label s: one when an example has been confirmed or selected as positive, zero when it has not been selected. A zero in s means ‘not confirmed positive in this dataset’, not ‘target absent’. This small distinction changes the training objective, validation procedure and interpretation of every headline metric.

A data scientist who maps every unreviewed claim to a negative class may create thousands of false negatives before fitting any model. The algorithm may then learn to imitate the existing review workflow, not recognize underlying fraud. A powerful model can amplify rather than repair this label-design error. The goal of PU learning is not to guess all hidden truths; it is to make disciplined inferences from an incomplete observation process.

Two labels, two different questions

Use y for the unknown true condition and s for the observed positive-confirmation indicator. By construction in a clean PU problem, s=1 implies y=1: a labeled positive should really be positive. The reverse does not follow. A true positive may have s=0 because nobody observed, tested or verified it. This asymmetric label structure is what distinguishes PU learning from arbitrary missing labels or ordinary label noise.

Ask the dataset owner two separate questions. First, what is a positive example in the real world? Second, what event caused an example to be marked as a known positive? A positive molecular assay result, a human-reviewed incident and a legal finding are different confirmation mechanisms. Each can have false results; if the ‘confirmed positive’ label itself is unreliable, the basic PU assumption is violated and the problem is more complicated than standard PU learning.

Keep y and s separate in variable names and diagrams. Do not call the observed s=0 rows ‘negative examples’ in dashboards simply because a library expects a zero or minus-one integer. Explain that these are unlabeled cases. That precise language makes it easier to detect when a classifier’s apparently high score is only high at predicting which cases entered the review queue.

How the labeling mechanism shapes the answer

Imagine two real positive cases. One has obvious features and attracts human reviewers; the other is subtle and remains unnoticed. If the first type is much more likely to be labeled, the confirmed-positive sample does not represent all positives. A model trained on confirmed positives may become a model of what attracts reviewers, not what defines the underlying phenomenon. This is selection bias inside the label acquisition process.

The classic simplifying assumption is selected completely at random, or SCAR. Among true positives, every example has the same probability c of being selected for the observed positive set, independent of the measured features. This does not say examples are selected randomly from all cases. It says the subset selected from the positive population is representative of that population. Under SCAR, the set of confirmed positives can teach us what positives look like, subject to statistical uncertainty.

SCAR is a mathematical assumption, not a common property guaranteed by the word ‘unlabeled’. Health records, enforcement datasets, customer complaints and teacher referrals may violate it because severity, visibility, geography or access changes the probability of confirmation. An alternative selected-at-random model may allow labeling propensity to depend on measured covariates, but that requires richer data and stronger modeling. When selection depends on hidden factors, the inference may remain non-identifiable without new evidence.

The central probability relationship

Let p(x)=P(y=1 given x) denote the true positive probability for features x. Let q(x)=P(s=1 given x) denote the probability an item is confirmed positive in the available data. Under SCAR, with c=P(s=1 given y=1), the chain rule and the assumption that negatives are never labeled positive give q(x)=c × p(x). The model predicting confirmation q is therefore a scaled view of the probability of genuine positivity p. The constant c matters.

Suppose a calibrated selection model assigns q(x)=0.24 to a case, and independent evidence supports c=0.60. The implied positive probability under SCAR is p(x)=0.24/0.60=0.40. That arithmetic is valid only under the assumptions and calibration conditions just described. A raw boosted-tree score is not automatically an accurate probability, and dividing it by c does not magically make it one. The corrected value must lie between zero and one; if q/c exceeds one, at least the fitted quantities or assumptions need attention.

This relationship appears in the 2008 Elkan–Noto study, Learning classifiers from only positive and unlabeled data. The point is not to mechanically apply a ratio whenever negative labels are missing. It is to know exactly what sampling conditions make the ratio meaningful and what independent information identifies the selection fraction c.

A complete fictional population example

Take 1,000 fictional support requests. After an independent full audit, suppose the hidden truth is 250 requests genuinely require specialist attention and 750 do not. The historical triage system positively confirmed 150 of the 250 real positives. This means the confirmation fraction among positives is c=150/250=0.60. The remaining 850 unlabeled records comprise 100 genuinely positive cases and 750 genuinely negative cases. The unlabeled pool is therefore 100/850≈11.76 percent positive, not zero percent.

The observed positive proportion is 150/1000=0.15. Given c=0.60, the implied true prevalence is π=0.15/0.60=0.25, which matches the hypothetical audit: 250/1000=25 percent. The labeled positive count is not the true incidence. Nor is 100/850 the population prevalence. That denominator describes the mixture inside the unlabeled set; 250/1000 describes the true population prevalence. Confusing those quantities would distort policy choices.

This is an illustrative audited example, not a claim about any real school, organisation or population. In operational use we would not know the 250 without additional labels or assumptions. The example exists to separate observable counts from unobservable truth and to make the required pieces of evidence visible. The arithmetic can be checked by summing 150 confirmed positive + 100 hidden positive + 750 true negative = 1,000.

How a misleading precision score arises

Continue the same fictional population. Suppose a candidate classifier predicts positive for 105 of the 150 confirmed positives, 65 of the 100 hidden positives, and 45 of the 750 genuine negatives. It predicts positive for 105+65+45=215 cases. Against the audited truth, true positives equal 105+65=170 and false positives equal 45. True precision is 170/215≈79.07 percent and true recall is 170/250=68 percent.

If an analyst incorrectly treats all unlabeled cases as negative, the confusion matrix changes. The 65 recovered hidden positives are counted as false positives alongside the 45 actual negatives. The apparent true positives are only 105, apparent false positives 110, apparent false negatives 45, and apparent true negatives 740. The naïve precision becomes 105/215≈48.84 percent; naïve recall becomes 105/150=70 percent. The second evaluation is about matching the confirmation indicator s, not identifying the real target y.

This is not a rounding nuisance. The naïve precision drops by more than thirty percentage points even though the classifier found many previously unknown true positives. In discovery workflows, rewarding a model only for rediscovering confirmed cases can penalize exactly the novelty the project seeks. State which label is used in the metric and how independently verified the denominator is before treating a score as model quality.

Can class prevalence be recovered from unlabeled data?

Some PU methods estimate a class prior: the fraction π of the overall population that is positive, or α, the fraction of positives inside an unlabeled mixture. These are related but different parameters. Estimating them can support thresholds, expected review workload and risk correction. Yet prevalence cannot simply be read from the number of confirmed positives, because the historical labeling fraction is unknown in most real settings.

In the fictional 1,000-case sample, the observed 150 positive confirmations are consistent with several different hidden populations. If c were 0.50, the implied π would be 0.30. If c were 0.75, π would be 0.20. Both produce an observed confirmation rate of 0.15 under SCAR. Without a justified value for c or enough information to identify it, the data do not distinguish those scenarios. A system that displays π=25 percent with decimal precision but cannot explain its c estimate is giving false certainty.

Class-prior estimation approaches rely on additional structure, such as identifiable mixture components, anchor-like regions, irreducibility conditions, external validation or reliable assumptions about selection. No single recipe works for every dataset. Report how the prior was obtained, a plausible range, and how model decisions change across that range. An uncertain class prior should be treated as a sensitivity variable rather than a hidden constant inside the code.

Three defensible training strategies

One approach trains a model to distinguish labeled positives from unlabeled cases, then uses a justified selection-probability correction. This can be practical under SCAR and is conceptually transparent, but it requires care with calibration and estimation of c. A second approach treats selected portions of the unlabeled set as reliable or likely negatives, perhaps with iterative refinement. That can be useful when expert review is scarce, but mistakenly choosing hidden positives as negatives creates its own feedback loop.

A third family estimates classification risk directly from positive and unlabeled distributions using a class prior. Under suitable assumptions, the expected risk can be expressed through expectations over positives and the unlabeled mixture, allowing learning without explicit negative labels. With flexible models, an unbiased empirical estimator may become negative and drive severe overfitting. The non-negative PU risk approach of Kiryo and collaborators was designed to address that failure; see Positive-Unlabeled Learning with Non-Negative Risk Estimator.

The best method depends on what is genuinely known. If a small but reliable sample of negatives can be obtained through targeted investigation, a standard supervised baseline might outperform elaborate PU machinery and be easier to validate. An honest comparison includes that baseline. Complexity is not a mark of seriousness; assumptions and evidence are.

Why non-negative risk matters in flexible models

In ordinary supervised learning, a loss represents a cost of prediction mistakes, and expected loss should not be negative for common non-negative losses. PU risk decompositions can include a subtraction term that estimates the negative-class contribution from positive and unlabeled samples. On finite samples, this empirical subtraction can make the estimated quantity negative even though the population risk is not. A flexible model can exploit that numerical artifact and appear to improve its objective while becoming worse at classifying true unseen negatives.

Non-negative PU learning alters the optimization so the estimated negative-class risk contribution does not drive the solution into an unphysical negative region. This guards against one form of overfitting, but it does not resolve label-selection bias, incorrect class priors or domain changes. A stable optimizer cannot compensate for a flawed scientific definition of the target. It is possible for a numerically well-behaved model to answer the wrong question very accurately.

An evaluation should compare train and held-out behavior, inspect prediction distributions, and test whether results change dramatically under plausible class-prior assumptions. If sophisticated methods outperform only when estimated class priors are tuned on the test set, the comparison is invalid. Keep model-selection decisions and uncertainty estimation separate from the truly untouched final evaluation sample.

Selected at random is not selected completely at random

Suppose obvious positives are discovered at a rate of 80 percent and subtle positives at 20 percent. If the deployment population contains both kinds, there is no single feature-independent c that correctly adjusts every individual’s selection score. Dividing q(x) by the overall average confirmation fraction can overstate positivity for some groups and understate it for others. The selection process must be studied by subgroup, mechanism and time.

Selected at random, in the PU literature, often means propensity to receive a positive label can depend on observed features. Then one may write q(x)=e(x) × p(x), where e(x) is the probability a genuine positive case is selected given its features. Estimating both quantities from limited observations is hard: many combinations of propensity and true risk can produce the same observed confirmation pattern. Additional structure, validation or externally measured propensities are needed.

Recent primary research on selected-not-at-random effects includes Kumar and Lambert on PULSNAR and class proportion estimation and Teisseyre and colleagues on testing the SCAR assumption. Neither removes the need to justify the actual data-generating process. A model report should explain which selection assumption was tested, which was adopted, and which consequences remain uncertain.

Where independent negatives come from

A small reference set with explicit positive and negative truth labels is often the highest-value investment. Sample cases from the intended deployment population, not only from the historical review queue. Apply a clear, repeatable adjudication protocol by trained reviewers or authoritative measurement. Record ambiguous or unresolved cases rather than forcing them into negatives. If the cost of adjudication is high, create a stratified reference sample and use appropriate weighting to recover population-level estimates.

Beware verification bias: if only high model scores are sent for truth review, evaluation will be richest where the model was already confident and weak where it may have missed cases. Sample across score bands, time periods, sources and demographic or operational subgroups that matter for fairness and reliability. Make the sampling probabilities known so unbiased or weighted estimates can be built when necessary. Reviewers should be blinded to model scores when feasible to avoid anchoring their judgments.

Independent truth is not always cheap or perfect. A molecular assay may itself have sensitivity limits; an expert review may have disagreement; legal outcomes may arrive months later. The evaluation protocol should state the observation window, adjudication rules and what remains unknown. The alternative is worse: reporting a polished F1 score against administrative statuses that never represented the true condition.

Train–validation splitting must respect discovery history

Randomly splitting rows is dangerous when repeated observations share the same entity, family, account, product or historical investigation. If one record for an entity is in training and another is in the test set, the model may memorize entity-specific signals rather than generalize. A temporal setting creates another hazard: confirmations that happened after the prediction date must not be included as if they were then known. PU learning adds selection bias to the already serious risk of leakage.

Design the split around the actual deployment question. If predicting risk for new organisations, hold out entire organisations. If prioritizing future investigations, hold out later periods and freeze features at the decision date. If multiple tests refer to the same compound, use scientifically appropriate grouped splitting. The scikit-learn documentation describes grouped cross-validation schemes such as GroupKFold and LeaveOneGroupOut: cross-validation and grouped splits.

The separate data-leakage guide explains the broader validation problem. PU-specific evaluation adds the crucial question: are labeled positives in each split produced by the same confirmation process, and does the reference sample contain verified negatives? A leakage-free split is necessary but not sufficient for valid PU metrics.

What to measure instead of pretending all unknowns are negative

When a truth-labeled reference set exists, report precision, recall, false-positive rate and calibration against those verified outcomes, with uncertainty intervals appropriate to the sample design. Where reference labels remain incomplete, report ranking or retrieval diagnostics explicitly as proxy measures rather than smuggling them into claims about genuine detection rates. Precision at a fixed review budget can be especially useful if the top-ranked cases are independently adjudicated.

For example, a team can investigate the top 40 new unlabeled cases ranked by a PU model. If independent review confirms 24 genuinely positive, the observed yield among those 40 is 24/40=60 percent. This is a valid statement about that reviewed top-40 set, assuming truthful adjudication. It does not establish population recall, because the number of positives below rank 40 remains unknown. Nor does it establish an unbiased overall precision unless the sampling and target decision set justify that extrapolation.

Track operational costs too: specialist review hours, false alarms, delayed discoveries and subgroup failure rates. A model with slightly higher apparent precision might still be worse if it consistently misses harder-to-detect cases. Predeclare the review budget and outcome definitions. Report the denominator alongside every percentage so readers can distinguish model discrimination from selection effects and changes in how often investigators label cases.

Calibration is not one number

A model can rank likely positives correctly while producing systematically wrong probabilities. In PU settings this risk is unusually acute because the observed labels describe confirmation rather than true class membership. Even under SCAR, a correction by c depends on calibrated estimation of the confirmation probability q(x). If c is wrong by twenty percent, the estimated absolute risks may be badly miscalibrated despite a strong ranking metric.

Examine reliability diagrams on independently adjudicated cases, preferably stratified by relevant data segments. Compare predicted risks with observed frequencies using sampling weights if the reference sample is not representative. If calibration varies substantially by subgroup, check whether selection propensities also vary. Do not fit or tune the calibration map on the final test sample and then call the same sample an unbiased evaluation.

For an explanation of proper probability calibration under ordinary labeled evaluation, see probability calibration if that destination is live; the core principle is that score meaning must be checked against observed truth, not inferred from the model’s confidence. In this article the more important point is the extra correction and uncertainty introduced by missing confirmed negatives.

A decision threshold should reflect the job

Suppose a specialist can review 50 cases each week. A rank-based workflow can select 50 candidates for further investigation without pretending each selected case is definitively positive. In a safety-critical setting, an automatic action against a person may be inappropriate even when an investigative ranking is useful. A model’s job description should distinguish ‘prioritize investigation’ from ‘make a final accusation’.

Consider two possible costs in a fictional queue: reviewing a false alarm costs one unit, while missing a genuine case until the next period costs four. If a calibrated posterior p were genuinely available and review perfectly resolved the condition, an idealized one-step expected-cost decision might compare 1×(1-p) for reviewing against 4×p for deferring. Review is cheaper when 1-p < 4p, or p>0.20. This threshold is an instructional simplification, not a real-world policy recommendation: review capacity, imperfect verification, fairness constraints and long-term effects change the optimization.

Notice how selection bias matters twice. It can corrupt the estimate p and it can make the cost of missing subtle positives invisible in historic records. Use thresholds only after validating the target probability and the actual utility model. When probability calibration is unreliable, a human-review ranking with a declared budget may be safer than a hard automated cutoff.

Diagnosing a surprisingly high score

A PU model reports 98 percent ‘accuracy’. The first question is accuracy against what? If 98 percent of items lack a confirmed-positive flag, predicting unlabeled for everyone can yield 98 percent accuracy on s while finding zero positive cases. This is a trivial classifier exploiting observation prevalence. A more complex algorithm may achieve a similar score with no practical benefit. Always compare against baselines derived from the class and label frequencies.

Other warning signs include near-perfect test performance after repeated observations leak across splits, large swings in estimated prevalence when one parameter changes, implausibly high corrected probabilities, and large differences between easy-to-observe and hard-to-observe subgroups. Investigate whether features encode the review process itself: reviewer ID, notification history, investigation status and post-outcome actions can be excellent predictors of s but invalid predictors of the underlying condition before discovery.

The repair begins with a label audit, event timeline and independently adjudicated test sample. Involve the domain expert who understands why historical positives were found. More model capacity will not fix the difference between ‘was inspected’ and ‘was positive’. In fact it may make this distinction harder to notice because administrative traces are often extremely predictive.

Changed-context transfer: wildlife observations

Transfer the idea from administrative cases to a wildlife observation project. Confirmed sightings of a rare bird are positives; squares of a landscape without a reported sighting are unlabeled, not automatically bird-free. However, the probability a positive square is reported may depend on path access, observer effort, camera placement, weather and species detectability. The SCAR assumption is particularly implausible when observers visit accessible locations much more frequently.

Build a structured observation model: record effort, season, location, detector type and an explicit observation window. Where possible, collect repeat visits and credible non-detection information, which may support an ecological occupancy or detection model rather than generic PU classification. An unvisited square should never be interpreted as an observed absence. A carefully selected random survey of locations provides data about unseen parts of the landscape and can validate predictions away from popular observation routes.

The same first-principles decision follows: separate presence from detection, model the observation mechanism, and evaluate against a scientifically justified reference process. Unlike a customer case, the underlying truth can change by season or movement, so the target’s time horizon must also be specified. A good method transfers its question, not merely its code.

A second transfer: educational support detection

Suppose a school knows which learners received specialist literacy assessment. Those are examples of a referral-and-confirmation process, not automatically a complete list of learners needing support. Non-assessed students include both pupils who are doing well and pupils whose difficulties were not yet identified. The observation probability may differ by attendance, parent advocacy, teacher workload, language background or school year.

An AI tool that treats non-referred learners as negative can reproduce differences in who had access to assessment. A PU approach might help prioritize careful support screening, but only with strong governance. The training objective must be linked to educational benefit, never used as a substitute for professional diagnosis or as a covert basis for denying opportunities. Obtain appropriate consent and privacy protection, minimize sensitive data and allow human review and contestation.

For broader learning context, the education and transfer guide examines how evidence should improve real decisions. This example illustrates why missing labels are not neutral: an institution’s past capacity and decisions shape the dataset. Distinguishing a genuine learning need from the probability of referral is essential for fair use.

When PU learning is not the right method

Do not automatically use PU learning if negatives are available but inconvenient to label; a small supervised sample might be more reliable. Do not use it when positive confirmations themselves are highly erroneous unless the method explicitly models that noise. Do not rely on basic SCAR correction when cases are selected for confirmation based on hidden severity or unrecorded investigator judgement and there is no way to validate the propensity mechanism.

Other problem classes may fit better. Multi-class uncertainty is not identical to one positive class mixed with unknown negatives. Regression with selectively observed outcomes needs its own missingness and selection models. Anomalies with no defined positive ground truth may call for anomaly detection, but calling their output probability of fraud would be a category error. Survival analysis is relevant when events are delayed or right-censored; the absence of an event within a short window does not mean it can never occur.

The most useful early decision may be to redesign data collection: introduce a random-audit fraction, clear adjudication categories, a recorded selection mechanism and stable timestamps. Better observed labels can create more value than another architectural trick. Models should be chosen after defining the evidence, not used to conceal evidence that was never collected.

A practical project plan for trustworthy PU modelling

Start by naming y and s, the target horizon and the exact selection event. Draw a small count table showing known positives and unknowns without assigning the unknowns to negatives. Audit the confirmation pathway for time, subgroups and access effects. Then choose a validation design that prevents shared entities and future information from contaminating the test set. Make independent reference adjudication a project milestone, not an optional afterthought.

Build three baselines: a trivial confirmation-frequency baseline, a simple supervised model on whatever verified positives and negatives exist, and a transparent PU approach whose class-prior and selection assumptions are written down. Only then compare more complex non-negative risk estimators or propensity-aware models. Use a reproducible training manifest that fixes feature versions, split assignments, class-prior estimates and evaluation rules. Report the difference between case prioritization and calibrated true-class probabilities.

Plan a human escalation route when estimated selection bias is large or the model disagrees with expert observations. Audit whether an active-learning review programme changes the label distribution over time. A deployment can improve the data it learns from, but only if new confirmations are recorded with their own selection propensities; otherwise the system may train on its own prior decisions and gradually narrow the observed world.

Primary research and what it actually establishes

Elkan and Noto’s 2008 paper establishes important connections between positive-label selection and ordinary classification under explicit assumptions. The publication is Learning classifiers from only positive and unlabeled data. Kiryo and colleagues’ 2017 NeurIPS paper develops a non-negative risk estimator and analyzes its overfitting behavior: Positive-Unlabeled Learning with Non-Negative Risk Estimator.

For more recent limitations of selection assumptions, see PULSNAR and non-randomly selected positives and testing selected-completely-at-random assumptions. These are research studies, not a universal deployment certification. For validation implementation, consult scikit-learn’s current cross-validation guidance and its metric definitions.

Every scientific claim in this article is bounded: the probability identity requires SCAR; the class counts are labeled fictional; the metric examples are exact arithmetic from those counts; and the recommended validation steps address documented sources of bias. A reader should be able to tell a derivation from a hypothesis, a tool feature from a study result, and a verified observed outcome from a model forecast.

Frequently asked questions

Is unlabeled the same as negative? No. Unlabeled means the target state has not been established. Is PU learning just standard semi-supervised learning? It is a specialized asymmetric-label setting: confirmed positives are available, while the remainder mixes true classes. Can I compute precision by treating unknown cases as negatives? You can compute precision against the confirmation indicator, but it is not generally precision against the true condition. Disclose that distinction.

Does SCAR mean all records are sampled randomly? No. It concerns the probability that a genuinely positive record receives the observed positive label, conditional on being positive. Can a deep learning model discover the missing negatives automatically? It can learn patterns, but without assumptions or independent truth it cannot establish what the labels never identified. Does nnPU fix biased positive discovery? It addresses a specific risk-estimation overfitting issue, not arbitrary selection bias.

How much test data do I need? There is no universal number. It depends on expected prevalence, intended subgroup claims, required uncertainty and review costs. Is this appropriate for high-stakes educational or medical decisions? It may support research or careful triage under relevant safeguards, but should not be treated as automatic diagnosis or entitlement determination. When evidence is too weak, invest in additional labeling and professional judgment.

Reader pathways and closing principle

If you are learning machine learning, begin with the two-label distinction and the fictional 1,000-record calculation. If you are engineering a classifier, focus on the selection mechanism, class-prior sensitivity and independent test construction. If you are evaluating AI for an organisation, request the model card’s definition of ‘negative’, its adjudication protocol and a confusion matrix computed against verified truth. If you are teaching or supporting learners, ask whether the data record opportunity and referral rather than underlying need.

The broader How AI Works series covers the model pipeline; this specialist guide covers a particular failure of the evidence entering that pipeline. The central lesson is that unknown is not false. A useful AI system acknowledges the gap, models how observations were obtained, and earns its outcome claims through data that can actually test them.