eduKate Singapore

VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Next useful routeCanonical hubHELP

How AI Works | Label Shift — Changing Base Rates, Confusion Matrices and Prior Correction

—

by

eduKate Asia

A classifier can seem dramatically better or worse without changing a single model weight. Under label shift, the prevalence of genuine classes changes while the distribution of features within each class remains stable. In that special case, the model’s conditional error rates can be reused to estimate target class prevalence from unlabeled predictions and, with additional calibration assumptions, to correct posterior probabilities. The key is to check the class-prior mechanism rather than call every distribution change label shift.

This guide derives the mixture equation, solves fictional 1,000-case helpdesk and 500-unit factory examples, and explains failure conditions, sampling, score calibration and governance. The examples are arithmetic teaching laboratories, not reports of real model deployments. The article is a specialist route within How AI Works, not a duplicate general account of how AI learns.

Contents and reading routes

For the core mechanism, begin at the distribution assumptions and mixture equation. For classroom use, follow the checked helpdesk and factory cases. For operations, start at monitoring design, delayed labels, uncertainty and failure diagnosis. Each heading below is a direct internal link.

Why unchanged code can produce changed performance

Imagine a classifier that once flagged seventeen of every hundred incoming cases and now flags thirty-one. The code has not changed, and neither have the features used to build a prediction. Is the model broken, the intake process different, or the underlying class mix simply different? Label shift gives a precise hypothesis: the prevalence of actual labels has changed while the features of examples inside each genuine class are distributed as before. It is a hypothesis about the data-generating process, not a synonym for every noticeable change.

The practical reader problem is distinguishing a changed population from changed model skill. A classifier’s precision can improve because positive cases become more common even if it has learned nothing. Conversely, the same model can appear worse when rare positive cases become rarer. The right analysis tracks class priors, conditional error rates and evaluation denominators separately. Without that separation, teams may retrain an adequate model unnecessarily, celebrate an imagined improvement, or set review workloads from the wrong base rates.

What the label-shift assumption says

Let X represent model inputs and Y represent the true category. The source distribution describes the environment from which training or reference evaluation data came. The target distribution describes the population on which the deployed system acts. Label shift assumes that P_target(Y) differs from P_source(Y), while P_target(X given Y) equals P_source(X given Y). The proportion of classes moves, but the distribution of features conditional on the true class stays the same.

This statement is strong. It does not follow merely from the observation that a dashboard shows a different count of positive predictions. In some applications a changed disease mix or a changed customer channel could approximate the assumption; in others, new equipment, changed behavioural patterns or revised labels make it inappropriate. A good report names the hypothesis explicitly and explains what could falsify it. The algebra of prior correction is valid only to the degree that its assumptions represent the target process.

Compare other kinds of distribution shift

Covariate shift is commonly modelled as a change in P(X), with P(Y given X) remaining stable. Label shift keeps P(X given Y) stable while P(Y) moves. A change in the meaning of the relationship between X and Y is often called concept drift, although terminology differs. Real deployments can mix these effects, so an analyst should not classify a shift solely by looking at one two-dimensional chart or by naming the first method that seems to fit.

Ask operational questions first. Was the intake eligibility rule revised? Did a sensor change resolution? Did one group stop reporting difficult cases? Did an annotator redefine a positive outcome? Has the product attracted a new population? Such facts often explain why the conditional invariance does or does not hold. A changed predicted-positive rate can be a useful warning, but it cannot identify a causal source without independent evidence. Model monitoring should trigger investigation rather than force every change into one mathematical category.

A confusion matrix is a set of denominators

For a binary classifier, true positives are positive cases predicted positive. False negatives are positive cases predicted negative. False positives are negative cases predicted positive. True negatives are negative cases predicted negative. Sensitivity or recall is TP divided by TP plus FN, which conditions on the actual positive class. False-positive rate is FP divided by FP plus TN, which conditions on the actual negative class.

Precision is TP divided by TP plus FP, which conditions on a positive prediction. Specificity is TN divided by TN plus FP; negative predictive value is TN divided by TN plus FN. Accuracy counts TP plus TN over all evaluated cases. Because these denominators differ, precision may change even when sensitivity and false-positive rate do not. Every dashboard should state its population and decision threshold. An unexplained ‘positive rate’ could mean prevalence, positive predictions, or positive predictive value; mixing these is one of the most common sources of false conclusions.

Derive the prediction-rate equation

Let p be the true target prevalence of positives, s the sensitivity of a fixed classifier, and f its false-positive rate. The predicted-positive probability q has two mutually exclusive contributions. Genuine positives make up fraction p and are flagged with probability s, contributing s times p. Genuine negatives make up fraction 1 minus p and are flagged with probability f, contributing f times (1 minus p). Thus q = s p + f(1-p) = f + (s-f)p.

When s differs from f, solving for prevalence gives p = (q-f)/(s-f). This mixture identity lets a team use unlabeled target predictions and known conditional performance to estimate a changed class prior. It does not turn the model’s predictions into ground truth. It also does not prove that s and f stayed constant. Those are modelling inputs estimated from held-out labelled source data and subsequently checked against genuine target outcomes. A calculator can evaluate the equation perfectly while the model assumptions remain wrong.

Worked fictional helpdesk: construct the population

Consider a teaching example about an invented school helpdesk that routes difficult enquiries to a specialist. The original population has 10% genuinely complex cases. A held-out source evaluation estimates sensitivity at 80% and false-positive rate at 10%. We now stipulate a target queue of one thousand cases with a genuinely complex prevalence of 30%, while keeping those two conditional rates constant. The numbers are constructed to demonstrate the mechanism; they are not observed eduKate operating data.

The new queue contains 300 complex cases and 700 ordinary cases. Sensitivity of 80% produces 240 true positives and 60 false negatives because 0.80 times 300 = 240 and 300 minus 240 = 60. The 10% false-positive rate produces 70 false positives and 630 true negatives because 0.10 times 700 = 70. Reconciliation gives 240 + 60 + 70 + 630 = 1,000. That equality checks the ledger, and the class denominators remain 300 and 700 respectively.

Worked precision: why the apparently better model did not learn

The fixed classifier flags 240 true positives and 70 false positives, or 310 cases. Its predicted-positive rate is 310/1,000 = 31%. Among flagged cases, precision is 240/310, approximately 77.42%. Recall remains 240/300 = 80% and false-positive rate remains 70/700 = 10%. Negative predictive value is 630/(630+60) = 630/690, approximately 91.30%. These quantities are all consistent because each measures a different conditional event.

In the original population, a 10% positive prior gives q = 0.80 times 0.10 + 0.10 times 0.90 = 0.17, or 17%. Precision at that prior is 0.08/(0.08+0.09) = 0.08/0.17, approximately 47.06%. The move to 77.42% is a base-rate effect, not proof the classifier became more intelligent. Announcing that ‘precision improved by thirty percentage points because the AI learned’ would be false in this invented world where the model and its conditional rates were held unchanged.

Recover a target prior without target labels

Now suppose the target queue has no confirmed outcomes yet. The system can still count 310 flagged predictions among 1,000 cases. Insert q = 0.31, s = 0.80 and f = 0.10 into the mixture formula: p = (0.31-0.10)/(0.80-0.10) = 0.21/0.70 = 0.30. Estimated prevalence is 30%, matching the constructed underlying population. The apparent accuracy of this recovery follows from the exercise assumptions, not a real validation result.

In a live setting, q is estimated from a sample and s and f are themselves uncertain. The inferred prior may be biased by selection, threshold changes, delayed feedback or changed class features. Always distinguish ‘estimated prevalence under label shift’ from ‘measured prevalence among confirmed cases’. Prefer uncertainty intervals and later independent labels. If the estimator returns a negative prevalence or a value greater than one, treat that as a diagnostic problem with assumptions or sampling, not a literally meaningful proportion.

Inversion becomes unstable when class signals overlap

If sensitivity and false-positive rate are similar, the denominator s-f approaches zero. Imagine sensitivity 0.55 and false-positive rate 0.45. Then the denominator is 0.10. A 0.02 change in predicted-positive rate causes a 0.02/0.10 = 0.20 change in inferred prevalence if those conditional rates are assumed fixed. A modest estimation error could create a twenty-percentage-point prevalence error.

With sensitivity 0.80 and false-positive rate 0.10, the denominator is 0.70 and the same 0.02 variation corresponds to about 0.0286, or 2.86 percentage points. Neither example sets a universal threshold, but the comparison shows why matrix conditioning matters. In a multiclass problem, an analogous confusion-matrix inversion may be mathematically possible yet numerically fragile. Inspect the matrix and its uncertainty before making a consequential policy adjustment.

The model’s own output is not a verified outcome

A positive model decision is a prediction, not a label. The target positive-call rate q can be observed immediately, while actual outcomes may arrive slowly and be selected unequally. A black-box shift estimator uses q as a statistical signal under independently estimated conditional performance. It does not reveal which individual cases are truly positive. This distinction matters when a school, laboratory or workplace plans scarce review resources based on the output.

A defensible monitoring note might say that prevalence is estimated at thirty percent using the held-out reference confusion matrix and current unlabeled prediction frequencies. It should also report the date range, number of eligible cases and assumptions. As genuine target labels arrive, use them to challenge and refine the estimate rather than treating the earlier inference as permanent truth. Label-shift monitoring can inform decisions, but it should not launder provisional guesses into confirmed facts.

Prior-odds correction for an individual score

Under the label-shift assumption, the likelihood ratio P(X given positive)/P(X given negative) remains stable. Bayes’ rule then relates the source posterior odds for an individual to the target posterior odds by multiplying by the ratio of target to source class-prior odds. A source posterior is not simply multiplied by the new prevalence. Odds must be adjusted and then converted back to a probability.

For a numerical demonstration, suppose a score is genuinely source-calibrated at 0.50 for an example, the source prior is 0.10 and the target prior is 0.30. Source posterior odds equal 1. Source prior odds equal 0.10/0.90 = 1/9; target prior odds equal 0.30/0.70 = 3/7. The odds multiplier is (3/7)/(1/9) = 27/7. New posterior odds are 27/7 and the adjusted probability is 27/(27+7) = 27/34, about 79.41%. This is a constructed calculation, not a promise that any deployed score supports such correction.

Calibration is not the same thing as class shift

Calibration means a score can be interpreted as a probability over a defined evaluation population: among cases given roughly 0.7, about seventy percent should realize the event in a well-calibrated setting. A model can discriminate effectively while its scores are miscalibrated. It can also be calibrated on a previous class mixture and lose that property when prevalence changes. Label shift concerns population composition and within-class stability, not the intrinsic trustworthiness of every numeric score.

Validate source calibration separately if intending posterior correction. Then assess the target prior estimate, data pipeline and subgroup applicability. A corrected score should be tested on genuinely labelled target outcomes when those exist. Do not promise per-person reliability from a population-level mixture calculation alone. Source calibration is already a modelling claim requiring evidence, and the added shift assumption is another claim requiring scrutiny.

Hold the threshold still while diagnosing shift

Changing the classification threshold alters sensitivity, false-positive rate and positive-call fraction. If the threshold and population both change simultaneously, a raw before-after comparison cannot tell which caused the observed effect. Start diagnosis with a fixed model version, threshold, label definition and feature pipeline whenever possible. If an intervention must proceed during an evolving event, record the version boundaries and do not combine incompatible windows without adjustment.

After estimating prevalence, the organisation may decide to change the decision threshold because errors have different costs or specialist capacity is limited. That is a separate optimization problem. In the fictional thousand-case queue, 310 flags imply 310 review events if every flag goes to a reviewer. The workload matters even if the estimator is statistically sound. Evaluate the human consequences of misses and unnecessary reviews, not just the appeal of a higher precision number.

Separate model monitoring from service policy

A model-monitoring component might report that positive-call frequency has increased, that the shift estimate is unstable, or that conditional performance appears to differ. It need not decide how a school or hospital should react. The service owner must define whether to audit a sample, add review staff, narrow automation, escalate risky cases or pause a decision process. Statistical evidence enters a policy process; it does not replace the policy owner.

A recommendation also needs a stopping rule. When the estimate is unreliable, a conservative operational response may involve more human review or a temporary abstention policy. When the cost of missed cases is high, waiting for perfect evidence may be unacceptable; that trade-off should be explicitly owned. Avoid hidden automatic changes that silently recalibrate scores or reroute people merely because a distribution statistic crossed a convenient threshold.

Sampling windows and the hidden denominator

One incoming request may produce several messages, predictions and later edits. If the monitoring job treats each message as a case while reference evaluation treated each request as a case, the denominators differ. Repeated predictions from a small number of difficult requests could make the target positive-call rate appear elevated. Define the unit of analysis, eligibility rules, observation window and how duplicates are handled before counting.

Preserve model version, feature-pipeline version, prediction timestamp, outcome timestamp and label policy. Measure missing predictions and abstentions explicitly. If only successful tool executions enter the dashboard, failures may be systematically excluded. The most impressive-looking chart is not necessarily the most representative one. A small carefully sampled target audit often provides more diagnostic value than an enormous biased event log.

Delayed outcomes can create fake prevalence trends

Some cases get verified labels on the same day while others take weeks. A report based only on resolved cases may overrepresent easy negatives or urgent positives depending on workflow. This is a selection mechanism: the probability of being labelled by the cutoff depends on characteristics related to the label. Simply dividing the positive resolved cases by all resolved cases may misstate the prevalence of all incoming requests.

Use event-time cohorts and explicitly track pending outcomes. Distinguish a labelled audit selected at random from labels generated as a side effect of specialist review. When missing outcomes are not random, report the limitation and consider a designed sample with follow-up. Never label an unresolved case negative merely because a spreadsheet cell is blank. The best diagnosis combines unlabeled prediction monitoring with later independent outcome collection.

Subgroups may violate an aggregate assumption

A classifier’s sensitivity could be different for different languages, regions, age groups or observation channels. If the target population changes subgroup composition, the aggregate sensitivity and false-positive rate may change even if rates inside each subgroup stay constant. A single old aggregate confusion matrix may then be inappropriate for the new mixture. This is why a global label-shift estimate should not automatically be interpreted as equally reliable for everyone affected.

Estimate performance and priors at a meaningful subgroup level only when labels, sample size and privacy rules permit. A small subgroup percentage can have wide uncertainty; do not imply precision from a handful of observations. If group definitions themselves changed or are unreliable, flag that limitation. Fairness evaluation should examine which people experience false positives, false negatives or delayed service, rather than relying on one aggregate accuracy or prevalence number.

Changing the label definition breaks a silent assumption

Suppose the helpdesk redefines ‘complex’ to include enquiries formerly called ordinary. The measured percentage of complex requests rises, but that change may be a change in classification rules rather than in the world. Previously negative examples can become positive by definition, so the old confusion matrix and source posterior no longer describe the same target variable. The mixture equation can be evaluated numerically, but its interpretation has broken.

Version the label handbook and preserve the evaluation population used under each version. If a mapping is possible, run a bridging audit in which qualified reviewers apply both definitions to a sample. Where mapping is impossible, establish a new reference rather than disguising a semantic break as routine label shift. Stable meanings are part of the statistical prerequisites, not clerical notes.

When within-class features change

A fraud detector may see new fraud tactics; a medical classifier may see a changed test instrument; a student-support model may see a different intake channel with unfamiliar expressions of need. In each case, P(X given Y) may move. Sensitivity and false-positive rates from the source environment can cease to apply. Inverting a target positive-call fraction against those stale rates might produce a plausible number that is nevertheless incorrect.

Investigate feature distributions inside newly labelled target cases where possible. Use subject-matter knowledge to identify changes in mechanisms, not merely generic anomaly thresholds. Novelty or out-of-distribution flags may identify unfamiliar observations, but they do not prove that pure label shift occurred. When evidence contradicts the assumption, return to representative labelled evaluation, feature repair or a more general model-update strategy.

Importance weights versus a new training run

If target and source class-conditional features truly match, source examples can be importance-weighted by the ratio of target class prior to source class prior. This makes a weighted source risk reflect the new mixture under the assumption. Alternatively, a team may adjust calibrated posterior odds or decision thresholds. These are different interventions with distinct requirements; a shift estimate alone does not say which is appropriate.

Large class-prior ratios can produce high-variance weights, especially for rare classes. Resampling can change training composition without accurately representing deployment. Retraining on model-generated pseudo-labels can reinforce earlier errors if they are treated as ground truth. Use held-out evaluation and compare the simplest interventions first. If the entire feature-label relationship is unstable, new high-quality target labels and a redesigned model may be more important than a prior multiplier.

The statistical uncertainty cannot be hidden

Target q is an estimate from finite observations; reference sensitivity and false-positive rates are estimates from finite labelled examples. Errors in any of them change inferred prevalence. Because the denominator is s-f, its uncertainty matters as much as the numerator’s. An interval estimate, bootstrap analysis or Bayesian method can communicate this uncertainty if implemented and interpreted properly. A single decimal-place point estimate without uncertainty is insufficient for high-impact decisions.

As a simple diagnostic, recalculate prevalence using several plausible values of q, s and f. Check whether those combinations remain within zero to one and whether the implied review workload changes significantly. Do not silently clip impossible results and claim the formula worked. When plausible parameter values lead to radically different estimates, say that the monitoring data cannot support a stable prevalence claim. More reliable labels or a more informative classifier may be required.

Transfer case: factory defects with changed prevalence

A second fully fictional laboratory changes the task from enquiries to defective products. Assume the classifier’s sensitivity is 90%, false-positive rate is 5%, source defect prevalence is 5%, and a new 500-unit batch has a stipulated true defect prevalence of 20%. This gives 100 defective and 400 acceptable units. The resulting confusion matrix has 90 true positives, 10 false negatives, 20 false positives and 380 true negatives. Sum check: 90 + 10 + 20 + 380 = 500.

The model calls 110 units positive, so q = 110/500 = 22%. Inversion yields p = (0.22-0.05)/(0.90-0.05) = 0.17/0.85 = 0.20. Precision is 90/110, approximately 81.82%, and negative predictive value is 380/390, approximately 97.44%. The numbers are intentionally constructed. A real factory should also investigate whether the production change affects defect characteristics, since altered components could violate the assumed stable rates even when the prevalence arithmetic is internally correct.

Multiclass settings need matrix diagnostics

With more than two classes, predicted-class frequencies form a vector and source conditional prediction rates form a confusion matrix. Under label shift, the target prediction vector equals that matrix multiplied by the target prior vector, with orientation handled consistently. Estimating priors becomes a constrained inverse problem. A theoretically invertible matrix can still be poorly conditioned, and negative estimates may result from noise if constraints are ignored.

State the matrix convention in documentation: are rows actual labels or predicted labels, and do columns or rows sum to one? Reversing that convention yields a deceptively polished wrong answer. Preserve an independent evaluation set for the confusion matrix, and examine weakly represented classes separately. If some classes are indistinguishable to the predictor, it may be impossible to estimate their separate prevalences from black-box prediction counts alone.

Abstention is an outcome that must be counted

Many responsible classifiers can abstain, defer or send a case for human assessment. If monitoring silently drops abstentions, its denominator changes and the estimated positive-call mix may no longer represent the whole target population. Treat abstention as a documented prediction category or write a careful conditional analysis that explicitly limits what population the estimates cover.

An abstention can improve safety while making a simplistic ‘accuracy’ dashboard look worse by reducing automatic coverage. Evaluate automatic decision coverage, referral load, human review outcomes and errors on the cases that remain automated. A shift estimator that cannot distinguish categories reliably should sometimes produce an insufficient-evidence status. The right output is the one that supports a safe decision, not the one with the most decimal places.

A seven-step operational investigation

Start by naming the unchanged model version and target decision. Second, check that source and target have compatible label definitions and units of observation. Third, count target prediction frequencies including duplicates and abstentions explicitly. Fourth, inspect source confusion-matrix quality and conditioning. Fifth, compute the prior estimate with uncertainty. Sixth, obtain an independent target-label audit to test the conditional-rate assumption. Seventh, compare possible interventions and document who is authorized to change deployment.

Do not run this as an opaque ‘recalibrate’ button. The method should emit its assumptions, source version, target period, sample counts and diagnostic flags. A human reviewer should be able to reproduce the key equation from reported rates and explain what would invalidate it. If a pipeline times out or labels become unavailable, keep the previous verified state and mark the estimate as pending rather than publishing an unsupported revised prior.

Failure diagnosis: which component should be repaired?

If the predicted-positive fraction changes immediately after a threshold deployment, verify threshold configuration before proposing population shift. If inferred prevalence is impossible, inspect counts, matrix orientation, threshold mismatch and assumptions. If labels arrive but show changed sensitivity, investigate within-class feature changes rather than forcing an old confusion matrix. If precision changes but sensitivity does not, calculate the base-rate effect before celebrating new model capability.

If a correction raises harm to a subgroup, review the decision policy and representative evaluations. If a queue becomes overloaded, treat capacity as an operational control, not a number the model can wish away. If target labels mostly come from escalated cases, redesign the audit sampling frame. Repairs should be chosen after locating the broken assumption. An expensive retraining process is not automatically better than a short, carefully designed measurement study.

Why governance and accountability still matter

Who is responsible for declaring the target population representative? Who verifies the label policy, approves the prior correction, and decides when a model must stop making autonomous recommendations? These are organizational roles, not statistical outputs. A trustworthy deployment links monitoring indicators to documented escalation thresholds and keeps human decisions visible, especially when the model affects access to services or resources.

Prevalence estimates can carry sensitive implications for groups of people. Collect only necessary data, safeguard access to labels and refrain from presenting unstable subgroup fractions as individual characteristics. The NIST AI Risk Management Framework describes a broad setting for mapping, measuring and managing AI risks. It does not turn the algebra in this article into a validation of any particular deployed application.

Limits of the research and what to read next

Lipton, Wang and Smola’s 2018 peer-reviewed work on black-box label-shift estimation analyses how a confusion matrix and unlabeled target predictions can detect and correct class-prior changes when the relevant conditions hold. That is a technical result under explicit assumptions, not evidence that any classifier is automatically safe under deployment drift. The worked examples here are original teaching cases with stipulated values and checked fractions.

Probabilistic calibration, data leakage, label quality and missing observations are related but distinct reader problems. A team may need all of those diagnostics to trust its target estimates. The most useful next step is to reproduce one small example with its own held-out counts and then design a target audit that could disconfirm the label-shift story. An ability to explain why an estimate might fail is a core part of responsible machine learning.

Frequently asked questions

Does label shift mean my model has forgotten its training? No. It refers to a change in the class proportions of the target population under an assumption about the within-class input distribution.

Can a classifier become more precise without retraining? Yes. Under fixed sensitivity and false-positive rate, a higher positive prevalence can raise the precision among cases predicted positive.

Can I estimate prevalence without target labels? Sometimes, using target prediction frequencies and appropriate source confusion-matrix estimates, but the result remains conditional on the assumptions.

Why can the equation give a negative estimate? Finite-sample noise, a weak classifier, incompatible definitions or failed assumptions can make inversion unreliable. Negative prevalence is not meaningful.

Does prior correction replace evaluation? No. It must be checked against representative target outcomes and the actual decision costs.

What should a student remember? Distinguish how common an event is, how often the model flags it, and how often a flag is correct. These fractions use different denominators.

Primary evidence: Lipton, Wang and Smola (2018), Detecting and Correcting for Label Shift with Black Box Predictors; scikit-learn documentation on probability calibration; NIST AI Risk Management Framework Core.

Related lessons: classification thresholds and review capacity, data leakage and honest validation, missing-data evaluation, and How AI Works.

Discover more from eduKate Singapore

Subscribe now to keep reading and get access to the full archive.

Continue reading