How AI Works | Class Weights and Focal Loss — Training on Rare Outcomes Without Fooling Yourself

Did you know? When 95% of training examples belong to one class, a classifier can score 95% accuracy by predicting that class every time and still miss every rare event. Class weighting and focal loss change what the training algorithm pays attention to. Class weights increase the loss attached to scarce labels; focal loss reduces the influence of examples a model already handles confidently. Neither technique automatically supplies calibrated probabilities, a sensible decision threshold, or evidence that the system will work when deployed.

This guide answers the practical question: what should you do when a valuable outcome is rare, such as a fault in an assembly line, an unusual document category, or an important support request? First establish trustworthy labels and a leakage-safe split. Compare an unweighted baseline with deliberately weighted and focal-loss models, evaluate on an untouched population-shaped test set, examine precision, recall and review capacity, and calibrate or re-estimate probabilities when the decision needs them. Use the calculations below as worked examples, not claims about any measured production system.

Class imbalance is a property of the observed dataset, not proof that the minority class is more valuable. If the rare class is mislabeled, expensive to investigate, or drifting over time, increasing its training influence can amplify errors rather than solve them. Start with the reader’s operational question—what mistake costs more, and who can act on a prediction?—then choose a loss function in support of that answer.

Contents

• Why accuracy can hide the very errors that matter
• Class-weighted loss: whose errors count more during learning?
• Focal loss: down-weight easy examples rather than every majority example
• Worked case: 1,000 records, 50 positives, and a review team
• Weights, thresholds and probabilities are different controls
• Build a fair comparison before fitting anything
• What changes in a gradient update?
• Eight errors that make weighted learning seem better than it is
• Transfer exercise: from a 5% case rate to a 2% case rate
• When weighting, focal loss, or neither is appropriate
• Frequently asked questions
• Reader pathways: students, engineers, reviewers and leaders
• Primary sources and further reading

Why accuracy can hide the very errors that matter

Imagine a hypothetical service that receives one thousand application records. Nine hundred and fifty are ordinary and fifty need specialist review. A model that says ‘ordinary’ for all one thousand records is correct 950 times. Its overall accuracy is 950/1,000, or 95%. Its recall on specialist-review cases is zero out of fifty. That contrast is not a minor reporting detail; it changes whether the model solves the actual problem. Accuracy answers how often all classifications agree with labels; recall answers how many truly relevant cases the process found.

An unweighted classifier does not literally decide to ignore rare classes. It optimises an average loss over examples, and the numerous majority observations can dominate the average gradient. Whether this happens depends on class difficulty, model architecture, sampling, features, optimiser and objective. Sometimes a rare class is easy to separate, and ordinary cross-entropy is sufficient. Sometimes balanced sampling fixes exposure without a special loss. The right starting point is an observed failure mode, not a default rule that imbalance always demands correction.

Work backward from action. If human specialists can review only eighty cases daily, a ranking that produces seven hundred alerts cannot be treated as successful just because it catches many true cases. A screening model may reasonably favour recall when misses are costly, but somebody still has to handle false positives. A maintenance model may require early warnings rather than final classifications. When costs are asymmetric, spell out the cost of each error and the unit of evaluation before adjusting any training weight.

Separate three questions people often mix up. First, how is the model trained? Class weighting and focal loss belong here. Second, what does the model score mean? Probability calibration and prevalence matter here. Third, which scored items become actions? Thresholds, top-k routing, abstention and human review capacity matter here. Changing the first question does not settle the other two. The companion article classification thresholds and review capacity explores that downstream decision problem in greater depth.

Class-weighted loss: whose errors count more during learning?

A classifier is usually trained by updating parameters to reduce a loss. For a binary label y, the familiar cross-entropy loss penalises assigning low probability to the observed class. When a record with label one is given predicted probability 0.1, its loss is much higher than when the same record is assigned 0.9. In weighted loss, the penalty for an example is multiplied by a positive weight associated with its class. Larger weight means that a comparable error from that class contributes more to the objective.

Let n be the total number of labelled training observations, K the number of classes, and n_c the count in class c. A commonly used balanced heuristic sets w_c = n/(K × n_c). The scikit-learn compute_class_weight documentation explicitly gives that formula. The resulting total nominal weight for each class is n/K, as long as all classes are represented. It is a useful baseline heuristic, not a mathematically necessary optimum or a promise of equal real-world error costs.

For the fictional set of 1,000 training records with 950 ordinary and 50 specialist cases, the ordinary-class weight is 1,000/(2 × 950) = 0.526315789…, and the specialist-class weight is 1,000/(2 × 50) = 10. Under this rule, the 950 majority examples have total nominal weight 500, while the 50 minority examples also have total nominal weight 500. The calculation changes the objective, not the counts: the dataset still contains only fifty distinct minority records and cannot magically acquire additional variety.

Notice the denominator. If there are three classes, replace K = 2 with K = 3 and use the count of each class. If some classes have only one example, the inverse-frequency weight may become enormous, making gradients noisy or enabling overfitting to an annotation mistake. A class with no training examples gives a zero denominator, which must be addressed as a data-coverage problem rather than solved with an infinite weight. For multilabel predictions, each label has its own positive and negative prevalence and should be handled with the loss function designed for that output structure.

Your choices need not be confined to inverse frequencies. You may choose bounded weights after tuning on validation data, use a cost-sensitive loss aligned to actual operational consequences, or report that standard loss performs better. Keep the weight definition in the model record, including the training counts and label mapping. If a sample has an independent reliability weight, understand how that weight combines with class weighting. Accidentally multiplying several large factors can make one observation control a whole optimisation step.

Focal loss: down-weight easy examples rather than every majority example

Class weights care which class a record belongs to. Focal loss cares how easily the current model predicts the labelled outcome. In its standard binary exposition, let p_t be the probability assigned to the true class. Cross-entropy is −log(p_t). Focal loss is −(1 − p_t)^γ log(p_t), often with an additional class-balancing factor α_t. Gamma, written γ, controls how strongly easy examples are down-weighted; γ = 0 recovers cross-entropy before any optional α balancing.

The key mechanism becomes clear from two deliberately simple observations. An easy record has p_t = 0.9. Its ordinary cross-entropy is approximately 0.10536. With γ = 2, the focal factor is (1 − 0.9)² = 0.01, so its focal loss is about 0.0010536 before α. A hard record has p_t = 0.2. Its ordinary cross-entropy is about 1.60944; the factor is (0.8)² = 0.64, giving focal loss about 1.03004. These are per-example loss values, not measurements of gradient size or proof that one trained model will outperform another.

This design was introduced in the computer vision paper Focal Loss for Dense Object Detection by Lin and colleagues. In their problem, a very large number of easy background positions could overwhelm rarer informative positives during training. The paper’s specific experiments should not be presented as evidence that focal loss must improve every healthcare, school, finance or tabular classification application. The transferable idea is attention to difficult examples; the domain-specific evidence still needs collecting.

An important subtlety: difficulty is not the same as usefulness. A sample that remains hard might be mislabeled, ambiguous, out of distribution or irreducibly noisy. Focal loss can keep paying attention to those records. If the labels are inconsistent, that behaviour may damage generalisation. Before turning gamma up, inspect a sample of persistent high-loss records with knowledgeable reviewers and look for systematic missing features or taxonomy mistakes. The right repair may be annotation policy, not an exotic loss.

Focal loss and inverse-frequency weighting are not mutually exclusive; α_t can introduce class-specific balancing inside the focal objective. But simply combining aggressive inverse-frequency weights with a large gamma can over-concentrate training on very few examples. Start with a simple, inspectable comparison such as plain cross-entropy, moderate weighting alone, and focal loss with documented gamma. Add combinations only if a leakage-safe validation set supports the extra complexity.

Worked case: 1,000 records, 50 positives, and a review team

Here is a complete hypothetical example. A company labels 1,000 test records independently of training. Fifty truly require specialist attention and 950 do not. The model under study marks 140 records for review. After checking the labels, 40 are true positives and 100 are false positives; the model misses 10 positives and correctly leaves 850 negatives unflagged. Every count in this paragraph is illustrative. The four confusion-matrix cells sum to 40 + 100 + 10 + 850 = 1,000, so the denominators are consistent.

Precision is TP/(TP + FP) = 40/(40 + 100) = 40/140 = 28.57%. Recall is TP/(TP + FN) = 40/(40 + 10) = 40/50 = 80%. Specificity is TN/(TN + FP) = 850/(850 + 100) = 850/950 = 89.47%. Accuracy is (TP + TN)/N = (40 + 850)/1,000 = 89%. F1 is 2TP/(2TP + FP + FN) = 80/(80 + 100 + 10) = 80/190 ≈ 42.11%. Stating all these numbers exposes trade-offs that an accuracy-only report conceals.

The system flags 140 of 1,000 records, a review rate of 14%. If the team can review only 80 per day, that particular decision rule exceeds capacity by 60 cases daily. Weighted loss may have helped the model distinguish classes, but it cannot retroactively make the alert queue manageable. At a different threshold, the queue might shrink, but which positives are lost depends on the score distribution. There is no legitimate way to infer the new true-positive count from the confusion matrix alone; individual validation scores are needed.

Compare this model fairly with an unweighted baseline and a focal-loss candidate on the exact same held-out records. Report not just a single chosen operating point but the precision-recall curve, area under the precision-recall curve where appropriate, and recall among the top 80 scored items. Also measure whether the scores mean what they claim to mean, because reweighting can alter calibration. A ranking model that finds 35 real cases within 80 reviews may be more useful for this team than the illustrated 140-alert rule, even if its global accuracy is lower.

Before declaring a winner, interrogate the fifty true positives. Do some come from the same customer, machine, event or document template? A random row split can leak near-duplicates across training and evaluation. The data-leakage and grouped-split guide explains why model validation must respect shared entities and future information. Also check the label audit: fifty nominal positives do not imply fifty independently verified facts.

Weights, thresholds and probabilities are different controls

Training weights change the relative incentives during fitting. A threshold turns a score into an action. A probability calibration method tries to make stated probabilities match observed frequencies under suitable assumptions. Confusing these layers produces surprising failures. You can increase rare-class recall during fitting yet still choose a threshold that floods a team with false positives, or choose a cautious threshold that misses most rare cases. The whole pipeline—not the loss function alone—determines a user-visible result.

Suppose a model outputs 0.7 for an event. It is unsafe to read that as a 70% real-world chance merely because the output passed through a sigmoid or softmax. Class weights alter the optimisation target, so the resulting score may no longer represent the natural prevalence of the event. Case-control sampling, oversampling and undersampling can do something similar. Calibration against a representative, independent validation set can help; if base rates change after deployment, a past calibration check does not settle the new situation.

For a strictly specified setting where only the class prior changes and class-conditional feature distributions remain stable, mathematically justified prior correction may be possible. Real applications often violate those assumptions because operations, populations, sensors or definitions have changed. See label shift and changing base rates for the limits of that correction. When the cost of one missed case is extremely high, route uncertain cases to human attention rather than pretending weighting makes the risk disappear.

The practical lesson is to tune a loss on one subset, calibrate with an appropriate separate subset when reliable probabilities matter, set a threshold or review quota using declared costs and capacity, then report final performance once on a held-out test set. Do not choose weights, temperature and action threshold repeatedly on the test set until the story looks good. That turns the final test into a hidden development set and exaggerates future performance.

Build a fair comparison before fitting anything

First freeze the intended target: what qualifies as a positive case, what must be available at prediction time, and when the prediction will be used. Decide which unit is independent. A single patient can have many visits; one machine can generate many alarms; one student can submit many similar exercises. If related observations appear in both training and testing, the model may look capable because it recognises entities rather than learning a transferable signal. Use grouping or time-based separation when the application demands it.

Second describe the production prevalence rather than assuming the training prevalence represents it. Collect enough genuine minority cases to estimate important metrics with defensible uncertainty. A test set with five positives is inadequate for a precise 90%-recall claim: one different classification changes observed recall by twenty percentage points. Report the numerator and denominator next to every rate. For operational reviews, retain the event type, time horizon and review eligibility so the denominator can be reconstructed.

Third keep one baseline boring. Train the appropriate unweighted model, inspect its learning behaviour, then add class weights from the training labels only. Do not calculate weight factors using future evaluation labels. For a focal-loss variant, record gamma and alpha settings, reduction convention and any sample weighting. Match model capacity, preprocessing, training budget and evaluation data across alternatives. Seeds and repeated runs can expose unstable minority-class outcomes; a single lucky run is not a proof of improvement.

Fourth decide primary and secondary scores in advance. A useful primary score might be recall at a fixed daily review quota, or expected loss under stated error costs, not the highest percentage on a favourite chart. Secondary checks should include precision, minority subgroup performance, false-negative review, probability reliability, model latency and training stability. If none of the methods gives an operationally acceptable result, the honest conclusion is that the data or task design requires repair.

Finally separate selection from confirmation. Use validation data to choose among weighting schemes and thresholds, then evaluate once on the final untouched data. If deployment reality changes, that is a new evaluation phase, not permission to relabel prior tests as independent. It is often useful to compare performance on representative data and on a specifically enriched challenge set, but never blend the two without explaining their differing denominators.

What changes in a gradient update?

A loss value becomes useful for learning when the optimiser computes derivatives with respect to parameters. If example i contributes L_i and weight w_i, the weighted aggregate can be written as Σw_iL_i divided by an explicit normaliser. Some implementations divide by number of observations, some by sum of weights, and some have other reduction conventions. That choice changes the scale of the gradient and therefore interacts with the effective learning rate. Do not assume two libraries interpret the same-looking weight vector identically.

For binary cross-entropy with logits, numerical stability matters: implementations often combine the sigmoid and cross-entropy calculation so extreme logits do not cause avoidable floating-point problems. In PyTorch the BCEWithLogitsLoss reference distinguishes positive-class weighting with pos_weight from general sample weighting. In multi-class classification, CrossEntropyLoss accepts a per-class weight vector. Check the library version, tensor shapes, class ordering and reduction mode rather than copying a parameter from an unrelated example.

With weighted cross-entropy, a rare example with weight ten contributes ten times its unweighted loss before reduction, but not necessarily ten times the final parameter change. Other samples share the batch, gradients may point in competing directions, momentum accumulates history, and clipping may cap the update. A model’s learning dynamics are collective. That is why validation evidence is essential even when the weight arithmetic itself is correct.

Focal loss changes the derivative more intricately than simply multiplying cross-entropy by a constant, because p_t depends on the model parameters. The modulating factor changes during training as the model becomes more or less confident. An item can be heavily emphasised early and then almost ignored as it becomes easy. Conversely, a noisy item can remain influential. A carefully checked implementation should be compared against a simple reference for representative logits, not just inspected for a plausible scalar loss.

If gradients or metrics become unstable, inspect the number of positive examples in each mini-batch, the distribution of per-example losses, the ratio of weighted loss coming from hard minority cases, and clipping frequency. Try smaller weights or gradient accumulation before assuming a sophisticated optimiser is needed. The training system’s numerical behaviour can undermine an otherwise reasonable cost model.

Eight errors that make weighted learning seem better than it is

Error one is counting the minority class twice: oversampling to fifty-fifty and then applying weights calculated for the original highly imbalanced dataset. That compounds the minority influence. Either calculate weights for the actual effective training distribution and objective or justify the deliberate combined weighting in an experiment. Error two is copying class weights into the wrong label order. A vector of [10, 0.526] has the opposite meaning from [0.526, 10] if the classes are indexed ordinary=0 and specialist=1.

Error three is evaluating on an artificially balanced test set while claiming performance for a naturally rare event. A 50%-positive test set can yield impressive precision for a classifier that will face only 2% positives in practice. Precision depends on prevalence; use a population-shaped test set or explain what can be transported and what cannot. Error four is reporting accuracy alone when a model can ignore the minority class. Always include counts, class-specific measures and the chosen operating point.

Error five is treating hard labels as perfect. Minority labels are especially vulnerable to small absolute annotation errors: if ten out of fifty are wrong, a fifth of that minority class is unreliable. A higher loss weight gives those errors more pull. Error six is treating focal loss as a cure for missing minority subgroups. If every positive in training comes from one device generation, no loss can teach the model what an unseen generation looks like.

Error seven is optimising the test set by repeated trial and error. The result is a selection artefact, even without copying rows between splits. Error eight is assuming a score trained with weights remains a naturally calibrated probability. Explicitly check calibration with a representative sample, and be ready to correct or abandon the probability claim. The probability calibration guide illustrates how to inspect reliability separately from discrimination.

Repair these errors systematically. Draw a table with labels, counts, chosen weights and exact mapping. Write down how examples are sampled and how the loss is normalised. Export held-out scores, confusion-matrix counts and per-class calibration information. Ask a domain reviewer to inspect hard false positives and false negatives. If an apparent improvement vanishes after fixing the split or label issue, report that discovery rather than retaining the original inflated metric.

Transfer exercise: from a 5% case rate to a 2% case rate

Now change the context rather than copying the first example. A hypothetical inspection line has 1,500 independently labelled training images. Thirty contain the rare defect and 1,470 do not. For binary balanced weights, the rare-defect weight is 1,500/(2 × 30) = 25. The common-class weight is 1,500/(2 × 1,470) ≈ 0.510204. Total nominal defect weight is 30 × 25 = 750; total nominal normal weight is 1,470 × 0.510204 ≈ 750. Verify these sums before feeding a weight vector into a library.

A separately evaluated, fictional decision rule identifies 24 of the 30 defect images, misses six, and flags 147 normal images incorrectly. It correctly leaves 1,323 normal images alone. The four cells sum to 24 + 6 + 147 + 1,323 = 1,500. Precision is 24/(24 + 147) = 24/171 ≈ 14.04%. Recall is 24/(24 + 6) = 80%. Even though recall matches the earlier example, the review burden is 171 flags to find only 24 genuine defects. This is what a low base rate can do to a seemingly good detector.

Suppose the inspection team can review at most sixty flagged images per shift. What should change? The correct answer is not ‘raise the defect weight to sixty’. That number has no relevant unit or mathematical connection to capacity. Instead rank predictions from a validation set, measure actual defect recall among the highest sixty scores, compare potential thresholds and consider whether a second automated verification stage could reduce unnecessary reviews. Report the new false negatives and risk after checking actual scores; they cannot be derived from a single 171-alert confusion matrix.

Next suppose a new camera replaces the old one. Training weights were based on old labels; the new images may have different focus, lighting, frequency of visible defects and false alarms. Do not keep an old threshold merely because the arithmetic was once correct. Test representative images from the new camera with fresh labels, inspect calibration and compare performance across both cameras. New examples may be more useful than adding even larger weights to old ones.

Finally ask whether some defects are too rare to learn reliably from thirty examples. It may be more defensible to establish a human escalation route for unfamiliar patterns, improve image capture, collect targeted specimens under proper safety procedures, or use an anomaly-detection baseline. Stronger penalties do not create information. That conclusion transfers to fraud alerts, quality checks, moderation queues and academic triage: the scarcity of evidence has to be addressed at the source.

When weighting, focal loss, or neither is appropriate

Try plain loss first when the minority class is already separable, evaluation is strong and operational targets are met. Try class weights when repeated training shows the minority class receives insufficient influence relative to the task, with enough reliable labels to support learning. Try focal loss when a very large population of easy examples dominates optimisation and the hard examples carry genuine additional information. Try sampling changes when batch composition prevents exposure to informative minority cases. Always verify these hypotheses empirically.

Consider costs before doing mathematics. A false negative may cause harm in one domain, while a false positive may consume limited specialist time or trigger an unjustified intervention in another. A single ratio derived from inverse prevalence is not a moral valuation or a policy decision. In consequential uses, reviewers should be able to challenge the labels, thresholds, and downstream actions. Fairness requires subgroup analysis; improving aggregate recall can still worsen outcomes for a small population.

Not every model uses cross-entropy. Ranking models, anomaly detectors and structured predictions may need a different objective. For multiclass problems with heterogeneous error costs, a full cost matrix or hierarchy can be more meaningful than one weight per class. For multilabel settings, co-occurrence and prevalence interact. Do not mistake a tutorial formula for a universal recipe. Document what the training loss approximates, why it is appropriate and which cases remain outside its scope.

Loss weighting also carries resource costs: re-training, tuning, review of noisy examples, and further evaluation. A simpler data-quality improvement can outperform a complicated objective. When a successful result depends on tight assumptions—independent observations, stable labels, a representative test set—state them where readers can see them rather than burying them in footnotes.

Frequently asked questions

Does class weighting make a dataset balanced? No. It balances a particular nominal contribution to a training objective under a chosen formula. The observed labels, coverage of subgroups, and number of independent minority events do not change. You can have equal weighted totals and still have very poor information about one kind of positive case.

Is focal loss simply another name for class weighting? No. Class weights depend on class identity; focal loss changes emphasis with the model’s confidence in the true class. The two may be combined with an α factor, but their effects differ over the course of training. Their best settings are empirical questions.

Should I always use gamma equal to two? No. Gamma two is often used as an illustrative setting because it demonstrates stronger suppression of easy cases, and it appeared in the original focal-loss research. A different task may prefer gamma zero, a smaller gamma or no focal loss. Use controlled comparisons rather than importing another paper’s result as a guarantee.

Does the largest class always get a weight below one? Under the binary inverse-frequency formula, a majority class above half of the observations gets a weight below one and the minority class gets a weight above one. Other heuristics and normalisations differ. Inspect your framework’s reduction rules and do not attach substantive meaning to the number one without specifying the convention.

Will class weights guarantee higher recall? No. They change the loss landscape, and could increase, decrease or leave recall unchanged at a fixed threshold. A model may also lose calibration or precision. Compare held-out score distributions and choose an operating point under the actual resource constraints.

What if I have just five positive cases? There is rarely enough evidence to select among many complex models reliably. Investigate data collection, expert rules, transfer from related tasks, and explicit unknown-case handling. If you do train, communicate the enormous statistical uncertainty. Five positives cannot support a fine-grained sensitivity estimate.

Why did precision fall after weighting? The model may have moved more examples above your existing threshold, increasing true and false positives. A shift in decision score distribution, a prior mismatch or label noise can contribute. Evaluate precision-recall behaviour, not just the single old threshold.

Can I publish an impressive benchmark result without its denominators? That is poor practice. Report positive counts, negative counts, evaluation time period, splitting unit, decision threshold, queue size and relevant uncertainty. Readers need to know whether a reported 80% recall means eight of ten, eighty of one hundred or thousands of independently checked cases.

Reader pathways: students, engineers, reviewers and leaders

If you are a student, begin with the 950/50 example and compute four confusion-matrix cells before working on loss formulas. Change one number, such as false positives, and observe which measures change. Explain to another person why accuracy and recall are not interchangeable. Then derive the balanced-weight denominator from n, K and n_c without memorising a magic pair of values.

If you are implementing a model, preserve a training-only weight calculation, verified class-to-index mapping, documented loss reduction, split manifest, baseline scores, chosen threshold, and reproducibility record. For a training optimisation perspective, the AdamW optimisation guide explains why objective choices interact with parameter updates. Monitor minority examples that repeatedly dominate gradients or disagree with expert labels.

If you review an AI system, request the label definition, sample-size denominators, group and time leakage checks, representative baseline, full confusion matrix, threshold rationale and error-review procedure. Ask how the system will respond when prevalence or label policy changes. Do not confuse a positive case’s high loss with verified importance.

If you lead an education or service team, begin with the people doing the work. What is their capacity? How harmful is each kind of mistake? Where does a false alarm go? Who may override the result? Consider the broader learning principle in learning that transfers into life: a formula matters only when someone can apply, check and revise it in a changed situation.

Primary sources and further reading

The scikit-learn developers document the balanced class-weight formula and examples in compute_class_weight. Consult the documentation corresponding to the exact library version used by a project; the default parameters and accepted shapes can evolve.

The original peer-reviewed focal-loss study is Lin et al., Focal Loss for Dense Object Detection (ICCV 2017). It supplies the modulating-factor formulation and a particular computer-vision evaluation, not evidence for every later application.

The official PyTorch references for CrossEntropyLoss and BCEWithLogitsLoss help readers verify weighting arguments, logits handling and reductions. For related reader problems, consult the eduKateSG articles on data leakage, calibration, and the AI mechanisms overview.