eduKate Singapore

VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Next useful routeCanonical hubHELP

How Intelligence Works | Simpson’s Paradox — Group Mix, Reversed Rankings and Better Decisions

—

by

eduKate Asia

Did you know that Team A can have a higher overall success rate than Team B even when Team B succeeds more often in every comparable category? That is Simpson’s paradox. It happens when the groups being combined have different sizes or different mixes of easy and difficult cases. The aggregate number is still arithmetically correct; the mistake is treating it as the answer to a comparison it cannot support. This is especially important in schools, hospitals, customer support, hiring, public services and organisational dashboards.

The practical fix is to define exactly what the denominator represents, compare relevant groups before combining them, and decide whether the decision calls for actual workload totals, within-group performance, or a rate standardised to a common population mix. Do not claim a causal effect merely because a ranking reverses after stratification. A reversal is a reason to investigate assignment, selection and measurement, not permission to choose whichever percentage flatters a preferred story.

Contents

Why this reversal is mathematically possible

An overall success rate is a weighted average of subgroup success rates. For group g, the weight is that group’s number of cases divided by the total cases for that team or time period. Different teams can have different weights even if they report the same two subgroup labels. When the easier subgroup has a higher success rate, the team with more easy cases receives a higher aggregate score. Thus an aggregate comparison changes when either within-group performance or case mix changes; the dashboard often hides the second ingredient.

Simpson’s paradox is not a mathematical contradiction. Two statements can both be true: A has more successes per total case in its actual workload, and B has more successes per case within every relevant difficulty band. The first speaks to an actual mixture; the second speaks to conditional performance. Problems begin when a leader silently interprets the first as proof of comparative skill. Before making that leap, ask whether the teams faced the same kinds of cases and whether those case labels were determined before the result.

Worked example: the headline ranking

Consider two fictional teams resolving enquiries. Team A handled 100 enquiries and resolved 85. Team B also handled 100 and resolved 59. Their headline resolution rates are A = 85/100 = 85% and B = 59/100 = 59%. Based only on the aggregate, A appears 26 percentage points stronger. Many scorecards would display A in green and B in red. The arithmetic is flawless; all 200 cases and both denominator definitions are clear.

But an enquiry can be routine or complex. Suppose Team A was assigned 90 routine cases and just 10 complex cases, while Team B received 20 routine cases and 80 complex cases. No one has yet said whether the assignment was fair, whether difficulty was measured consistently, or whether either team controlled its case mix. These are precisely the missing facts that separate “number solved under actual demand” from “how well each team performs on comparable work.”

Worked example: the routine cases

Among Team A’s 90 routine enquiries, 81 were resolved. Its within-routine success rate is 81/90 = 90%. Among Team B’s 20 routine enquiries, 19 were resolved. Its within-routine rate is 19/20 = 95%. Team B is ahead by five percentage points among routine cases. That conclusion requires no model or statistical adjustment; it is a direct comparison of the observed proportions. The unequal denominators matter for uncertainty, but they do not change the arithmetic.

We can verify the group totals: A has nine routine cases unresolved, and B has one unresolved. Someone who looks only at the counts of unresolved routine cases could mistakenly conclude that B did better because it had fewer failures, forgetting that it received far fewer routine enquiries. Always compare count with the appropriate exposure. One failure among twenty is not the same performance as nine among ninety unless the case definition and assignment also support the comparison.

Worked example: the complex cases

Team A resolved four of its ten complex enquiries: 4/10 = 40%. Team B resolved forty of its eighty complex enquiries: 40/80 = 50%. Despite the same absolute count of complex successes, B handled eight times as many complex cases and achieved a ten-percentage-point higher complex success rate. B therefore beats A in both named strata: 95% versus 90% routine, and 50% versus 40% complex.

Now reconstruct the aggregate rather than trusting a convenient chart. A resolved 81 routine plus four complex = 85; 85/100 = 85%. B resolved 19 routine plus 40 complex = 59; 59/100 = 59%. We have verified the reversal in both directions without rounding tricks. The example is invented for teaching. It does not describe real teams or show that random allocation would have produced the same rates. The paradox is in the aggregation, not in hidden real-world evidence.

The missing variable is the case mix

For Team A, routine cases account for 90/100 = 90% of the work and complex cases for 10%. Its total rate is therefore 0.90 × 0.90 + 0.10 × 0.40 = 0.81 + 0.04 = 0.85. For B, routine cases are 20% and complex cases 80%. Its total rate is 0.20 × 0.95 + 0.80 × 0.50 = 0.19 + 0.40 = 0.59. Writing the weighted averages makes the otherwise surprising reversal predictable.

The weights differ so much that B’s higher conditional success rates cannot overcome its much more difficult actual workload. Put differently, the aggregate mixes two effects: performance conditional on difficulty and the composition of difficulty. A manager who asks “Who solved more of the work they actually received?” should see the real totals. A manager who asks “Who might perform better on the same mix of requests?” needs a different comparison. Neither question can be answered honestly by relabelling the other.

Standardisation asks a deliberately different question

Suppose the organisation chooses a common reference mix: half routine enquiries and half complex enquiries. Under that hypothetical mix, A’s standardised expected rate is 0.50 × 90% + 0.50 × 40% = 65%. B’s is 0.50 × 95% + 0.50 × 50% = 72.5%. On this declared mix, B is ahead by 7.5 percentage points. This does not rewrite the 85% and 59% actually observed. It estimates what the two sets of subgroup rates imply under a shared distribution.

The reference mix must be selected for a defensible reason: perhaps the next quarter’s expected workflow, a stable external population or an agreed policy distribution. If we instead use 30% routine and 70% complex, A becomes 0.30 × 90% + 0.70 × 40% = 55%, while B becomes 0.30 × 95% + 0.70 × 50% = 63.5%. The numerical results change with the mix; the fact that B wins in both strata means B stays ahead for every common convex combination of those two strata, assuming the subgroup rates are valid for that setting.

Actual performance and counterfactual comparison are not rivals

People often argue about whether to publish the crude or adjusted rate as if only one could be true. That is a false choice. The crude rate describes successes relative to the workload actually encountered; it matters for staffing, queue size, customer experience and resource allocation. A properly standardised rate describes a different comparison under an explicitly stated reference distribution; it can be useful when the question is about conditional performance rather than total output.

A public dashboard can show both, alongside the subgroup counts. For the invented teams, display A: 85 of 100 actual cases resolved, with 90 routine and ten complex. Display B: 59 of 100 actual cases resolved, with 20 routine and eighty complex. Then show the 50/50-standardised rates of 65% and 72.5%, labelled clearly as standardised estimates. Nobody should confuse an adjusted hypothetical rate with a count of services actually delivered.

The causal question is harder than the numerical question

Conditional success rates do not automatically identify what would happen if workers swapped teams, workloads or supervisors. Perhaps B has better training; perhaps case-difficulty codes conceal meaningful differences; perhaps unresolved cases are transferred selectively; perhaps difficult cases on A and B differ in ways the two labels cannot capture. Observational comparisons are vulnerable to selection, confounding and measurement effects even when the percentages are computed impeccably.

This is why Stanford’s substantive review of Simpson’s paradox connects the reversal to causal inference rather than treating it as a mere spreadsheet curiosity. A causal claim needs a credible assignment or adjustment strategy and a reason to believe the necessary assumptions. Stratification can expose a misleading aggregate; it cannot by itself prove that one team’s policy caused better outcomes. “Better in each observed stratum” is a description of data, not yet a controlled experiment.

Confounder, mediator and collider are different roles

A confounder is a factor that influences both which treatment or team is assigned and the outcome. Case difficulty could be a confounder if challenging enquiries are preferentially routed to B and are inherently harder to resolve. Adjusting for genuine pre-assignment difficulty may make a comparison more relevant. A mediator is a variable on the pathway through which a policy changes the outcome; automatically adjusting it away can erase part of the policy effect you want to measure.

A collider is a variable caused by two other variables. Conditioning on a collider can create a spurious association even when the original variables were not associated. For example, a dataset containing only escalated cases may select on both initial severity and staff response, distorting the apparent relationship between staff and outcomes. These roles cannot be read from column names. Draw the process and ask what caused each variable, when it was measured and why a row entered the dataset before deciding which breakdown is appropriate.

When to stratify and when not to

Stratify when an important pre-existing factor plausibly changes the outcome and varies between the groups you are comparing. Cases can be split by initial severity, product type or an externally defined complexity level if these are measured consistently before work begins. Stratification is less trustworthy when labels are assigned after seeing the result, when the strata are so narrow that most cells contain one case, or when one team has no examples of the comparison group.

The practical question is not “Can I slice this dashboard into 100 categories?” but “Which categories correspond to distinct mechanisms or decisions?” The chosen strata should be documented in advance where possible, and their definitions should persist across reporting periods. Repeated exploratory slicing until a preferred ranking appears is another form of selective reporting. Sometimes the honest conclusion is that the available data do not support a reliable comparison of two teams, particularly outside overlapping case types.

A second worked example: the metric falls while every group improves

Imagine a programme measuring task completion before and after a change. In the first period, 90 routine tasks had an 80% success rate: 72 successes. Ten complex tasks had a 20% rate: two successes. Total completion was (72 + 2)/100 = 74%. In the second period, ten routine tasks had a 90% success rate: nine successes. Ninety complex tasks had a 30% rate: 27 successes. Total completion was (9 + 27)/100 = 36%. The overall metric fell by 38 percentage points.

Yet routine-task performance improved from 80% to 90%, and complex-task performance improved from 20% to 30%. The decline in the aggregate is explained by the shift from 90% routine tasks to just 10% routine tasks. This is a fictional arithmetic example, not a claim that a real intervention improved performance. It shows why a responsible review should not praise or condemn a programme on a headline rate without first investigating how the work entering the system changed.

Separate workload change from within-group change

One useful audit decomposes change into the effects of changed group weights and changed group rates. There is no unique allocation of an interaction term unless you specify a decomposition convention, so avoid presenting one decomposition as nature’s only possible answer. A simple descriptive display can hold old subgroup rates fixed under the new mix, then replace them with the new rates. In the second example, applying the old routine and complex rates to the new 10/90 mix yields 0.10 × 80% + 0.90 × 20% = 26%.

Compare actual earlier 74% with 26% under the new mix but old performance: 48 percentage points of the displayed decline are associated with composition under that chosen sequence. Updating the subgroup rates to the new 90% and 30% yields 36%, an increase of ten percentage points under the new mix. The net result is −48 + 10 = −38 percentage points. Change the decomposition order and its components can differ; show the convention and avoid interpreting this arithmetic as causal proof.

Measurement definitions can manufacture an apparent reversal

A customer success rate measured over tickets opened this month is not automatically comparable to a success rate measured over tickets closed this month. One denominator is an arrival cohort; the other is a completion cohort. Long-running complex cases can be excluded by the second rule until they close, making a team with slow, difficult work look artificially strong or weak depending on timing. A metric audit begins by naming the unit, start event, outcome, time horizon and handling of incomplete cases.

Suppose the dashboard calls every reopened ticket a new case. A team with more handoffs may inflate its denominator, while another team simply annotates the original record. Unless the case identity policy is consistent, a Simpson-style subgroup reversal may be compounded by counting errors. Verify uniqueness, duplicates, censored outcomes, and denominator definitions before debating whether the observed aggregate should be adjusted. No statistical sophistication rescues an untrusted measurement process.

Why denominators should be visible next to percentages

A 95% success rate from 19 of 20 cases has more sampling uncertainty than 95% from 950 of 1,000 cases, assuming comparable independent trials. Showing denominators discourages treating every displayed percentage as equally precise. In the fictional first example, Team A’s 40% complex success rate comes from four of ten cases. A change of one success would move it ten percentage points, much more than a one-case change to B’s 40 of 80 complex successes.

Choose intervals suited to the model and sampling design rather than pretending the exact sample fraction is a population truth. When cases are clustered by customer, class or team member, independent-binomial intervals may be too narrow. Small cells may also pose privacy risks. Publish counts when safe and suitable; otherwise disclose suppression rules and the uncertainty or limitation. The aim is an understandable comparison, not a forest of decimal places suggesting certainty the evidence does not support.

Why a pooled difference can change sign

Write the difference between A’s and B’s aggregate rates as Σ w_A,g p_A,g − Σ w_B,g p_B,g. If the weights were identical in both groups, the difference would be Σ w_g(p_A,g − p_B,g). In that common-weight situation, if B exceeds A in every stratum, A cannot exceed B in the weighted average. The reversal becomes possible precisely because the groups use different weights, or because grouping and comparison rules change.

This algebra gives an immediate diagnostic. If you are looking at a surprising ranking reversal, reconstruct the two sets of weights first, then the within-stratum proportions. Do not hunt for a mystical average. In our first case, A puts 90% of its weight on the routine rate while B puts 20% there. The different weights are large enough to overturn B’s five- and ten-point conditional advantages in the observed aggregates. The computation can be checked with elementary multiplication.

Decision quality depends on the question, not one winning percentage

A service manager deciding where to send additional complex enquiries might use subgroup outcomes and capacity constraints. A director assessing total unresolved backlog needs absolute counts and arrival patterns. An educator comparing learning programmes across schools should consider baseline attainment and selection rules, not merely compare percentages of students passing. A patient evaluating treatment evidence needs clinical comparability, uncertainty and causal study design, not whichever adjustment makes a treatment look strongest.

Every one of these decisions asks something different of the same table. Begin with a written decision question: “At the expected future mix of work, which process is likely to resolve more cases with acceptable quality?” This directs attention to reference mix, relevant outcomes, resource use and uncertainty. A less useful question is “Which team looks best on the screenshot?” because it invites selective grouping. Intelligent organisations teach people to connect a reported statistic to its denominator, mechanism and decision.

School results: a frequent trap in educational comparisons

A fictional school reports 80% passing among 100 students, while another reports 70% among 100. If the first school has a higher share of pupils who entered with strong prior attainment, the difference may be due partly to case mix. But do not assign children to simplistic fixed categories or assume that a lower initial score determines a lower eventual outcome. A responsible comparison defines prior measures, asks how students were selected and examines progress, access and educational goals.

Even a stratified pass rate is not a complete measure of teaching quality. Schools can differ in subject availability, support provision, exam selection and cohort mobility. A high aggregate pass percentage can coexist with weak progress for some learners. A lower aggregate can coexist with exceptional support to learners who faced greater barriers. The relevant reader question is what the institution actually helped pupils learn, with what evidence and under which opportunities—not how to engineer a flattering league table.

Hiring, promotions and selection filters

Suppose one hiring team considers many senior candidates and another mostly entry-level applicants. The share of candidates passing an assessment can differ even if evaluators apply the same rubric within comparable job families. Conversely, a seemingly favourable adjusted comparison can still conceal unfair access to interviews or inconsistent scoring. The denominator of applicants is not the denominator of interviewed people, and neither is automatically the eligible population. Each stage is a selection mechanism.

For performance audits, preserve stage counts and role definitions. Separate applicants, eligible applicants, shortlisted candidates, assessed candidates, offers and acceptances. Ask who was excluded before the metric was calculated. Breaking data into subgroups can reveal an aggregate reversal, but an assessment of fairness also needs to examine whether those subgroups and stage transitions themselves result from unjustified policies. Analysis should not be used to excuse inequity by conditioning on variables created by the process under scrutiny.

Healthcare and public statistics: crude versus standardised rates

Health statistics offer a carefully documented example of why population mix matters. The US CDC technical notes on incidence and death rates explain that crude rates are influenced by the age distribution of a population and that age adjustment supports comparisons across populations with different age structures. The CDC age-adjustment definition, updated 16 September 2026, presents direct standardisation as a weighted sum of age-specific rates under a standard population.

That official guidance does not mean every organisational group should be adjusted using age or that an adjusted number is a literal count of people. The right weights depend on the question and the populations under comparison. In health planning, crude burden can be vital for resource needs, while age-adjusted rates can make comparisons less dominated by differing population ages. Present both with a clear description of what each is intended to answer, and avoid treating standardisation as a causal experiment.

What to put on a trustworthy dashboard

For each group, show the number of cases eligible, the number of successes, the calculated rate and the stratifying factors with their definitions. Offer an optional standardised measure with the reference mix stated directly beside it. Show the date window, inclusion rules and whether unresolved cases were excluded, treated as failures or handled as censored observations. A chart should not silently switch its denominator when users apply a filter. Users need enough context to reconstruct the headline value.

Add a short interpretation notice: “Actual totals describe the work received; adjusted comparisons describe the selected reference mix.” Show when subgroup counts are too small for stable comparisons. For live dashboards, protect against a partial refresh that updates numerators but not denominators. Let reviewers drill down to validated records under appropriate permissions. A percentage without its counting rule is a persuasive shape, not an audit-ready measurement.

Detecting a possible reversal in routine analysis

Start with a simple table of outcomes by candidate group and a small set of plausible pre-outcome factors. Compute marginal rates, subgroup rates and subgroup sizes from the same row-level records. Flag cases where the sign of an aggregate difference differs from the sign in each sufficiently supported stratum. Do not run every possible split across hundreds of columns and publicise only the most dramatic reversal; exploratory search inflates the chance of finding a misleading pattern.

When a candidate reversal appears, verify the row counts and source records, check which observations were missing or duplicated, and ask whether the subgroup factor is a confounder, mediator, collider or merely correlated marker. Compute a common-weight comparison with a justified reference distribution. Then perform sensitivity checks to alternative grouping boundaries and missingness assumptions. The result should be a diagnosis of what the original metric can and cannot say, not a contest to replace one oversimplified headline with another.

A counterexample: subgroup rates that do not reverse

If two teams have the same case mix, standardising to that mix changes neither team’s aggregate. Suppose both receive 50 routine and 50 complex cases. A resolves 45 of 50 routine and 20 of 50 complex, giving 65 of 100 overall. B resolves 47 of 50 routine and 25 of 50 complex, giving 72 of 100. B exceeds A in each stratum and also in the aggregate. There is no paradox to solve; the weights are aligned.

Even with unequal mixes, a reversal is not guaranteed. A sufficiently large within-stratum advantage can dominate composition, or the group weights may simply be too similar. That is why “Simpson’s paradox” should not become a label pasted onto every suspicious chart. Use it precisely when pooling strata hides, eliminates or reverses an association. The correct mathematical test is in the numbers, and the correct substantive interpretation is in the process that produced them.

Error diagnosis: five ways an audit goes wrong

The first error is denominator drift: one chart uses applications received and another uses decisions completed. The second is label drift: “complex” changes definition between reporting periods. Third is a selection filter: only escalated cases enter one dataset. Fourth is an after-the-fact subgroup: severity is assigned using information observed after the outcome. Fifth is false precision: a difference of several percentage points is presented as decisive despite ten cases or correlated repeated records. All five can create persuasive but misleading stories.

Repair starts with an event and measurement map. Restore common denominator rules; reconcile totals to individual records; preserve the pre-outcome timing of candidate stratifiers; disclose small-cell uncertainty; and retain an audit trail of any regrouping. If the dataset cannot support a comparative conclusion, say so. Asking for better data can be more intelligent than producing another adjusted percentage from inadequate evidence.

Changed-context transfer: supplier performance

Move the lesson to an equipment procurement decision. Supplier A delivered 60 simple parts and 40 specialised parts. On simple parts its on-time rate was 54/60 = 90%, and on specialised parts 16/40 = 40%. Its overall on-time rate was 70/100 = 70%. Supplier B delivered 20 simple and 80 specialised parts. It achieved 19/20 = 95% simple and 40/80 = 50% specialised, for 59/100 = 59% overall. B is stronger in each comparable stratum even though A has the higher raw headline.

For a forthcoming order expected to contain 25% simple and 75% specialised parts, A’s mix-standardised expected rate is 0.25 × 90% + 0.75 × 40% = 52.5%. B’s is 0.25 × 95% + 0.75 × 50% = 61.25%. The arithmetic is a transfer of the mechanism, not a real forecast. Procurement must additionally assess price, supply constraints, timing, category consistency, capacity, quality definitions and whether the new order resembles the historical strata.

How to communicate two true results without confusing readers

Write the sentence with its denominator attached: “A resolved 85% of the actual enquiries assigned, most of which were routine.” Then write the conditional comparison: “B had higher observed resolution rates within both routine and complex categories.” Then write the standardisation result with its hypothesis: “At a declared equal mix of those two categories, B’s observed stratum rates imply a 72.5% weighted rate versus A’s 65%.” These sentences can coexist without accusing anyone of miscalculation.

Separate description from recommendation. If the question is about rewarding skill, investigate assignment and uncertainty before choosing a policy. If the question is about overall service coverage, use actual counts and demand forecasts. If the organisation must communicate publicly, avoid making the least representative single number the headline. A good explanation preserves the tension that the data genuinely contain. It does not eliminate complexity by hiding the denominator.

Limitations: two categories are rarely the whole world

Our worked examples use perfectly labelled routine and complex cases, no missing outcomes and equal total row counts. Real records contain selection effects, changes in process, mislabelled difficulty, censoring, clustering and hidden common causes. A reversal can disappear under a better grouping or appear after conditioning on an inappropriate variable. Direct standardisation can be unstable when one group has very few cases in a stratum, and it may require extrapolation when a group has no comparable observations.

A larger dataset does not automatically remove these limitations. More rows can yield precise estimates of the wrong estimand if the outcome definition is biased. An adjusted score should be accompanied by the assumptions under which the adjustment is relevant and by the uncertainty arising from small or missing cells. For high-stakes policies, involve domain experts and appropriate statistical review. The aim is not to make a dashboard look scientific; it is to keep decisions proportionate to what the evidence actually supports.

FAQ: Are both aggregate and subgroup statistics correct?

Yes, if both were computed from consistent records and denominators. The aggregate is a weighted average under the observed mix, while the subgroup rates condition on category membership. Their differing rankings do not create an arithmetic contradiction. The important mistake is using one rate to answer the other rate’s question without checking the underlying selection and grouping process.

FAQ: Should I always adjust for more variables?

No. Conditioning on a true pre-existing confounder can help a particular comparison, but adjusting for a mediator may remove the effect you wanted to assess, and adjusting for a collider can introduce bias. Too many small strata can also produce unstable estimates. Choose adjustment variables from substantive knowledge of the data-generating process, not simply because a software package offers them.

FAQ: Does the paradox prove discrimination or unfairness?

No. It can reveal that an aggregate comparison conceals meaningful group differences and may prompt a fairness investigation. But it does not independently identify the causal mechanism. An unfair assignment process could create the mix, or the mix could have a legitimate operational explanation. The correct response is to examine access, policy, selection and comparability, not to declare guilt or innocence from one reversal.

FAQ: Can I compare only actual totals when planning resources?

Actual case counts and crude rates are often essential for queue staffing and public service planning. A service dealing with more complex requests may need more resources even if its aggregate resolution percentage is lower. But actual totals should not be presented as evidence about performance on interchangeable workloads. Include subgroup demand and the reference population for any hypothetical comparison.

FAQ: What if one subgroup has almost no cases?

Report the limitation explicitly. An estimated rate based on one or two observations can change dramatically with one additional outcome. Standardising with such a rate can create a deceptive level of certainty. Consider broader predefined strata, a suitable hierarchical model or a request for more data, while being clear about assumptions. Sometimes the honest decision is to postpone a comparative ranking rather than manufacture one.

Sources and a pathway for readers

For a technical account of the phenomenon, its conditional relationships and causal interpretation, read the Stanford Encyclopedia of Philosophy: Simpson’s Paradox, substantively revised in June 2026. Original empirical and methodological research includes Kievit and colleagues’ practical guide in Frontiers in Psychology and Julious and Mullee’s BMJ discussion of confounding and Simpson’s paradox. For a contemporary primary official method on age-adjusted comparisons, use the CDC’s 2026 age-adjustment explanation. These sources support the mechanism, not the fictional figures in this guide.

For broader judgement, go to the How Intelligence Works hub. Continue with Base-Rate Neglect — Rare Events and Better Decisions when the error is forgetting prevalence inside a probability calculation, or Ambiguity Aversion — Known Risks and Unknown Probabilities when the difficulty is uncertain chances rather than weighted aggregation. The Gold Standard of Education links this reading habit to transferring evidence across changed contexts.

Discover more from eduKate Singapore

Subscribe now to keep reading and get access to the full archive.

Continue reading