eduKate Singapore

VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Next useful routeCanonical hubHELP

How AI Works | Inverse Propensity Weighting — Overlap, Extreme Weights and Causal Comparisons

—

by

eduKate Asia

Contents

Why comparing two observed groups is not automatically causal

Did you know? If students who choose extra practice already differ from students who do not, a simple comparison of their eventual scores mixes the programme with the reasons for joining it. Inverse propensity weighting is one statistical approach to such a problem: estimate how likely each observed participant was to receive their actual treatment or programme given measured prior characteristics, then give less frequently observed treatment-pattern combinations more weight in the analysis.

The practical answer is not simply ‘divide by a probability’. First define the causal question and target population, specify treatment and outcome, identify pre-treatment confounders, estimate treatment probabilities without leaking later information, check overlap and weight stability, and compare outcomes under the intended estimand. The method can remove confounding explained by measured variables under strong assumptions, but it cannot manufacture missing counterfactual evidence or eliminate unmeasured confounding.

Suppose an optional learning programme attracts many already confident students. Their improved results could reflect the programme, prior confidence, family support or several causes at once. Weighting uses measured covariates to create a pseudo-population in which treatment assignment is more balanced on those covariates. This is an analytical construction, not proof that real participants were randomly assigned.

The authoritative methodological foundations include Cole and Hernán, Constructing Inverse Probability Weights for Marginal Structural Models and the original open textbook Hernán and Robins, Causal Inference: What If. They explain why exchangeability, positivity and consistency matter. Numerical cases below are invented educational illustrations, not measured benefits of eduKate tuition or any named service.

The estimand comes before the predictive algorithm

A causal estimand defines the quantity an analyst wants to learn. For a binary intervention A, one common target is the average treatment effect in a population: the average outcome if everyone received treatment minus the average outcome if nobody received it, under a well-defined intervention and relevant assumptions. A different estimand might be the effect among people who actually received treatment.

The distinction matters because weighting schemes can target different groups. Average treatment effect (ATE) weights typically use the inverse probability of the treatment each person actually received. Average treatment effect among the treated (ATT) uses another weighting arrangement so controls resemble the treated population. These targets are not synonyms, especially if treatment response differs by baseline risk.

A prediction model may estimate whether a particular person is likely to join a programme. But strong predictive accuracy is not itself the causal question. The propensity model supplies balancing weights for a separate outcome comparison. A model that predicts treatment assignment extremely well can even produce poor overlap for causal analysis, because certain people almost never appear in the comparison condition.

Write down the intervention, comparison, population, outcome and time window before fitting anything. Does ‘receiving tuition’ mean attending one lesson, completing a ten-week programme or receiving a personalised study intervention? Does the outcome mean a school score, durable independent transfer or eventual enrolment? Different definitions describe different causal questions.

Many observational analyses fail before statistical estimation because the intervention is vague. No clever weight can turn a poorly defined comparison into an accountable claim about what would happen under a policy change.

What the propensity score actually predicts

For binary treatment A and measured baseline covariates L, the propensity score e(L) is the conditional probability of receiving treatment: P(A=1|L). For someone who received treatment, the ordinary inverse-probability weight for ATE is 1/e(L). For someone who did not, it is 1/[1−e(L)]. Both represent the inverse probability of the observed assignment given their covariates.

Consider two otherwise similar students with estimated e(L)=0.20. A treated student has weight 1/0.20=5, because receiving treatment was relatively uncommon in that covariate stratum. An untreated student has weight 1/0.80=1.25 because remaining untreated was more common. The weights help reshape each treatment group toward the target covariate distribution.

These weights are not confidence scores that the intervention will work. A person with weight five is not five times as likely to benefit, and a low propensity does not prove that treatment is good or bad for them. The score describes assignment likelihood under a fitted or specified treatment model.

Propensity scores can be estimated with logistic regression or suitable machine-learning methods, but flexibly predicting assignment is not a substitute for causal design. The chosen covariates should reflect a defensible confounding structure and occur before the intervention. Inserting variables affected by treatment can introduce bias or change the estimand.

The final question is whether weighting made the measured baseline groups comparable enough to support the intended analysis. A low log-loss for the propensity predictor does not certify balance. Diagnostics of the reweighted covariate distributions and available treatment overlap are essential.

Three assumptions a weight cannot replace

Conditional exchangeability means that, after accounting for the measured confounders, treatment assignment is as-if independent of the potential outcomes under the intended causal model. In practical terms, there must be no important unmeasured common cause of treatment and outcome remaining in the comparison. This is a strong substantive assumption, not something that a fitted propensity model automatically verifies.

Positivity or overlap means that each relevant covariate combination in the target population has a nonzero chance of receiving every treatment condition being compared. For a binary intervention, we want 0<e(L)<1 where the causal estimand requires both possibilities. If nobody with a certain eligibility profile ever receives a treatment, observed data cannot directly reveal the counterfactual outcome for that group under that treatment.

Consistency means an individual’s observed outcome under the treatment they actually received corresponds to the relevant potential outcome for that well-defined treatment. Ambiguous versions of the intervention can undermine this premise. If ‘extra help’ ranges from five minutes to a year of intensive tutoring, a single binary treatment may hide multiple incompatible interventions.

Interference between units can matter too. A student’s participation may change a classmate’s learning opportunities or teacher attention, contradicting simplistic models that treat every student’s outcome as unaffected by others’ assignments. Clustered schools and group interventions require suitable designs and uncertainty handling.

These assumptions appear in the original epidemiological methodology. The Cole and Hernán article explains the practical meaning of exchangeability and positivity. Statistical balance is evidence about measured covariates; it does not prove the absence of hidden causes.

A fully solved example with two risk groups and 100 participants

Consider a fictional observational study of one optional practice programme with exactly 100 learners. Eighty are in a lower-baseline-risk stratum L and twenty in a higher-risk stratum H. Within L, twenty receive the programme and sixty do not, so the empirical treatment propensity is 20/80=0.25. Within H, fifteen receive it and five do not, so the propensity is 15/20=0.75. These fractions are constructed as known assignment probabilities for the example, not claimed estimates from real data.

Suppose the outcome is meeting one prespecified independent learning criterion. In L, twelve of twenty treated learners meet it, and twenty-four of sixty untreated learners do. Their observed rates are 12/20=0.60 and 24/60=0.40. In H, six of fifteen treated learners meet it, and one of five untreated learners does, giving 6/15=0.40 and 1/5=0.20. Each stratum shows a twenty-percentage-point difference in the observed rate.

First compute the unadjusted treated and untreated means. Across both strata, eighteen treated learners meet the criterion among 35 treated: 18/35≈0.514286. Untreated successes total twenty-five among 65 untreated: 25/65≈0.384615. The crude observed difference is approximately 0.129670, or 12.97 percentage points. That is smaller than the twenty-point within-stratum contrasts because the treated group contains a higher share of high-risk learners.

Now apply ATE weights. Treated learners in L receive 1/0.25=4 each; treated learners in H receive 1/0.75=4/3 each. The weighted treated denominator is 20×4+15×(4/3)=80+20=100. Weighted treated successes are 12×4+6×(4/3)=48+8=56, so the weighted treated mean is 56/100=0.56.

Untreated learners in L receive 1/(1−0.25)=4/3 each, while untreated learners in H receive 1/(1−0.75)=4 each. The weighted untreated denominator is 60×(4/3)+5×4=80+20=100. Weighted untreated successes are 24×(4/3)+1×4=32+4=36, so the weighted untreated mean is 36/100=0.36. The weighted difference is 0.56−0.36=0.20, or twenty percentage points.

This is also the standardised within-stratum difference: L accounts for 80% of the target population and H for 20%, giving 0.8×(0.60−0.40)+0.2×(0.40−0.20)=0.8×0.20+0.2×0.20=0.20. The agreement is built into the constructed example. It illustrates the weighting identity under this simplified setup; it does not establish a real causal treatment effect.

A pseudo-population is an accounting tool, not new participants

In the worked example, there were 100 real participants, not 200. The weighted denominators of 100 for treated and 100 for untreated are pseudo-population totals created by giving observations different analytical influence. The same real learner contributes according to the assigned weight; no additional learners were observed or created.

The pseudo-population can make the baseline strata proportions match the target group in each treatment arm. Before weighting, only 20/35≈57.14% of treated participants were in L while 60/65≈92.31% of controls were in L. After weighting, L accounts for 80/100=80% of each weighted arm. This is exactly the original target distribution of L in the combined study population.

Balance of the illustrated stratum does not mean all confounders were balanced. There may be prior grades, attendance, socioeconomic variables or other determinants unmeasured in the dataset. The synthetic weights were based on only the two-stratum construct. Their mathematical success should not be overstated as substantive exchangeability.

A weighted outcome mean is not the same as ordinary counting of success cases. When presenting results, disclose whether a denominator is an actual sample count, an effective sample size, a weighted total or the population size. If a chart simply says ‘100 treated students improved’, it would be false in this example: only 35 real people received treatment, eighteen of whom achieved the outcome.

Clear labelling protects readers from an unexpected form of denominator neglect. Weights make observed units stand in for a target distribution; they do not provide direct experimental observations under both treatment conditions for the same individual.

Stabilised weights change scale, not the causal story

Unstabilised inverse-probability weights can be large, especially when an observed assignment was unlikely. A stabilised ATE weight often multiplies the numerator by the marginal probability of the observed treatment: for treated units, P(A=1)/e(L); for controls, P(A=0)/[1−e(L)]. In the illustrative study, P(A=1)=35/100=0.35 and P(A=0)=65/100=0.65.

For treated L participants, the stabilised weight is 0.35/0.25=1.4. For treated H participants it is 0.35/0.75=7/15≈0.466667. The weighted treated denominator becomes 20×1.4+15×(7/15)=28+7=35. Weighted successes are 12×1.4+6×(7/15)=16.8+2.8=19.6. The weighted treated mean remains 19.6/35=0.56.

For untreated L participants, the stabilised weight is 0.65/0.75=13/15≈0.866667. For untreated H it is 0.65/0.25=2.6. The denominator is 60×(13/15)+5×2.6=52+13=65. Weighted untreated successes are 24×(13/15)+1×2.6=20.8+2.6=23.4. Their weighted mean is 23.4/65=0.36.

Thus the weighted difference remains 0.20 in this constructed stratified example. Stabilisation reduces the overall scale of weights in each arm while preserving the same within-arm normalised comparison. This does not remove all variance or guarantee positivity; extreme ratios can still occur when an estimated denominator is near zero.

The choice between standard and stabilised weights depends on the design, estimator and inference method. Always state the numerator and denominator formula, because calling both simply ‘propensity weighting’ hides important implementation differences.

Positivity and overlap: what happens when the denominator approaches zero

Now change a treated learner’s estimated propensity to 0.01. Their unstabilised treated weight becomes 1/0.01=100. One such participant can exert as much nominal weighted influence as a hundred participants with weight one. That may make an estimate extremely sensitive to one unusual outcome or a slightly inaccurate propensity model.

If the estimated propensity were 0.001, the corresponding treated weight would be 1,000. This is arithmetically valid but usually an alarm about limited overlap, possible model extrapolation or data support. A numerical optimiser can return a finite answer even when causal identification is weak. A stable-looking decimal is not a guarantee of credible inference.

Inspect the distribution of predicted propensities separately for treated and untreated groups. Look for parts of the covariate space with practically no comparison units, concentrations of near-zero or near-one probabilities, and large maximum weights. Visual overlap plots, summary percentiles and weight histograms can expose problems that mean weight alone conceals.

Clipping or trimming very large weights can reduce numerical variability but can also introduce bias or change the target estimand. There is no universal safe threshold. Investigators should predefine or transparently justify their handling of limited overlap and report sensitivity analyses, rather than silently discarding inconvenient cases after seeing the outcome.

When structural positivity truly fails, there may be no data-based way to estimate the target intervention effect for the full original population under the proposed assumptions. Narrowing the target to an overlap population or changing the study question may be more honest than aggressively extrapolating.

The original literature distinguishes structural absence of treatment options from chance missing cells in finite data. Modelling may interpolate under assumptions in the latter case, but it cannot create observations where treatment is impossible by design.

Effective sample size explains why 100 weighted units are not 100 independent observations

Weighted observations can carry very unequal influence. One useful diagnostic is Kish-style effective sample size: ESS=(sum of weights)^2/(sum of squared weights), computed for the group and estimator where it is meaningful. ESS is a description of weight concentration under this formula; it does not automatically become a valid independent-sample count for every causal model.

In the worked treated group, twenty lower-risk participants have weight four, and fifteen higher-risk participants have weight 4/3. The total of weights is 20×4+15×(4/3)=100. The sum of squares is 20×16+15×(16/9)=320+26.666667=346.666667. The ESS is 100^2/346.666667≈28.85, smaller than the 35 actual treated participants.

For the untreated group, sixty L participants have weight 4/3 and five H participants have weight four. Their total weight is also 100. The sum of squared weights is 60×(16/9)+5×16≈106.666667+80=186.666667. Thus ESS≈10000/186.666667≈53.57, lower than the 65 actual controls.

Notice how an apparently balanced weighted pseudo-population of 100 in each group can rely on less effective information than the raw record count suggests. A few high-weight participants can dominate the contribution. The method’s uncertainty analysis must recognise the estimated weights and study design, not treat pseudo-population totals as actual independent observations.

If stabilised weights merely multiply all weights within one arm by a constant, this particular ESS formula is unchanged for that arm because the constant cancels. Stabilisation can still affect model fitting or numerical properties under other choices; ESS must be interpreted within the actual estimator.

A good propensity model is not necessarily a good balancing model

An assignment model may achieve impressive discrimination by nearly separating treated from untreated participants. For a prediction competition that could look desirable. For causal weighting, near-perfect separation often means some populations have almost no treatment overlap, producing unstable weights and undermining extrapolation.

After weighting, compare the distributions of measured pre-treatment covariates across the treatment arms. Standardised mean differences or other balance diagnostics can help quantify remaining imbalance. Examine important interactions and non-linear patterns when the design calls for them, rather than assuming a single mean comparison proves balance of the full distribution.

Keep outcome data out of leakage-prone propensity fitting when the design aims for a treatment-assignment model determined by baseline variables. The study protocol should distinguish covariates measured before assignment from variables caused by treatment. Conditioning on a post-treatment mediator or collider can alter the causal interpretation in ways that a flexible algorithm cannot repair.

Do not select the propensity model merely by comparing area under an ROC curve. Prediction accuracy and covariate balance are related imperfectly. A statistically modest propensity model that achieves strong balance and overlap may be more useful for the target causal analysis than a classifier with extreme fitted probabilities.

When machine learning estimates propensities, flexible learners may help model complex relationships, but cross-fitting, regularisation and uncertainty handling require care. Report how the score model was trained, validated and used; a black-box score without diagnostics is not credible causal evidence.

Weighting versus regression, matching and randomisation

Outcome regression models the relationship between outcomes, treatment and observed covariates. Standardisation uses that fitted relationship to predict average potential outcomes under treatment options. Propensity weighting instead models treatment assignment to balance the covariates. Both rely on substantive identification assumptions, and either can be biased by serious model misspecification.

Matching seeks comparable treated and untreated units within the measured covariate space. It can improve interpretability but may exclude unmatched participants and thereby change the target population. Weighting generally uses all eligible observations while changing their influence, unless trimming or overlap restrictions are applied. Neither method eliminates hidden confounding merely by improving observed balance.

Randomised assignment, when implemented correctly, creates an experimental basis for treatment comparison. That often offers stronger protection against confounding than observational modelling, but practical trials still face nonadherence, missing outcomes, interference and loss to follow-up. Inverse weighting can also be used for some missingness or censoring problems under corresponding assumptions.

Doubly robust estimators combine treatment weighting and outcome modelling in ways that may remain consistent when one of two specified nuisance models is correct under other assumptions. The phrase doubly robust does not mean safe against all errors or positivity failure. The analysis still requires a defined estimand, appropriate data, valid inference and careful implementation.

The original methodological paper on inverse weights explains why model assumptions and weight distributions matter. Causal inference is not an automatic upgrade from prediction just because it uses a mathematical correction.

Treatment weighting and sampling weighting are related but different

Inverse probability methods also appear when analysts compensate for unequal sampling, survey nonresponse or outcome observation. The common mathematics gives greater influence to observations with lower probabilities of being included or observed. But the actual probability being inverted is different from the probability of receiving a treatment.

For a survey, the relevant weight might reflect the probability of inclusion in the sample. For treatment effect estimation, it reflects the probability of receiving the observed treatment given confounders. For censoring, it may reflect the probability of remaining observed through a period. Mixing these without defining the mechanism leads to meaningless combined weights.

Consider an invented survey of a class where only half of one student subgroup responds and every member of another responds. Assigning weight two to responders from the first subgroup and weight one to responders from the second can reconstruct counts under appropriate response-model assumptions. That is not an estimate of the effect of attending tuition; it is a correction for a missing-observation mechanism.

If both treatment selection and survey participation matter, an estimator may involve more than one component of weighting. Its interpretation depends on which population is targeted and assumptions about both mechanisms. Multiplying weights mechanically without a design can magnify instability and obscure the estimand.

Readers should therefore ask a simple question before calculating: probability of what? If the denominator describes admission to an observed sample rather than treatment receipt, name it accordingly. The mathematical family is shared, but the reader job and causal claim are not.

Uncertainty and observational findings require independent checks

Confidence intervals must reflect the study design and the fact that propensity scores may be estimated rather than known. Treating estimated weights as fixed in every standard error calculation can misstate uncertainty. Different estimators use robust variance methods, bootstrapping or other techniques under specific assumptions.

Clustering matters if students share a classroom, site, mentor or other common influence. Treating correlated observations as independent can produce overconfident intervals. A model should account for the actual randomisation or sampling units and relevant sources of dependence.

Outcome misclassification can also undermine results. If the programme group receives more careful testing or has different missing-data patterns, weighted baseline covariate balance does not ensure comparable outcome measurement. Define outcomes before examining results and evaluate how data quality differs by group.

Unmeasured confounding is not solved by large sample size. A study can precisely estimate an association biased by a hidden cause. Sensitivity analysis, negative controls where appropriate, domain evidence and independent replication help communicate what remains uncertain.

Most importantly, a small change in a fitted propensity model that causes a large change in the estimated effect is a warning worth reporting. Stability checks are more useful than selecting only the specification that produces an attractive story.

Ten mistakes that make a weighting result look stronger than it is

Error one is reporting a causal effect without naming the intervention and target population. Repair by defining the estimand. Error two is placing post-treatment information into the propensity score model. Repair by enforcing temporal feature boundaries. Error three is interpreting a low propensity as evidence a participant benefits more. Repair by separating assignment likelihood from outcome response.

Error four is ignoring nonoverlap. Repair by checking treated and untreated propensity distributions and considering whether the target population needs narrowing. Error five is trimming large weights secretly. Repair by documenting thresholds, rationale and sensitivity. Error six is treating the pseudo-population’s total weighted count as a real sample size.

Error seven is choosing a model only because it predicts treatment well. Repair by assessing reweighted covariate balance and weight concentration. Error eight is ignoring clustering, which makes uncertainty look too precise. Error nine is assuming measured balance proves there are no hidden confounders.

Error ten is reporting a programme success guarantee from an observational weighted difference. The repair is to state exactly what was observed, which assumptions support a causal interpretation and which evidence remains unavailable.

Across these failures, the trustworthy practice is more important than the formula. A useful study tells readers which people and opportunities were counted, how assignment was modelled, how extreme weights were handled and why the outcome comparison is meaningful.

Changed-context problem: a job-training programme with missing overlap

Transfer the logic from students to an invented workplace training programme. Suppose a company wants to know whether training improves successful completion of a new task. Staff in one specialised role all receive training, while staff in another role never receive it. If role also predicts the outcome, treatment and job role are completely confounded in the observed data.

For the first role, estimated e(L)=1; for the second, e(L)=0. A control weight for the first role would require 1/(1−1), which is undefined. A treated weight for the second role would require 1/0, also undefined, although no such observations are present. This exposes structural lack of positivity for a population-wide training effect adjusted for role. No finite reweighting of the actual records can reveal counterfactual outcomes for treatments never observed in those roles.

One possible repair is to change the design: collect ethically appropriate comparisons where both options are available, introduce randomised access if permitted, or restrict the estimand to roles with genuine overlap. Another is to report descriptive differences without claiming a full-population causal effect. The chosen response depends on the real decision and available evidence.

Now change one condition. Within each role, both trained and untrained staff exist, but one role has a treated probability of 0.02. Positivity is technically present under the model, yet an observed treated individual in that role has weight 1/0.02=50. Practical overlap remains poor, and the estimate may be dominated by very few cases. A mathematically finite weight is not proof of adequate information.

The transferable lesson is that statistical correction works within an identifiable evidence structure. When the structure lacks genuine comparisons, gathering better data or changing the question is more honest than increasing model complexity.

Where this guide stops and what it cannot promise

Inverse propensity weighting is not an AI system automatically discovering causal truth. The propensity model can be machine-learned, but the causal interpretation comes from design, assumptions and substantive understanding of how treatment and outcomes arise. A sophisticated learner cannot prove exchangeability from observational data alone.

Small denominators in estimated propensities create extreme weights; clipping may reduce variance while changing bias or target interpretation. Positivity failures cannot be repaired by arbitrary numeric substitutions. The correct action may be to restrict the population, revise the intervention definition or conduct a better study.

The examples here assume two clearly defined baseline strata and known empirical assignment probabilities. Real analyses involve estimation error, missing data, interactions, longitudinal exposures and potentially time-varying confounders. Marginal structural models extend the framework but require extra care; do not generalise the toy arithmetic to complex clinical decisions.

Study results may change when the target population changes. A causal effect among volunteers is not necessarily the same as an average effect for all eligible people. An estimator should announce its scope rather than describe one numerical answer as universal.

All learner and workforce figures in this article were invented teaching examples. No actual model was trained, no treatment was allocated and no external study result was inferred from them. They illustrate how to verify denominators and reveal where evidence ends.

Frequently asked questions

What is a propensity score? It is the probability of receiving a particular treatment conditional on measured baseline characteristics, under a defined assignment model.

Why take its inverse? Observations with uncommon treatment assignments are upweighted so the treatment groups can better represent the chosen target population across measured covariates.

Does a large weight mean large treatment benefit? No. It describes an unlikely assignment under the propensity model, not a predicted individual effect.

What is overlap? It means relevant covariate profiles have nonzero chances of receiving each treatment condition needed for the target comparison. Very weak overlap produces unstable weights.

What is positivity? The formal assumption that required treatment probabilities are nonzero for covariate profiles in the target population. It is essential to identifying certain causal effects.

What is a pseudo-population? It is an analytical reweighting of real observations, not a newly collected sample or an actual randomised experiment.

What are stabilised weights? They multiply inverse-probability weights by a marginal treatment probability or other specified numerator, often reducing their scale while preserving a related estimand under appropriate design.

Can inverse weighting remove unmeasured confounding? No. Its causal interpretation still requires suitable exchangeability and other assumptions, and it cannot account for relevant causes absent from the analysis.

Is propensity weighting the same as class weighting for ML training? No. Class weighting may emphasise rare prediction labels; propensity weighting aims to adjust an assignment or observation mechanism for a defined target population or causal estimand.

Should a school publish a causal success claim from one weighted comparison? Only if the design, assumptions, evidence and uncertainty genuinely support it. The arithmetic itself cannot guarantee a causal conclusion.

Reader pathways and primary methodology

If you are learning statistics, reproduce the four-cell example. Verify the two propensity scores, all four weights, crude comparison, weighted denominators, weighted successes, stabilised rates and effective sample sizes. Then change one cell count and ask which assumptions and arithmetic must be revisited.

If you build AI or data models, separate treatment prediction from causal estimation. Check temporal leakage, positivity, covariate balance and extreme weights. Record the exact estimand before selecting ML algorithms. Do not optimise a propensity classifier in isolation from the causal reader job.

If you are a parent or organisational decision maker, ask whether the people who joined a programme differed from those who did not. Demand clear denominators, comparable outcome definitions and explicit uncertainty before interpreting numerical claims of improvement. The companion denominator-neglect guide helps readers keep counts and rates straight.

Primary methodology: Cole and Hernán, Constructing Inverse Probability Weights for Marginal Structural Models; Hernán and Robins, Causal Inference: What If; PSweight package methodology for alternative estimands; and CRAN causal-inference vignette with weighting examples.

For a broader route, visit How AI Works and The Gold Standard of Education. A responsible answer explains what the data can support and what it cannot.

Discover more from eduKate Singapore

Subscribe now to keep reading and get access to the full archive.

Continue reading