How Off-Policy Evaluation in Education Works | Test a New Teaching Policy With Old Logged Decisions Without Inventing Missing Outcomes

eduKateSG Learning Node Series · 0282

A school has two years of adaptive-learning logs. A new policy looks promising. Must thousands of students experience it before anyone can estimate whether it is better?

Not always. Off-policy evaluation, often shortened to OPE, tries to estimate the value of a new decision policy using data collected under a different policy.

The idea is powerful because educational experimentation is costly. Learner time matters. Bad actions cannot be tried casually. Historical systems already contain millions of decisions about hints, questions, examples and progression. If those logs can support credible counterfactual evaluation, a team can reject weak policies before live deployment and prioritise the most promising candidates for controlled trials.

But OPE is also easy to misuse. Historical data contain outcomes for actions that were actually chosen. They do not contain outcomes for actions that were not chosen. A new policy may want to act precisely where the old policy rarely did. No estimator can conjure reliable evidence from an empty part of the action–context space.

Off-policy evaluation works by reweighting and modelling logged decisions so we can estimate a new policy’s expected outcome—while making explicit where the old data do and do not support the counterfactual claim.

The 50-Second Read

• The behaviour policy generated the historical data.
• The target policy is the new policy we want to evaluate.
• We observe reward only for the action that the behaviour policy actually chose.
• Propensity scores record or estimate the probability that the behaviour policy selected each logged action.
• Importance weighting gives more weight to observations that are informative about the target policy.
• Inverse-propensity estimates can have very high variance when target and behaviour policies differ strongly.
• Clipping and self-normalisation can stabilise estimates but add bias.
• Doubly robust estimators combine propensity weighting with an outcome model.
• Good OPE requires overlap: the historical policy must have given meaningful probability to actions the target policy wants to choose.
• Unobserved confounding can invalidate causal interpretation.
• Sequential educational decisions are harder than one-step bandit decisions because errors compound over trajectories.
• OPE should screen and compare policies; high-stakes deployment still needs credible prospective validation.

Canonical Owner Boundary

This node owns counterfactual evaluation of a new educational decision policy using logs generated by another policy. How Contextual Bandit Tutoring Works owns online one-step action selection. How POMDP Teaching Works owns sequential decisions under hidden state. OPE asks a different question: before deploying a new policy, what can old logged decisions defensibly tell us about what would have happened under that new policy?

1. The Counterfactual Is Missing by Construction

A learner received Hint A. We observed the next answer. We did not simultaneously observe the same learner receiving Hint B.

The unchosen outcome is counterfactual. Off-policy evaluation is therefore not ordinary prediction. It is an attempt to estimate a policy value under missing potential outcomes created by the old policy’s own decisions.

2. The Behaviour Policy Is Part of the Dataset

Educational logs are not neutral samples. A system chose which problem to show, which hint to reveal, which learner to route, and when to stop.

The policy that made those choices shaped the data. Harder questions may have been given only to stronger students. Intensive hints may have been reserved for learners already struggling. Observed reward differences therefore mix action effects with policy selection.

OPE begins by treating that selection mechanism as first-class information.

3. Propensity Scores Record How Likely the Logged Action Was

If a contextual bandit chose Hint A with probability 0.60, Hint B with 0.30 and Hint C with 0.10, the selected action’s propensity is the probability assigned by the behaviour policy at that decision point.

Propensities matter because a rare action provides different counterfactual information from a common one. If the new policy strongly favours an action the old policy almost never used, the estimator must rely heavily on very few observations.

This is why logging action probabilities at decision time is far better than trying to reconstruct them later.

4. Inverse Propensity Weighting Reconstructs a New Policy From Old Data

One classic estimator weights each logged reward by the ratio between the probability that the target policy would choose the observed action and the probability that the behaviour policy chose it.

If the target policy would choose an action more often than the old policy did, that observation receives more weight. If the target policy would rarely choose it, the observation receives less.

Conceptually, the weighting tries to transform data generated under one action distribution into an estimate for another.

5. Importance Weights Can Explode

Suppose the old policy gave an action probability 0.001 while the new policy would choose it with probability 0.5. The importance ratio becomes enormous.

One rare observation can dominate the estimate. The mathematical estimator may remain unbiased under ideal assumptions while being practically unusable because variance is extreme.

Educational teams should inspect weight distributions, not merely report one policy-value number.

6. Overlap Is an Evidence Boundary

OPE requires support or overlap: contexts where the target policy chooses an action should be contexts where the behaviour policy sometimes chose that action too.

If the old system never gave advanced problems to low-proficiency learners, historical logs contain no direct evidence about what would happen if a new policy does so.

No clever estimator removes this fundamental limit. Extrapolation is not observation.

7. Effective Sample Size Can Collapse Long Before Row Count Does

A dataset may contain ten million interactions and still provide weak support for one target policy.

If importance weights concentrate on a tiny fraction of records, the effective sample size can be much smaller than the raw row count suggests. This is one reason “big data” does not automatically solve counterfactual evaluation.

8. Clipping Trades Variance for Bias

One practical strategy caps extremely large importance weights. This stabilises variance because no single observation can dominate without limit.

The cost is bias. The estimator no longer perfectly represents the mathematical target policy. Clipping should therefore be reported as a deliberate bias–variance trade-off, not hidden inside preprocessing.

9. Self-Normalised Importance Sampling Stabilises the Scale

Self-normalised estimators divide weighted rewards by the sum of weights. They often have lower variance and better finite-sample behaviour, especially when propensities vary strongly.

Again, stability comes with altered statistical properties. Different estimators answer the same policy-value question with different bias and variance profiles.

10. Direct Outcome Models Take a Different Route

Instead of reweighting logged rewards, a model can predict expected reward for each action under each context, then simulate the target policy’s choices through that model.

This can be efficient when the outcome model is accurate. It can also be badly wrong if the model extrapolates beyond data support or misses important interactions.

11. Doubly Robust Estimation Combines Two Imperfect Routes

Doubly robust estimators combine a propensity-based correction with an outcome model. Under standard assumptions, the estimate can remain consistent if either the propensity model or the outcome model is correctly specified.

This is attractive in education because both models are hard: the behaviour policy may be complicated and the learner-response model may be imperfect.

“Doubly robust” does not mean invulnerable. If both components are wrong, or if overlap is poor, the estimate can still fail.

12. Logged Deterministic Policies Are Hard to Evaluate Beyond Themselves

If the old tutor always gave exactly one action in each context, there may be no variation from which to estimate alternatives.

Randomised exploration, even when conservative, creates future scientific value because it establishes support for policy comparison. A production system that optimises only immediate performance can accidentally destroy the evidence needed to improve itself later.

13. Unobserved Confounding Is a Serious Threat

Suppose teachers manually overrode the system and gave intensive hints when they noticed confusion that was never logged. The dataset records hint action and weak outcome, but not the teacher-observed confusion that influenced both.

Standard propensity correction based only on logged variables cannot remove confounding from an important unobserved variable.

The strongest OPE designs therefore invest in decision logging: what information was available when the action was chosen, not merely what happened afterward.

14. Off-Policy Evaluation Is Easier for One-Step Decisions Than Long Sequences

In contextual bandits, the action affects immediate reward but typically does not explicitly change future context through a modelled transition.

In sequential reinforcement learning, a policy changes the trajectory. A different hint today can alter what problem appears tomorrow, which changes later states and later rewards. Importance ratios multiply across steps and variance can grow dramatically with horizon.

This is one reason sequential OPE remains technically demanding.

15. The Curse of Horizon Is a Distribution Problem

As trajectories become longer, the probability that the behaviour policy happened to take a sequence resembling the target policy can become tiny.

Modern OPE research develops methods that estimate density ratios or value functions without naively multiplying long chains of action probabilities. For example, 2025 work in AISTATS studied spectral representations for primal–dual off-policy evaluation to improve tractability of distribution-correction estimation.

Source: Primal-Dual Spectral Representation for Off-policy Evaluation.

16. Offline Policy Learning and Off-Policy Evaluation Are Neighbours

OPE asks: “How valuable is this target policy?” Offline policy learning asks: “Can we learn a better target policy from the same logged data?”

The second problem is harder because the optimiser can exploit weaknesses in the evaluator. If an estimator is overoptimistic in poorly supported regions, policy learning may deliberately steer toward those regions.

Conservative or pessimistic methods counter this by penalising actions where evidence is weak.

17. Current 2026 Educational Work Brings These Ideas Into Tutoring

A 2026 paper on counterfactual learning of adaptive instructional policies uses logged student–item data to learn new policies in an offline contextual-bandit framework. It explicitly estimates round-specific behaviour policies and uses them both as propensity models for OPE and as diagnostics of how adaptive the original system really was.

The work is important because it connects abstract OPE machinery with a practical tutoring question: can we improve an instructional policy without collecting another round of learner data first?

Source: Counterfactual Learning of New Adaptive Instructional Policies Using Logged Data.

18. The Behaviour Policy Can Itself Be a Research Object

Estimating how the old system chose actions can reveal whether it was truly adaptive.

If action probabilities barely change with learner context, a supposedly adaptive platform may have functioned mostly as a static recommender. If one action dominated nearly everywhere, the logs may offer little support for learning a diverse new policy.

19. Reward Logging Must Match the Policy Objective

If the target policy is supposed to optimise delayed retention but historical logs contain only immediate correctness, OPE cannot directly estimate the desired objective.

Teams often discover too late that their telemetry captured clicks and scores but not the long-term outcomes needed for policy comparison. Data architecture should therefore be designed backward from future decision questions.

20. Cross-Domain Comparison: Observational Medicine

Hospitals often want to compare treatments using records generated by clinicians who selected treatment based on patient condition. That selection creates confounding. Propensity methods and doubly robust estimators are used because the treatment assignment was not random.

Educational policy logs have the same structure: the old system selected actions based on learner context, so naive average outcomes can mislead.

21. Cross-Domain Comparison: Backtesting a Trading Strategy

A new trading rule can look brilliant in historical data if it exploits information that was not actually available at the decision time or if it trades assets under conditions the historical strategy never faced.

Educational OPE has analogous leakage risks. Context must represent information available when the action was chosen. Future outcomes cannot quietly enter the policy model.

22. Missing-Node Scan

The missing node may be off-policy evaluation when a team wants to deploy a new adaptive policy solely because a predictive model scores well; when logged action probabilities were never stored; when the target policy chooses actions the behaviour policy almost never tried; when importance weights are highly concentrated but only the point estimate is reported; when a reward model extrapolates far outside data support; when teacher overrides created unlogged confounding; when immediate outcomes are used to evaluate a policy whose stated objective is long-term learning; or when offline optimisation repeatedly searches until it finds a policy that exploits evaluator error.

23. Failure Mode: Treating Millions of Rows as Millions of Useful Counterfactuals

The raw dataset is enormous, but overlap for the target policy is poor.

Repair: report effective sample size, weight concentration and coverage by context–action region.

24. Failure Mode: Reconstructing Propensities From a Black Box

The old production system is gone. Engineers estimate what it “probably” did from partial logs.

Repair: preserve policy version, action probabilities, eligible action set and context snapshot at decision time in future systems.

25. Failure Mode: The New Policy Leaves the Support of the Old Policy

The target policy strongly prefers actions almost never observed historically.

Repair: constrain the new policy toward supported regions, collect targeted exploration data, or run a small controlled prospective experiment rather than pretending the logs answer the question.

26. A Practical Educational OPE Workflow

1. Define the target policy and educational reward precisely.
2. Identify the behaviour policy that generated each log segment.
3. Recover or validate action propensities.
4. Confirm that context uses only information available at decision time.
5. Inspect overlap between behaviour and target policies.
6. Compute weight distributions and effective sample size.
7. Use multiple estimators where possible: direct, IPS, self-normalised and doubly robust.
8. Run sensitivity analyses for clipping and model specification.
9. Audit subgroup coverage and policy value.
10. Reject policies that rely on unsupported extrapolation.
11. Use OPE to shortlist policies, not to eliminate prospective testing.
12. Validate high-impact policy changes with randomised or otherwise causally credible deployment.

27. Evidence and Current Research

OPE is a mature topic in reinforcement learning and causal inference, but education adds special constraints: learner opportunity cost, nonstationary learning states, delayed outcomes, policy-driven curriculum exposure and ethically limited exploration.

Current work is increasingly connecting generic OPE machinery with instructional policy learning. The 2026 counterfactual instructional-policy paper demonstrates an explicit behaviour-policy and off-policy-evaluation pipeline across four real-world datasets. Large-scale tutoring work from 2025 similarly uses offline policy evaluation before live deployment of bandit-based feedback policies.

For the general methodological frontier, 2025 AISTATS work on spectral OPE and broader offline reinforcement-learning research continue to address high variance, poor coverage and distribution shift. These technical advances matter educationally because the core limitation is the same: a new policy asks questions that historical behaviour may not have answered.

28. The Return Path

Return to the two years of adaptive-learning logs.

They are valuable, but not because they contain every answer. They contain a record of what one historical policy chose to do and what happened afterward.

Off-policy evaluation asks how far that evidence can travel toward a different policy without pretending that unobserved outcomes were observed.

The mark of a strong off-policy estimate is not confidence. It is a clear boundary between what the logs support, what the model extrapolates, and what still needs a real experiment.

Research and Further Reading

• Counterfactual Learning of New Adaptive Instructional Policies Using Logged Data
• Learning to Optimize Feedback for One Million Students
• Primal-Dual Spectral Representation for Off-policy Evaluation
• Reinforcement Learning in Education: A Systematic Literature Review

eduKateSG Learning Node Series · 0282 · Previous: 0281 — How Contextual Bandit Tutoring Works.