SG HGW Forecast Backtest Protocol v0.1 — Rolling 4w/12w Validation + Threshold Calibration

TITLE: SG HGW Forecast Backtest Protocol v0.1 — Rolling 4w/12w Validation + Threshold Calibration
VERSION: CivOS Unified Spec v1.x
MODE: Almost-Code / LLM-first / WordPress paste-ready
OWNER: eduKateSG Fence Learning Systems
SCOPE: Singapore EducationOS HGW forecasting (Z0–Z2), scalable to Z3/Z4
GOAL: Verify predictive accuracy, calibrate thresholds, and improve routing rules
      using rolling-window backtests (4-week and 12-week horizons).
CANONICAL CLAIM:
- Prediction is only real if it survives backtesting.
- Education forecasting is not “predict the exact score.”
- It is: predict Phase shifts, collapse risk events, coupling breaks, and dominant failure modes.
- This protocol makes HGW forecasts falsifiable, improvable, and stable.
========================================================
0) WHAT WE BACKTEST (WHAT “ACCURACY” MEANS)
========================================================
We backtest four target predictions:
Target-1: Phase Shift Accuracy
- predicted Phase(t+T) vs observed Phase(t+T)
- T ∈ {4 weeks, 12 weeks}
Target-2: Collapse Risk Event Accuracy
- predicted CollapseRiskScore (Low/Med/High)
- observed collapse events: P2→P1 or P2→P0 under timed load spike
Target-3: Coupling Break Accuracy
- predicted coupling break edges vs observed failure signatures
- e.g., ENG inference→MATH WP; ALG/GRAPH→PHY motion
Target-4: Dominant Failure Mode Accuracy
- predicted FM-x vs observed FM-x
========================================================
1) DATASET DEFINITION (MINIMUM VIABLE BACKTEST SET)
========================================================
DataUnit := WeeklyRecord for one learner
WeeklyRecord includes:
- WeekIndex
- TTC
- HoursPerWeek
- VolatilityIndex
- BN
- Subject mini-test results (Math/Eng/Sci/Hum)
- NIT/NIT-S/NIT-ARG trigger counts
- Timed completion rates
- Samples (1 paragraph + 1 science mechanism line) if available
- Output: observed Phase per subject
- Observed major event flag (exam week / big test)
BacktestSet sizes:
- MVP: 10 learners × 12 weeks
- Strong: 30 learners × 12–24 weeks
- Cohort: 1 class × 1 semester
========================================================
2) STANDARDISED TESTS (TO MAKE DATA COMPARABLE)
========================================================
To reduce noise, standardise:
Math:
- TT1: 10–15 min core set
- TT-M: 15 min mixed set
English:
- RR-10: 10 min retrieval test
- PARA-10: 10 min paragraph (coherence + structure)
Science:
- DR-5: 5 min definition check
- MR-90: 90-second mechanism explain
- GRF-10: 10 min graph/formula mini-set
Humanities:
- CAUSE-10: 10 min causality paragraph
- EVID-5: evidence line check
Phase is assigned from timed transfer behavior using the same rubric each week.
========================================================
3) BACKTEST WINDOWS (ROLLING HORIZONS)
========================================================
Rolling forecasts:
At week t:
- run PROMPT-HGW-A or PROMPT-HGW-B
- store predicted outputs for horizons:
  - T=4 (week t+4)
  - T=12 (week t+12)
Then when week t+4 arrives:
- compare predicted vs observed
Same for week t+12.
This creates many overlapping forecast trials.
========================================================
4) SCORING METRICS (HOW WE GRADE FORECASTS)
========================================================
----------------------------------------
4.1 Phase Shift Score (PSS)
----------------------------------------
Per subject:
- exact match: 1.0
- off by 1 phase: 0.5
- off by 2+: 0.0
PSS_T = average across subjects for horizon T.
----------------------------------------
4.2 Collapse Event Score (CES)
----------------------------------------
Define collapse event:
- P2→P1 or P2→P0 during a load spike week
- OR NIT/NIT-S triggers surge + timed completion crash
CES rules:
- predicted High and collapse occurs: TP
- predicted Low and collapse occurs: FN
- predicted High and no collapse: FP
- predicted Low and no collapse: TN
Compute:
- Precision, Recall, F1 for “High-risk” classification.
----------------------------------------
4.3 Coupling Break Score (CBS)
----------------------------------------
CBS_T = (# correctly predicted coupling breaks among top 3 predictions) / 3
Allow partial credit:
- predicted correct domain but wrong edge: 0.5
----------------------------------------
4.4 Failure Mode Score (FMS)
----------------------------------------
- correct FM-x: 1.0
- adjacent (Speed vs Retrieval; NIT vs NIT-S) confusion: 0.5
- wrong: 0.0
========================================================
5) THRESHOLD CALIBRATION (TUNING THE ENGINE)
========================================================
We calibrate the key thresholds:
Thresholds to tune:
- Emergency Mode trigger:
  - TTC < 8 weeks AND BN > 3 (v0.1 default)
- NIT trigger:
  - NIT_w ≥ 2/week
- Drift trigger:
  - d(TR)/dt negative for 2 consecutive cycles
- Coupling break trigger:
  - downstream signature appears twice within 2 weeks
Calibration method:
- grid search across small discrete options, choose settings that maximize:
  - CES F1 (catch collapses)
  - while keeping FP reasonable (avoid panic)
Practical tuning options:
- TTC threshold: {6, 8, 10}
- BN threshold: {2, 3, 4}
- NIT threshold: {1, 2, 3}
- Drift cycles: {1, 2, 3}
========================================================
6) MODEL IMPROVEMENT RULES (WHAT TO CHANGE WHEN IT FAILS)
========================================================
If Phase predictions are off:
- tighten Phase rubric definition
- increase weight of timed transfer tests
If Collapse events missed (FN high):
- increase sensitivity to:
  - NIT/NIT-S derivatives
  - buffer index and volatility
  - coupling risk
If Too many false alarms (FP high):
- reduce emergency triggers
- require 2 signals (e.g., TTC+BN plus rising NIT)
If Coupling predictions weak:
- add signature detection templates
- add subject-specific coupling variants (e.g., vocab retrieval→writing output)
========================================================
7) BACKTEST REPORT TEMPLATE (WHAT YOU PUBLISH)
========================================================
BacktestReport := {
  Dataset description,
  Time window,
  Standardised tests used,
  PSS_4w, PSS_12w,
  CES (Precision/Recall/F1) for 4w and 12w,
  CBS_4w, CBS_12w,
  FMS_4w, FMS_12w,
  Top failure cases,
  Threshold changes,
  Updated v0.x rules (versioned forward only)
}
========================================================
8) REQUIRED FAILURE TRACE (BACKTEST GOVERNANCE)
========================================================
Trace:
No standardised tests →
no comparable records →
forecasts unfalsifiable →
rules drift →
panic decisions →
hidden fragility.
Repair:
Standardise micro-tests →
rolling-window backtests →
calibrate thresholds →
version forward only →
stable prediction engine.
========================================================
END: SG HGW Forecast Backtest Protocol v0.1
========================================================

WORDPRESS PASTE-READY STRUCTURE
H1: SG HGW Forecast Backtest Protocol — Making Education Prediction Real
H2: What “accuracy” means (Phase, collapse, coupling, failure modes)
H2: Minimum dataset definition
H2: Standardised micro-tests (weekly)
H2: Rolling 4-week and 12-week backtests
H2: Scoring metrics (PSS/CES/CBS/FMS)
H2: Threshold calibration (Emergency mode, NIT, drift)
H2: How to improve the model when it fails
H2: Backtest report template (publishable)
H2: Failure trace + repair trace