Choose your reading route
eduKateSG · Why Mathematics? · PRESS Statistic: Leave-One-Out Prediction Error
Start with the question, audit the worked example, or jump to the assumptions and practice. Every chapter stays open for uninterrupted reading.
Understand the question
Define the quantity and follow the mechanism.Route 2 · Chapters 5–9
Reproduce the example
Audit the numbers one step at a time.Route 3 · Chapters 10–15
Check assumptions
See where design and calibration matter.Route 4 · Chapters 16–20
Explore applications
Connect the method to a concrete investigation.Route 5 · Chapters 21–24
Study and verify
Practise, inspect failure cases and read further.
Full chapter index · Practice and checking · Mathematics Learning Hub
Why is mathematics important when a regression model fits its training data well but may predict new cases poorly? The prediction error sum of squares, usually called PRESS, leaves out each observation in turn, predicts that omitted observation, squares the resulting prediction error and adds the errors.
PRESS converts the idea of honest prediction into a reproducible calculation. A smaller value can indicate better leave-one-out prediction for models fitted to the same response data, but it is not a universal score across unrelated datasets.
Did You Know? For ordinary least squares, every leave-one-out residual can be calculated from the ordinary residual and leverage, so the model need not be refitted n separate times.
Full chapter index
Chapters 1–4 · Understand the question
Chapters 5–9 · Reproduce the example
Chapters 10–15 · Check assumptions
Chapters 16–20 · Explore applications
Chapters 21–24 · Study and verify
Ordinary residuals judge cases that helped fit the model. PRESS residuals judge each case using a model fitted without that case. This makes them closer to an unseen-case test while still using the available sample efficiently.
Penn State defines the i-th PRESS residual as y_i minus the prediction from the model fitted without observation i, and PRESS as the sum of their squares. Read Penn State's regression methods notes.
For each i, fit the model to all rows except i, predict y_i at x_i, calculate d_i = y_i – y-hat_i(i), square d_i and sum over all observations. Every row becomes a validation case exactly once.
| Ordinary residual | Leverage | One minus leverage |
|---|---|---|
| 2.4 | 0.2 | 0.8 |
For a correctly specified ordinary least-squares calculation, d_i = e_i/(1-h_ii), where e_i is the ordinary residual and h_ii is leverage. Stanford's course notes give the corresponding leave-one-out mean squared error formula. Read the Stanford LOOCV shortcut.
| PRESS residual | Squared contribution |
|---|---|
| 3.0 | 9.0 |
Suppose an ordinary residual is e=2.4 and leverage is h=0.20. The PRESS residual is 2.4/(1-0.20)=3.0. Its PRESS contribution is 3.0 squared, or 9.0.
A high-leverage observation has helped pull the fitted surface toward itself. Leaving it out can move its prediction substantially, so dividing by 1-h_ii inflates the ordinary residual.
Penn State gives predicted R-squared as 1-PRESS/SSTO. It compares leave-one-out prediction error with total response variation. It can be negative when prediction is worse than using the response mean benchmark.
Compare candidate models on the same observations, response scale and validation rule. A smaller raw PRESS across different datasets or units has no meaningful interpretation.
Training SSE cannot increase when unnecessary predictors are added to ordinary least squares. PRESS may increase because the extra terms fit sample noise and weaken omitted-case prediction.
Leave-one-out validation reuses one dataset and can have high variability. A genuinely independent test set, temporal validation or site-level validation may better match the intended deployment.
Leaving out one row is inappropriate when rows share a student, patient, classroom, device or time series. Leave out the true independent unit or use blocked and forward-looking validation.
If many models are tried and the smallest PRESS is reported, selection itself can overfit the validation criterion. Preserve a final test set or use nested validation when choices are extensive.
A large PRESS contribution can arise from a valid rare case, a recording error, missing structure or unstable leverage. Investigate the record and model; do not delete it merely to improve the score.
Fit the ordinary least-squares model, save residuals and hat diagonals, calculate each PRESS residual as e/(1-h), square and sum. Cross-check a few cases by explicit deletion and refitting.
Use tested influence or cross-validation functions, preserve the preprocessing pipeline inside every fold and group rows correctly. Data cleaning or standardisation performed before the split can leak information.
PRESS and leave-one-out reasoning appear in engineering calibration, environmental prediction, economics, health research, education analytics and machine learning. The transferable idea is to test each prediction without letting that case teach the model.
Compare linear and quadratic models for a small calibration dataset. Calculate training SSE and PRESS, then explain why the rankings may differ. Inspect which cases contribute most to the PRESS total.
PRESS measures squared prediction error under a specific leave-one-out scheme. It does not measure fairness, causation, probability calibration, operational cost or performance after the data-generating process changes.
PRESS is not a p-value. A lower value does not guarantee future superiority. Predicted R-squared is not the same as adjusted R-squared. Leave-one-row-out is not always the right validation unit.
CHAPTER 20 OF 24 · Explore applications
20. Connection to recent mathematics articles
The regression leverage guide explains h_ii. The DFFITS guide develops a related case-deletion view.
Students should distinguish the ordinary residual from the omitted-case residual and show the denominator. Parents can ask whether validation matches the real prediction task and whether grouped observations were kept together.
- Calculate d when e=-1.8 and h=0.10.
- Explain why h close to one makes prediction fragile.
- Describe when leave-one-school-out is better than leave-one-student-row-out.
### Must the model be fitted n times? Not for the ordinary-least-squares shortcut, though explicit refits are useful checks.
### Can predicted R-squared be negative? Yes.
### Is smallest PRESS always best? It identifies a candidate under the declared scheme, not a guaranteed winner.
Use the Mathematics Learning Hub to connect residuals, leverage and validation. Build a table containing ordinary residual, leverage, PRESS residual and squared contribution for every case.
Extended practice laboratory
Practice case 1: Leave out first case
Setup: Use the article’s PRESS worked example as the starting record for case 1. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 2: Predict omitted case
Setup: Use the article’s PRESS worked example as the starting record for case 2. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 3: Calculate PRESS residual
Setup: Use the article’s PRESS worked example as the starting record for case 3. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 4: Square one contribution
Setup: Use the article’s PRESS worked example as the starting record for case 4. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 5: Sum four contributions
Setup: Use the article’s PRESS worked example as the starting record for case 5. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 6: Use leverage shortcut
Setup: Use the article’s PRESS worked example as the starting record for case 6. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 7: Increase leverage
Setup: Use the article’s PRESS worked example as the starting record for case 7. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 8: Reverse residual sign
Setup: Use the article’s PRESS worked example as the starting record for case 8. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 9: Compare training residual
Setup: Use the article’s PRESS worked example as the starting record for case 9. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 10: Calculate predicted R squared
Setup: Use the article’s PRESS worked example as the starting record for case 10. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 11: Create negative predicted R squared
Setup: Use the article’s PRESS worked example as the starting record for case 11. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 12: Compare linear model
Setup: Use the article’s PRESS worked example as the starting record for case 12. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 13: Compare quadratic model
Setup: Use the article’s PRESS worked example as the starting record for case 13. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 14: Inspect largest contribution
Setup: Use the article’s PRESS worked example as the starting record for case 14. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 15: Check data error
Setup: Use the article’s PRESS worked example as the starting record for case 15. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 16: Use grouped folds
Setup: Use the article’s PRESS worked example as the starting record for case 16. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 17: Use temporal folds
Setup: Use the article’s PRESS worked example as the starting record for case 17. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 18: Prevent preprocessing leakage
Setup: Use the article’s PRESS worked example as the starting record for case 18. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 19: Compare independent test set
Setup: Use the article’s PRESS worked example as the starting record for case 19. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 20: Repeat with robust loss
Setup: Use the article’s PRESS worked example as the starting record for case 20. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 21: Audit row alignment
Setup: Use the article’s PRESS worked example as the starting record for case 21. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 22: Check software output
Setup: Use the article’s PRESS worked example as the starting record for case 22. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 23: Avoid test-set tuning
Setup: Use the article’s PRESS worked example as the starting record for case 23. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 24: State validation unit
Setup: Use the article’s PRESS worked example as the starting record for case 24. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.
Practice case 25: Write a prediction conclusion
Setup: Use the article’s PRESS worked example as the starting record for case 25. Task: Change the input named in this case, recalculate every dependent quantity and state the comparison in one precise sentence.
Check: Check the defining identity, preserve the declared model and explain what additional graph, diagnostic or validation evidence is needed. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.