Choose your reading route
eduKateSG · Why Mathematics? · Deming Regression, Errors in Both Variables and Calibration Lines
Start with the question, audit the worked example, or jump to the assumptions and practice. Every chapter stays open for uninterrupted reading.
Understand the question
Define the quantity and follow the mechanism.Route 2 · Chapters 4–7
Reproduce the example
Audit the numbers one step at a time.Route 3 · Chapters 8–13
Check assumptions
See where design and calibration matter.Route 4 · Chapters 14–18
Explore applications
Connect the method to a concrete investigation.Route 5 · Chapters 19–25
Study and verify
Practise, inspect failure cases and read further.
Full chapter index · Practice and checking · Mathematics Learning Hub
Why is mathematics important when both variables in a calibration study contain measurement error? Ordinary least squares treats the horizontal variable as fixed or measured without error. Deming regression allows error in both paired methods and uses a declared ratio of their error variances to estimate slope and intercept.
This chapter uses five original pairs: X = 1, 2, 3, 4, 5 and Y = 2, 4, 6, 8, 9. With equal error variances, the Deming slope is about 1.8170 and the intercept about 0.3489. Ordinary least squares gives slope 1.8 and intercept 0.4. The small difference is enough to expose the underlying modelling choice and why calibration needs more than a familiar regression button.
Full chapter index
Chapters 1–3 · Understand the question
Chapters 4–7 · Reproduce the example
Chapters 8–13 · Check assumptions
Chapters 14–18 · Explore applications
Chapters 19–25 · Study and verify
Suppose two instruments measure the same changing quantity. Neither reading is exact. A scatterplot uses one method on X and the other on Y, but that choice does not make X error-free.
Ordinary least squares minimises vertical squared residuals. Deming regression minimises a weighted combination of horizontal and vertical deviations from the fitted line.
This symmetry is useful in method comparison, provided the error-variance ratio and model assumptions are defensible.
The pairs are (1,2), (2,4), (3,6), (4,8) and (5,9). The last Y reading sits one unit below the exact line Y = 2X; the first four sit on it.
The X mean is 3. The Y mean is 29/5 = 5.8.
These are synthetic teaching values. A real calibration study should include the intended operating range, replicates or external precision information, and traceable units.
| Item | X | Y |
|---|---|---|
| 1 | 1 | 2 |
| 2 | 2 | 4 |
| 3 | 3 | 6 |
| 4 | 4 | 8 |
| 5 | 5 | 9 |
Using sums of centred squares without dividing by n, Sxx is 10. The Y deviations from 5.8 give Syy = 164/5 = 32.8.
The centred cross-product Sxy is 18. These three quantities drive both ordinary and Deming slopes.
Centred sums avoid unnecessary intercept algebra. Once slope is found, the fitted line passes through the point of means.
The ordinary least-squares slope of Y on X is Sxy/Sxx = 18/10 = 1.8. The intercept is mean Y − 1.8 mean X = 5.8 − 5.4 = 0.4.
This answer is correct for its model. It minimises vertical residual squares and treats X values as fixed or effectively error-free.
Swapping X and Y and inverting the new slope generally does not return the same line. That directional behaviour matters in symmetric method comparison.
Deming regression uses lambda, commonly defined as the ratio of Y-error variance to X-error variance. Software conventions should be checked because some sources use the reciprocal.
For the worked example, lambda = 1, meaning equal error variances in the two methods on their current units.
This is an assumption, not a value estimated reliably from one pair per item. Replicates, validation data or instrument precision studies can support it.
For positive Sxy, one formula is [Syy − lambda Sxx + sqrt((Syy − lambda Sxx)² + 4 lambda Sxy²)] divided by 2Sxy.
Here Syy − Sxx = 22.8. Substitute Sxy = 18 and lambda = 1. The positive-root result is approximately 1.8170187.
The negative root corresponds to the other mathematical solution and is not the increasing relationship supported by these data.
The Deming line passes through (mean X, mean Y). Intercept equals 5.8 − 1.8170187 × 3, approximately 0.3489438.
The fitted equation is Y = 0.3489 + 1.8170X. If perfect agreement were the goal, the target would be intercept zero and slope one.
Our example is not close to equality because Y is roughly twice X. The regression describes conversion, not interchangeability.
| Method | Slope | Intercept |
|---|---|---|
| OLS Y on X | 1.8000 | 0.4000 |
| Deming, lambda = 1 | 1.8170 | 0.3489 |
OLS gives 0.4 + 1.8X. Deming with lambda one gives about 0.3489 + 1.8170X.
The difference is modest because the points nearly follow a straight line and the sample is tiny. Other data or variance ratios can yield a larger gap.
The lesson is not that Deming always changes results dramatically. It is that its objective function matches error in both axes.
Did you know that equal-variance Deming regression has a useful symmetry under swapping methods, with suitable transformation of the fitted line? Ordinary Y-on-X least squares lacks that symmetry.
Method comparison often has no naturally error-free axis. A symmetric model can therefore reflect the scientific question better.
Symmetry does not make the method assumption-free. Lambda, linearity, error independence and sampling still matter.
With the same centred sums, lambda 0.25 gives slope about 1.82065; lambda 4 gives about 1.80998. The fitted intercept changes accordingly.
In this example the sensitivity is small. A different dataset can be more sensitive. Report the chosen lambda and repeat the fit for plausible alternatives when it is uncertain.
Do not choose lambda to force slope one. That reverses the logic of measurement modelling.
If each method has replicate readings, within-item variance can help estimate precision. External quality-control data may also inform lambda.
The ratio must match units. Changing centimetres to millimetres scales variance by 100, so lambda changes unless both methods are transformed consistently.
Document how precision was estimated, on which range and under what conditions.
An intercept different from zero suggests constant offset within the linear model. A slope different from one suggests proportional difference.
Confidence intervals around both parameters are needed before making population claims. Even then, statistical compatibility with zero and one does not prove practical equivalence.
Evaluate bias at decision-relevant concentrations, not only parameter tests.
If the purpose is to predict Y from a future observed X, ordinary regression or another predictive model may be appropriate even when X has noise.
Deming regression targets the underlying linear relation under an errors-in-variables model. Prediction loss and calibration equivalence are different objectives.
State whether the line is for conversion, agreement assessment, scientific relation or prediction.
OLS uses vertical distances. Equal-variance Deming is related to perpendicular-distance fitting after appropriate scaling. With unequal lambda, axes are scaled by their error variances.
This geometric view helps students understand why slopes differ. The fitted line rotates because the notion of distance changes.
A plot with horizontal and vertical error bars can make the model tangible.
Deming regression still assumes a linear relationship. Curvature, thresholds or saturation can make one slope misleading.
Inspect residuals across the range and include enough data near important boundaries. A nonlinear calibration model may be needed.
Do not extrapolate beyond the observed range without scientific justification.
Squared-distance methods can be sensitive to outliers. Our last point changes the line noticeably despite a small one-unit departure.
Investigate data provenance before exclusion. Robust alternatives such as Passing–Bablok address different assumptions but are not automatic replacements.
Show sensitivity when one pair drives the conclusion.
Five pairs are enough for exact arithmetic, not a reliable calibration. Standard errors, confidence intervals or bootstrap procedures need adequate data and a suitable design.
Replicates from the same item do not create independent coverage of the range. Cluster structure must be respected.
Precision claims should match the number of independent items and the uncertainty in lambda.
Define the two methods, units, operating range and decision thresholds. Collect paired measurements, randomise order where possible and obtain precision evidence.
Plot the data with the identity line. Fit Deming regression with documented lambda, inspect residuals and calculate uncertainty for slope, intercept and bias at key values.
Use new data to validate any conversion. Pair the regression with difference analysis when interchangeability matters.
Verify means 3 and 5.8, Sxx = 10, Syy = 32.8 and Sxy = 18. Calculate OLS first.
Then substitute into the Deming formula with lambda one. Use enough decimal precision to reproduce slope 1.8170 and intercept 0.3489.
Finally swap units for Y by multiplying every Y value by ten. Explain how the coefficient and lambda must transform to preserve the same physical relationship.
- Deming regression does not make both methods accurate.
- Lambda is not an optional decoration.
- Slope near one does not alone prove agreement.
- A nonsignificant parameter test does not establish equivalence.
- High correlation does not validate conversion.
- Regression does not replace a difference plot.
The model is valuable because its assumptions are explicit, not because it removes judgement.
Using the five pairs:
- reproduce all centred sums;
- calculate OLS slope 1.8 and intercept 0.4;
- calculate equal-variance Deming slope about 1.8170;
- calculate intercept about 0.3489;
- compare predictions at X = 5; and
- describe why identity slope one is not plausible here.
Then repeat with lambda 0.25 and 4. Explain the direction and practical size of the change.
Students can explore this with two rulers or timing methods that both contain reading error. Draw the fitted line and the equality line separately.
Parents can ask: which variable is assumed exact, and is that realistic? That question often reveals why a familiar formula may not fit the task.
The activity builds algebra, coordinate geometry, statistics and scientific problem-solving with a direct link to engineering and laboratory careers.
What does lambda equal one mean?
It means the two error variances are assumed equal on the current measurement scales.
Is Deming regression the same as orthogonal regression?
Equal-variance Deming is closely related to perpendicular-distance fitting after appropriate scaling. General Deming uses the declared variance ratio.
Can it prove methods are equivalent?
No. Practical tolerance, uncertainty, range and difference behaviour must also be evaluated.
Why not use ordinary least squares?
OLS can be right for prediction or when X is effectively error-free. Deming addresses paired method comparison with error in both variables.
The NIST Deming regression documentation states the errors-in-variables model, variance ratio, fitted parameters and uncertainty outputs. The CLSI EP09 guideline page describes method-comparison bias estimation and precision-profile information for Deming regression.
Continue with the Mathematics Learning Hub and the series article on weighted kappa and declared error costs for another example of a result governed by an explicit loss model.
Verify pairing, units, range, centred sums, lambda definition, precision evidence, linearity, residuals, uncertainty and decision tolerance. Keep conversion, agreement and prediction claims separate.
The importance of mathematics is that it makes error visible on both axes and forces a calibration line to reflect how measurements are actually produced.
Extended practice laboratory
Practice case 1: Equal precision
Setup: Set lambda to one. Task: State the assumption.
Check: X and Y error variances are equal on current units. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 2: Reciprocal convention
Setup: Software defines lambda oppositely. Task: Prevent error.
Check: Check documentation and transform the ratio before comparing slopes. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 3: Error-free X
Setup: X uncertainty is negligible. Task: Compare models.
Check: Ordinary Y-on-X regression may match the scientific objective. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 4: Both noisy
Setup: Both instruments vary. Task: Choose geometry.
Check: Deming accounts for deviations in both coordinates under its error model. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 5: Unit change
Setup: Multiply Y by ten. Task: Update lambda.
Check: Y-error variance and its ratio scale by one hundred. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 6: Replicates
Setup: Collect duplicate readings. Task: Estimate precision.
Check: Within-item variation can inform the variance ratio. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 7: Unknown lambda
Setup: No precision data exist. Task: Run sensitivity.
Check: Show fits across plausible ratios rather than hiding an arbitrary choice. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 8: Identity test
Setup: Slope is near one. Task: Inspect intercept.
Check: Proportional agreement does not remove constant bias. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 9: Zero intercept
Setup: Intercept is near zero. Task: Inspect slope.
Check: Constant agreement does not remove proportional bias. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 10: Curvature
Setup: Residuals bend across the range. Task: Reject simplification.
Check: One linear calibration is inadequate without a defensible transformation. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 11: Outlier
Setup: One extreme pair rotates the line. Task: Audit influence.
Check: Investigate source and show sensitivity before exclusion. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 12: Prediction
Setup: The goal is future Y from observed X. Task: Clarify objective.
Check: A predictive loss may favour a different model than symmetric comparison. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 13: Conversion
Setup: Use the fitted line to transform X. Task: Validate.
Check: Test conversion on independent pairs and report prediction uncertainty. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 14: Threshold bias
Setup: Decisions occur at X equals four. Task: Calculate local bias.
Check: Evaluate fitted difference and uncertainty at the relevant value. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 15: Heteroscedasticity
Setup: Error grows with magnitude. Task: Adjust the model.
Check: Weighted Deming or transformation may be more suitable. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 16: Correlated errors
Setup: Methods share a preparation step. Task: Question assumptions.
Check: Independent error components may be violated and require a richer model. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 17: Repeated operators
Setup: Several people use each device. Task: Expand design.
Check: Operator effects and clustering belong in uncertainty and generalisation. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 18: Small sample
Setup: Only five pairs are fitted. Task: Limit claims.
Check: The example teaches arithmetic but cannot support precise calibration. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 19: Wide range
Setup: Add valid extremes. Task: Assess leverage.
Check: Range can stabilise slope yet make extreme points influential. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 20: Narrow range
Setup: X values are nearly equal. Task: Predict instability.
Check: Slope becomes sensitive because centred X variation is small. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 21: Axis swap
Setup: Exchange methods. Task: Check symmetry.
Check: A consistent Deming model transforms coherently when lambda is handled correctly. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 22: OLS comparison
Setup: OLS and Deming are close. Task: Interpret fairly.
Check: Similarity in one dataset does not erase their different assumptions. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 23: Tolerance
Setup: Slope interval includes one. Task: Avoid equivalence claim.
Check: Practical bias limits and a prespecified equivalence region are still required. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.
Practice case 24: Audit file
Setup: Reproduce the fit. Task: Preserve essentials.
Check: Save paired data, units, lambda definition, precision source, algorithm and software version. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer.