Choose your reading route
eduKateSG · Why Mathematics? · Coverage Probability, Fixed Tolerances and Agreement Rates
Start with the question, audit the worked example, or jump to the assumptions and practice. Every chapter stays open for uninterrupted reading.
Understand the question
Define the quantity and follow the mechanism.Route 2 · Chapters 4–7
Reproduce the example
Audit the numbers one step at a time.Route 3 · Chapters 8–13
Check assumptions
See where design and calibration matter.Route 4 · Chapters 14–18
Explore applications
Connect the method to a concrete investigation.Route 5 · Chapters 19–24
Study and verify
Practise, inspect failure cases and read further.
Full chapter index · Practice and checking · Mathematics Learning Hub
Why is mathematics important when a method must stay within a fixed tolerance? Coverage probability asks what proportion of paired differences fall inside an acceptable interval. It connects measurements directly to a decision boundary chosen in the original units.
This chapter uses twelve original differences: −3, −2, −1, 0, 1, 2, 2, 3, 4, 5, −4 and 1. With an allowed absolute difference of three units, nine of twelve pairs are covered, so empirical coverage is 0.75. The calculation is transparent, but a population claim also needs uncertainty, sampling quality and a probability requirement selected before the data are seen.
Full chapter index
Chapters 1–3 · Understand the question
Chapters 4–7 · Reproduce the example
Chapters 8–13 · Check assumptions
Chapters 14–18 · Explore applications
Chapters 19–24 · Study and verify
Suppose two methods are considered practically close when Y − X lies from −3 to +3 units, including the endpoints. The tolerance is therefore δ = 3.
Coverage probability is P(|Y − X| ≤ 3). The mathematical question is how often a paired deviation lands inside the approved band.
The band must come from consequences, standards or a justified design choice. Choosing it after seeing results makes the test circular.
The synthetic differences are −3, −2, −1, 0, 1, 2, 2, 3, 4, 5, −4 and 1.
Nine values lie between −3 and +3 inclusive. Three values, 4, 5 and −4, lie outside.
The example uses complete paired data in common units. Real studies must verify both conditions.
| Difference | −3 | −2 | −1 | 0 | 1 | 2 | 2 | 3 | 4 | 5 | −4 | 1 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Covered? | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes |
Create an indicator equal to one when |difference| ≤ 3 and zero otherwise. Add the indicators to obtain nine.
Divide by twelve: CP_3 = 9/12 = 3/4 = 0.75.
This is the observed proportion inside the band. It does not guarantee that exactly 75% of future pairs will be covered.
Our definition uses ≤, so differences −3 and +3 count as covered. Using a strict inequality would exclude them and reduce coverage.
For continuous measurements the distinction may have probability zero in theory, but rounded or discrete data can place many observations exactly on the boundary.
State the rule, measurement resolution and rounding procedure.
| Tolerance | Covered | Total | Empirical CP |
|---|---|---|---|
| ±2 | 7 | 12 | 7/12 |
| ±3 | 9 | 12 | 3/4 |
| ±4 | 11 | 12 | 11/12 |
A project might require at least 90% coverage within ±3. Our point estimate of 75% misses that target.
Another application might require 70%, in which case the point estimate passes. Uncertainty still matters.
Both the tolerance and minimum probability belong in the agreement criterion.
Did you know that coverage probability and TDI reverse which quantity is fixed? CP fixes ±δ and estimates a proportion. TDI fixes p and estimates a boundary.
Our CP at δ = 3 is 0.75. A TDI question might ask which boundary contains 75% or 90% of absolute deviations.
The two views can be reported together without treating them as independent evidence.
If paired items are independent and the pass/fail band is fixed, the count inside resembles a binomial outcome. With only twelve items, 9/12 has wide uncertainty.
Use a Wilson, exact or model-based interval appropriate to the design. Do not quote a normal approximation mechanically when sample size is small.
An interval shows that the population coverage may be materially different from 0.75.
Several readings from the same item, student or device share conditions. Their coverage indicators are correlated.
Treating them as twelve independent trials can make uncertainty too narrow. Use cluster resampling or a hierarchical model.
The point proportion remains descriptive, but its inference must match the sampling unit.
An absolute band of ±3 uses the measurement unit. A relative criterion might allow ±5% or a ratio range.
Relative differences require a stable denominator and careful handling near zero. Ratio criteria usually require positive values.
The tolerance scale should reflect practical consequences across the operating range.
Over-reading may be more harmful than under-reading, or vice versa. The acceptable interval need not be symmetric.
Coverage can be calculated for −2 ≤ difference ≤ 4, for example. Report both endpoints and the direction convention.
Do not force symmetry merely because |D| is convenient.
A constant shift moves the whole difference distribution toward one boundary. Even small spread can yield poor coverage if bias is large.
Wide random variation can also reduce coverage with mean near zero. CP alone does not distinguish these causes.
Show signed differences, bias and spread beside the coverage result.
Overall 75% coverage can hide strong middle-range performance and poor high-range performance.
Plot differences against pair mean and compute prespecified strata when the tolerance has different implications across the range.
Avoid post-hoc slicing that searches for a favourable subgroup.
If a method fails to return values on difficult items, those pairs disappear from a naive calculation. Coverage may look too high.
Report the denominator of all eligible items and the reasons for missingness. A failure may itself count as unacceptable under the decision rule.
Never impute a convenient in-band difference without a justified model.
With repeated readings, decide whether the criterion applies to an individual reading, an item mean or a future operator-device combination.
Averaging replicates usually reduces variability and changes the estimand. State the operational use.
The coverage number must match the unit on which decisions will be made.
Under a normal difference model with mean μ and standard deviation σ, coverage inside ±δ can be computed from normal cumulative probabilities.
That model can smooth a small sample but depends on distribution, stable variance and estimated parameters.
Compare model-based and empirical coverage when assumptions are uncertain, and show the difference plot.
The ±3 band is an acceptance region for individual differences. A confidence interval describes uncertainty about a parameter such as coverage.
They answer different questions. A narrow confidence interval around poor coverage does not make the methods acceptable.
Likewise, a wide interval cannot be repaired by widening the tolerance after the analysis.
Define difference direction, units, acceptable band and required probability. Collect representative paired data.
Count covered pairs, calculate CP and an interval using the correct sampling unit. Inspect bias, spread, range patterns and missingness.
Conclude against the predeclared two-part criterion: boundary and probability.
Draw a number line from −5 to +5 and mark all twelve differences. Shade the band from −3 to +3.
Count nine covered values and divide by twelve. Then recompute for δ = 2 and δ = 4.
Coverage must rise or stay equal as a symmetric band widens. If it falls, the calculation is wrong.
- Coverage probability is not the same as confidence level.
- The tolerance should not be chosen after seeing the data.
- Overall CP can hide range-specific failure.
- Boundary inclusion matters for rounded data.
- Missing hard cases can inflate coverage.
- A high CP does not prove validity.
The safest report names δ, the required probability, the observed numerator and denominator, and uncertainty.
Using the twelve differences:
- verify CP_3 = 9/12;
- calculate CP_2 = 7/12;
- calculate CP_4 = 11/12;
- list the three failures at δ = 3;
- reverse subtraction and confirm symmetric CP is unchanged; and
- propose an asymmetric band where reversal would matter.
Then add two repeated observations from the same item. Explain why the descriptive fraction changes while independence has not increased by two full units.
Parents can illustrate coverage with two clocks and a rule that readings within two minutes are acceptable. Count how often the rule is met.
Students should separate the allowed error from the success rate. A wide band can create high coverage without good precision.
This simple structure builds percentages, inequalities, probability and responsible data interpretation.
Is 75% coverage good?
Only relative to a justified tolerance and required probability. The number has no universal pass label.
Can the acceptable interval be asymmetric?
Yes. Use endpoints that match consequences and state the subtraction direction.
Why add an interval?
The sample proportion varies. An interval shows uncertainty about population coverage.
Is CP related to TDI?
Yes. One fixes the boundary and estimates probability; the other fixes probability and estimates a boundary.
The probability-criterion framework is discussed in Choudhary and Nagaraja, Tests for Assessment of Agreement Using Probability Criteria. The author resource on agreement statistics defines coverage probability for absolute and percentage tolerances.
Continue with Bland–Altman limits and the Mathematics Learning Hub.
Verify the paired denominator, difference direction, units, boundary inclusion, tolerance origin, required probability, missingness, clustering, range and interval method. Report bias and spread so the cause of failures remains visible.
The importance of mathematics is that “often close enough” becomes a direct fraction tied to a real tolerance and an honest measure of uncertainty.
Extended practice laboratory
Practice case 1: Tolerance two
Setup: Use the worked data and delta two. Task: Count covered pairs.
Check: Seven of twelve are inside. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 2: Tolerance three
Setup: Use inclusive endpoints. Task: Count again.
Check: Nine of twelve are inside. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 3: Tolerance four
Setup: Widen the band. Task: Check monotonicity.
Check: Eleven of twelve are inside. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 4: Strict boundary
Setup: Exclude values exactly three. Task: Recalculate.
Check: Rounded boundary observations no longer count. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 5: Reverse subtraction
Setup: Change D to minus D. Task: Track symmetric CP.
Check: Coverage inside plus or minus three is unchanged. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 6: Asymmetric band
Setup: Allow minus two to plus four. Task: Count directly.
Check: Direction now matters. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 7: Required 90%
Setup: Compare 0.75 with target. Task: Decide the point estimate.
Check: It misses before interval uncertainty. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 8: Required 70%
Setup: Compare again. Task: Interpret cautiously.
Check: The point estimate passes but sampling uncertainty remains. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 9: One more success
Setup: Add an independent in-band pair. Task: Update numerator and denominator.
Check: Coverage becomes ten of thirteen. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 10: One more failure
Setup: Add a large deviation. Task: Update correctly.
Check: Coverage becomes nine of thirteen. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 11: Clustered repeats
Setup: Add two readings from one item. Task: Assess independence.
Check: The descriptive fraction changes without two independent units. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 12: Common shift
Setup: Add two to all differences. Task: Recount.
Check: Systematic bias can move many pairs outside one edge. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 13: Wider spread
Setup: Double every difference. Task: Recount.
Check: Fixed tolerance coverage generally falls. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 14: Unit conversion
Setup: Scale values and tolerance together. Task: Preserve meaning.
Check: Coverage is invariant when both use the same conversion. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 15: Missing failure
Setup: Remove the difference five. Task: Assess selection.
Check: Observed coverage inflates. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 16: Device failure
Setup: No value is returned. Task: Define the rule.
Check: Operational failure may count as unacceptable rather than disappear. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 17: Near zero
Setup: Use percentage tolerances. Task: Check denominator.
Check: Relative differences can explode near zero. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 18: Range pattern
Setup: High pair means fail more often. Task: Improve analysis.
Check: Report prespecified range-specific coverage. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 19: Normal model
Setup: Estimate μ and σ. Task: Compute model CP.
Check: Model-based coverage requires a justified difference distribution. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 20: Empirical model
Setup: Count indicators only. Task: Name the strength.
Check: The numerator and denominator remain fully auditable. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 21: Confidence interval
Setup: Nine of twelve are covered. Task: Add uncertainty.
Check: A small binomial count has a wide interval. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 22: Tolerance versus CI
Setup: Confuse plus or minus three with 95%. Task: Correct the concepts.
Check: One is an individual acceptance band; the other measures parameter uncertainty. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 23: Validity
Setup: Coverage is high. Task: Limit the claim.
Check: Inter-method closeness does not establish accuracy. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.
Practice case 24: Audit file
Setup: Another analyst repeats CP. Task: Save essentials.
Check: Provide pairs, direction, endpoints, inclusion rule, clusters and interval method. Rebuild the relevant counts before interpreting the coefficient, preserve the stated design, and write one sentence separating the numerical result from the claim it can support. Then vary one defensible assumption, calculate again, and explain whether the conclusion changes because of new data, a new reference model, or a new decision purpose. Finish by naming the unit, tolerance, and evidence needed for transfer. Record one graph or table that would expose failure hidden by the summary.