How Science Works | How Scientists Test Whether Two Results Are Similar Enough

Imagine comparing a new image-based line-measurement method with a reference method for ordinary classroom posters. For each independently selected printed sheet, both methods measure the same line. The difference is new-method length minus reference-method length, in millimetres.

The target is the population mean paired difference, called μ. Positive values mean the new method reads longer on average. Negative values mean it reads shorter. We are not comparing two unrelated collections of lines: pairing allows each sheet to serve as its own comparison.

Suppose the intended application has a documented requirement that systematic average differences smaller than 2 millimetres in either direction are acceptable. The equivalence region is therefore −2 < μ < +2 millimetres. This is a hypothetical design requirement supplied before the results, not a universal threshold for measurement science.

Even a successful test would establish only the stated mean-bias conclusion for the sampled conditions. It would not prove that every individual reading differs by less than 2 millimetres. Nor would it establish that both methods give the true length, because two biased methods can agree with each other.

That distinction should appear before calculations begin. Otherwise a narrow finding about average differences can quietly become a much broader claim that the methods are interchangeable, accurate or suitable for every use.

Contents · Next chapter

An equivalence margin expresses a scientific or practical decision. It should come from the intended use, consequences of discrepancies, measurement capability and relevant external evidence. It is not chosen merely because a familiar statistical convention labels some standardised effect “small.”

For our poster example, imagine that the acceptable mean discrepancy was set during design review, using the intended reading distance and layout requirements. The study then asks whether the new method meets that previously specified mean-bias requirement. A precision-engineering task could need a radically smaller margin.

The margin and the variability answer different questions. The margin says how much mean difference matters. The standard deviation describes how variable the observed paired differences are. A noisy method does not deserve a wider acceptable-bias range simply because otherwise it would be difficult to pass.

Symmetric margins are convenient but not compulsory. If reading too long and reading too short have different consequences, justified lower and upper bounds can differ. Their units and direction must remain clear throughout the analysis.

If no defensible margin is available, scientists can still report an estimate and interval. They can also show how conclusions vary across explicitly exploratory margins. What they should avoid is discovering a convenient threshold after seeing the data and presenting it as an independent, prospectively specified requirement.

Contents · Previous chapter · Next chapter

A conventional two-sided difference test starts with μ = 0 and asks whether the data provide evidence against exact zero. The equivalence test instead starts with the possibility that the mean difference is too far below or too far above zero: μ ≤ −2 or μ ≥ +2.

To support equivalence, both of those possibilities must be rejected. The first one-sided test asks whether μ is greater than −2. The second asks whether μ is less than +2. Passing only one side does not establish that the difference lies between the bounds.

This is the two one-sided tests procedure, usually abbreviated TOST. We use a significance level of 0.05 for each one-sided component. The overall equivalence claim requires both components to pass; it is not made whenever either one passes.

Why is this not an automatic 10 percent false-equivalence rate? If the true mean lies outside the equivalence region, at least one component null is true. Declaring equivalence requires rejecting that true component as well as the other one. Under the specified valid testing model, the probability is therefore bounded by 0.05.

That argument concerns one prespecified equivalence claim. Testing many outcomes, subgroups or alternative margins and selecting a favourable result introduces additional multiplicity and selection issues. The two internal components of TOST should not be confused with a whole collection of opportunities to make separate claims.

Contents · Previous chapter · Next chapter

Take a fictional sample summary for 25 independent sheets. The mean paired difference is 0.50 millimetres and the sample standard deviation of paired differences is 2.50 millimetres. Assume the differences are independent and normally distributed, so the ordinary paired t model applies.

The standard error of the mean is 2.50 / √25 = 0.50 millimetres. There are 24 degrees of freedom. The one-sided 5 percent critical t value is approximately 1.7109.

For the lower boundary, calculate t_lower = [0.50 − (−2)] / 0.50 = 5. Its one-sided upper-tail p-value is approximately 0.0000208. This rejects the claim that the mean is at or below −2 under the model.

For the upper boundary, calculate t_upper = (0.50 − 2) / 0.50 = −3. Its one-sided lower-tail p-value is approximately 0.00310. This rejects the claim that the mean is at or above +2. Both p-values are below 0.05, so the prespecified TOST criterion is met.

If a single TOST p-value is reported for this construction, it is the larger of the two component p-values: approximately 0.00310. It is not their sum or average. The larger value captures the more difficult boundary to exclude.

A separate ordinary two-sided test of μ = 0 gives t = 1 and p approximately 0.327. Thus these fictional data support equivalence within ±2 millimetres while not establishing a difference from zero. The two results answer different questions and are consistent with each other.

Contents · Previous chapter · Next chapter

For this paired t model, TOST with two one-sided tests at 0.05 corresponds to checking whether the ordinary two-sided 90 percent confidence interval lies wholly inside the equivalence bounds. The interval is the mean plus or minus 1.7109 standard errors.

For the first example, that is 0.50 ± 1.7109 × 0.50, giving approximately [−0.355, 1.355] millimetres. Both ends are strictly inside [−2, 2]. The interval therefore gives the same equivalence decision as the two component tests.

The 90 percent interval here is not an attempt to conceal uncertainty by choosing a narrower display after looking at the result. It is the interval mathematically corresponding to the prospectively specified pair of one-sided 5 percent tests. A 95 percent interval may also be shown, but it answers the ordinary two-sided 5 percent difference-test question.

In this example, the 95 percent interval is approximately [−0.532, 1.532] millimetres. It includes zero, consistent with the non-significant difference test. In general, requiring a 95 percent interval to fit inside the margins would be a more conservative rule than this specified TOST procedure.

Neither interval assigns a 90 or 95 percent probability to the fixed population mean after the data are observed in the ordinary frequentist interpretation. The confidence level describes the repeated-sampling coverage of the procedure under its assumptions. Equivalence is supported at a stated error-control level, not proven with certainty.

Contents · Previous chapter · Next chapter

Draw a horizontal difference axis with vertical lines at −2, 0 and +2 millimetres. For each case, plot a thick 90 percent interval and a thinner 95 percent interval. Use the first for equivalence and the second for the ordinary difference test. All four fictional cases below use 25 independent paired observations and 24 degrees of freedom.

Case A: equivalent, with no detected difference. This is our mean 0.50 and standard error 0.50 example. Its 90 percent interval [−0.355, 1.355] lies inside the margins, while its 95 percent interval includes zero. Practical similarity is supported for the mean-bias target.

Case B: inconclusive on both questions. Keep the mean at 0.50 but increase the standard error to 1.50, corresponding to a paired-difference standard deviation of 7.50. The 90 percent interval is [−2.066, 3.066], so equivalence is not established. The 95 percent interval [−2.596, 3.596] includes zero, so a difference is not established either.

Case C: equivalent and statistically different. Use mean 1.00 and standard error 0.25, with standard deviation 1.25. The 90 percent interval [0.572, 1.428] fits within the margins. The 95 percent interval [0.484, 1.516] excludes zero. A small nonzero mean bias is detectable and still acceptable under the hypothetical requirement.

Case D: evidence of a difference beyond the margin. Use mean 3.00 and standard error 0.40, with standard deviation 2.00. The 95 percent interval [2.174, 3.826] is entirely above +2. This provides evidence of an unacceptably large positive mean bias under the model, rather than merely a failure to establish equivalence.

Contents · Previous chapter · Next chapter

Case B is the most important warning. Its point estimate is the same as Case A’s, but its uncertainty is much larger. The data are compatible with negligible bias and with bias outside the acceptable region. Calling the methods “different” because equivalence failed would overstate the evidence.

Conversely, calling Case B “the same” because the zero-difference test failed would make the opposite overstatement. The correct result is unresolved at the stated precision and thresholds. A study can fail to establish both a difference and equivalence without any contradiction.

Case C explains another common confusion. Statistical difference from zero is not the same as practical importance. With sufficiently precise measurement, researchers can detect a mean difference that remains well inside the range judged acceptable for the particular use.

The word “equivalent” should therefore always carry its object and boundary: equivalent in mean bias within ±2 millimetres for these conditions, for example. It does not mean numerically identical or similar in every scientifically relevant property.

Non-inferiority is related but asks a different question. It may require ruling out only an unacceptable disadvantage on one side. If a study needs both-sided similarity, passing a one-sided non-inferiority boundary does not supply the missing other half. Researchers should choose the question before seeing which test is easiest to pass.

Contents · Previous chapter · Next chapter

Equivalence studies need adequate precision. A narrow practical margin, a variable outcome and a true mean close to a boundary can make equivalence difficult to establish. Merely collecting a large-looking number of measurements does not guarantee adequate independent information.

A separate planning illustration makes this visible. Assume independent normally distributed paired differences with a genuinely known population standard deviation of 5 millimetres, true mean zero and fixed margins ±2. This known-variance z model is different from the earlier estimated-variance t examples.

At n = 25, the standard error is 1 millimetre. The 90 percent interval fits within the margins only when the observed mean lies between approximately −0.355 and +0.355. Under the assumed true mean zero, that occurs with probability about 0.278, or 27.8 percent.

At n = 100, the standard error is 0.50 millimetres. The permitted observed-mean range becomes approximately −1.178 to +1.178. Relative to the sampling standard error, this gives power about 0.981, or 98.1 percent. These values follow from the stated normal model; they are not recommendations for an actual study.

Planning should explore plausible nonzero means and uncertain variability as well. A mean nearer +2 or −2 leaves less room for the confidence interval. Estimated variance, clustering, missing data and departures from the working distribution require suitable calculations rather than copying the known-variance result.

Repeatedly measuring the same few sheets is not equivalent to sampling many independent sheets. Repeats may improve measurement precision, but the model must retain their dependence and distinguish within-sheet noise from variation across the target population of sheets.

Contents · Previous chapter · Next chapter

Schuirmann’s original 1987 methods paper compared the two one-sided procedure with an approach based on failing to detect a difference and assessing power. It also established the connection to the corresponding confidence-interval approach in its setting. The lesson carries beyond that original application: similarity requires a test designed for a similarity claim.

Lakens’s 2017 practical methods paper explains equivalence testing across familiar statistical settings and presents the distinction between difference testing and equivalence testing. Its worked framework helps expose the four possible combinations rather than forcing every result into a simple “effect” or “no effect” label.

Current methods research still refines these distinctions. Shieh’s 2026 original study compares equivalence procedures for standardised and unstandardised mean differences. It examines differences in rejection rules, confidence intervals, power and margins. That matters because changing from millimetres to an effect divided by an estimated standard deviation is not merely relabelling the axis.

Our worksheet keeps the target in raw millimetres and uses the paired mean-difference t model. Its numerical results should not be transferred unchanged to ratios, correlations, binary outcomes or standardised effects. Each requires a suitable estimand, margin and inferential procedure.

Likewise, normality and independence are not decorations attached to a formula. Extreme outliers, selective missing measurements or correlated errors can change the reliability of the interval. A robust or resampling approach may be appropriate, but its assumptions and equivalence-test calibration must be established rather than assumed.

Contents · Previous chapter · Next chapter

For Case A, a useful report would state the paired design, 25 independent sheets, mean difference 0.50 millimetres, standard deviation 2.50, prespecified margins ±2, 90 percent interval [−0.355, 1.355] and TOST p approximately 0.00310. It would conclude that the mean-bias equivalence criterion was met under the stated model.

The report should also explain the margin’s practical justification and the conditions sampled. If the sheets cover only one printer, paper type or line-length range, the evidence does not automatically extend to all printers, papers and image scales.

To see why individual agreement remains separate, imagine paired errors alternating around +10 and −10 millimetres. Their mean could be near zero even though individual readings are poor substitutes. A mean-equivalence result therefore needs complementary assessment of error spread, outliers and relevant performance requirements before any broader interchangeability claim.

For your worksheet, reproduce both t statistics for Case A, draw all four cases and classify them using the appropriate interval. Then tighten the hypothetical margin to ±1 without pretending it was the original requirement. Case A’s 90 percent interval no longer fits. This sensitivity exercise shows how conclusions depend on the chosen question; it does not authorise changing that question silently.

Finally, choose the reporting sentence justified by the calculations. For Cases A and C: “The data support mean similarity within the prespecified bounds.” For Case B: “The data remain too imprecise to establish similarity within those bounds.” For Case D: “The data provide evidence that the positive mean bias exceeds the acceptable upper bound.” Then identify the remaining measurement and generalisation limits.

Continue with Science Mastery and the How Science Works roadmap. Good evidence for similarity comes from a meaningful boundary and enough reliable information to test it, not simply from a result that failed to reach significance.

Contents · Previous chapter · Continue to Science Mastery