Probability becomes useful when an asymptotic tie is treated as the beginning of the comparison, not the end.
When equal efficiency still needs unequal sample sizes
Two tests can share the same first-order asymptotic efficiency and still require different sample sizes to reach the same power. Deficiency asks what additive sample-size correction remains after the leading ratio has tied.
This guide builds the idea from power matching, works an original numerical example, shows how finite-sample rankings can reverse and separates an expansion from evidence about a dataset.
Choose a reading route
Start with the route closest to the decision you need to make. Every route reaches the same worked example, transfer check and evidence limits.
- Concept route — separate first-order efficiency from second-order deficiency.
- Calculation route — match powers in a transparent worked example.
- Simulation route — estimate a gap without pretending it is a theorem.
- Evidence route — audit alternatives, calibration and remainder terms.
- Transfer route — reuse the method for another pair of tests.
Open the 12-chapter contents
- 1. An asymptotic tie can conceal a real cost
- 2. Three efficiency questions, three answers
- 3. Define deficiency by matching a target
- 4. Worked example: ratio one, gap four
- 5. How a finite-sample reversal appears
- 6. Why rank procedures invite this question
- 7. From expansion to sample-size correction
- 8. Simulation is a measurement, not a proof
- 9. Changed assumptions can move the crossing
- 10. What a defensible claim must state
- 11. Limits and misconceptions
- 12. A practical transfer routine
1. An asymptotic tie can conceal a real cost
First-order efficiency compares the dominant sample-size scale. If tests A and B need the same leading multiple of observations as an effect shrinks, their asymptotic relative efficiency is one. That does not make their powers equal at n = 40, 100 or 500.
A smaller additive gap may remain. Three extra observations disappear as a proportion of a huge sample, yet matter when recruitment is expensive or slow.
2. Three efficiency questions, three answers
Pitman efficiency studies local alternatives and leading sample-size ratios. Bahadur-type efficiency studies exponential significance slopes under fixed alternatives. Deficiency compares sample sizes after the leading ratio has aligned.
| Question | Object | Cannot settle |
|---|---|---|
| First order | leading ratio | small additive gaps |
| Second order | matched sample-size difference | far-from-limit behaviour |
| Finite sample | exact or simulated power | a general asymptotic theorem |
3. Define deficiency by matching a target
Let πA(n,θ) be the power of A. Choose m so that πB(m,θ) approximately equals πA(n,θ). The directional deficiency of B relative to A is d(n,θ) = m − n.
Positive d means B needs more observations; negative d means fewer. The ordering, alternative, level and target power belong in every report.
4. Worked example: ratio one, gap four
In an illustrative local-power model, A reaches power 0.80 at n = 96. B has the same leading information rate but a second-order loss worth four observations, so it reaches 0.80 near n = 100.
At this size the raw ratio is 100/96, but as the common leading requirement grows and the additive correction stays near four, the ratio tends to one. B’s deficiency relative to A is about +4. This is a teaching model, not a claim about a named rank test.
5. How a finite-sample reversal appears
Suppose B is slightly ahead at n = 30 because a discrete critical value is favourable, the curves cross near n = 60, and A is ahead afterwards because its next expansion term is better. No contradiction exists: the ordering depends on sample size and calibration.
Report the crossing region instead of naming a universal winner.
6. Why rank procedures invite this question
Ranks can be robust to monotone scale changes and broad distributional departures. Classic theorems give strong first-order guarantees for some rank tests, but ties, discreteness and studentisation still shape finite samples.
| Feature | Benefit | Caution |
|---|---|---|
| Ranks | reduced scale sensitivity | ties alter calibration |
| Local alternatives | clean leading comparison | path to the null matters |
| Distribution-free null | stable size under assumptions | power depends on the alternative |
7. From expansion to sample-size correction
Expand power in the local parameter and powers of n⁻¹ᐟ². Leading unequal terms determine a first-order ratio. If they match, the next unequal coefficient determines the correction needed to align powers.
The correction may approach a constant, grow slowly such as log n, or depend on the alternative sequence. Call it a constant only when the derivation supports that limit.
8. Simulation is a measurement, not a proof
Specify null, alternative, sample sizes, level, sidedness and exact test implementation. Simulate independent datasets, record rejection rates and attach Monte Carlo intervals.
Interpolate a target crossing only on a sufficiently dense grid. If simulation error is as large as the power difference, the deficiency is unresolved.
9. Changed assumptions can move the crossing
Skewness, heavy tails, ties, unequal groups and studentisation can move or remove a reversal. A permutation calibration may remove a size distortion that looked like power.
Compare actual type-I error before interpreting power. Rejecting too often is not a fair advantage.
10. What a defensible claim must state
State hypotheses, asymptotic regime, level, sidedness, nuisance assumptions, target power, ordering of tests, expansion order and remainder condition. For simulation, add replication count, random-seed policy and uncertainty intervals.
Reproducibility is part of the statistical result.
11. Limits and misconceptions
This deficiency is not Le Cam deficiency between experiments and not the Hodges–Lehmann estimator. It is an additive sample-size comparison between procedures.
A smaller deficiency in one local model does not prove uniformly higher power. Asymptotic equivalence does not promise equality at today’s n.
12. A practical transfer routine
Check size, plot power across sample sizes and several alternatives, match a target power, compute the directional sample-size gap, vary the data distribution and explain crossings through calibration or next-order terms.
That routine turns efficiency from a label into a falsifiable comparison.
Changed-context transfer check
Tests A and B both approach power 0.90 by n = 240. At n = 60 A leads by 0.03; at n = 120 they tie; at n = 180 B leads by 0.01. Do not average the gaps. Locate the crossing, verify size and compute directional deficiencies for a chosen target under each alternative.
A successful transfer states what stayed invariant, what changed and which conclusion must be recalculated.
Frequently asked questions
Is deficiency one minus relative efficiency?
No. Relative efficiency is usually a leading ratio; deficiency is an additive correction after that ratio aligns.
Can deficiency be negative?
Yes. The sign depends on which procedure is required to catch up.
Does a first-order tie make the tests interchangeable?
No. Calibration, robustness, ties and finite-sample power may still differ.
How many simulations are enough?
Enough that Monte Carlo uncertainty is small relative to the difference being interpreted; report the interval.
Sources and evidence trail
These sources support the definitions and limits above. The numerical teaching examples are original illustrations, not reproduced experimental results.
- Hodges and Lehmann, Deficiency — the foundational additive comparison.
- Chibisov, Calculation of Deficiency — second-order calculation methods.
- Albers, Deficiencies of Combined Rank Tests — first-order equality motivating a second-order result.
- Nikitin on Rank Efficiencies — places rank efficiencies in precise regimes.
- Wilcoxon–Mann–Whitney or t-test? — assumptions behind common comparisons.
Continue through the eduKate ecosystem
Return to Probability
The Gold Standard of Probability places this specialist problem inside the subject’s wider methods, limits and applications.
Follow the closest specialist routes
- Chernoff–Savage and Hodges–Lehmann Efficiency Bounds — the first-order result this guide extends.
- Le Cam Deficiency, Approximate Garbling and Transfer of Risk — the different decision-theoretic meaning of deficiency.
Continue to The Gold Standard of Education: Learning Transfers into Life and the How X Works hub.