Find your next learning step
A full nuisance-parameter maximum is safe but can be needlessly conservative; Berger–Boos restricts the search to a high-confidence set and adds its noncoverage probability to retain finite-sample validity.
Open the chapter contents · Probability root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The teaching chapters remain open; these grouped routes form the index.
Route 1 · Define the protected maximum
Route 2 · Work the nuisance ledger
Route 3 · Transfer the validity check
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · DEFINE THE PROTECTED MAXIMUM
1. Why nuisance maximisation can become too conservative
A finite-sample p-value must control its rejection probability for every parameter value allowed by the null hypothesis. When the null contains a nuisance parameter, a safe unconditional construction takes the largest tail probability over the whole nuisance space. That supremum is valid, but a remote nuisance value can dominate even when the data make it implausible.
Berger and Boos proposed a disciplined compromise. First construct a high-confidence set for the nuisance parameter. Then maximise the tail probability only inside that set and add the small probability, gamma, that the true nuisance value fell outside it. The addition is not decoration: it restores the finite-sample guarantee.
The method separates two tasks that are often blurred together: learning which nuisance values remain credible and protecting the test against the uncertainty left by that learning.
CHAPTER 2 OF 12 · DEFINE THE PROTECTED MAXIMUM
2. The Berger–Boos construction
Let eta denote the nuisance parameter and let p(x; eta) be a valid p-value if eta were known. Choose a confidence set C-gamma whose coverage is at least 1 − gamma under the null. The Berger–Boos value is the smaller of one and gamma plus the supremum of p(x; eta) over eta in C-gamma.
The guarantee follows by splitting error into two events. If eta lies inside the confidence set, the maximised p-value protects the test. If eta lies outside, that failure event has probability at most gamma. Adding gamma pays for the second event.
The confidence set, its coverage method, the nuisance parameterisation and gamma must all be specified before looking for a favourable result. A data-dependent confidence set is allowed because its noncoverage probability is explicitly included.
CHAPTER 3 OF 12 · WORK THE NUISANCE LEDGER
3. Worked example: a confidence-set maximum
Suppose two independent groups each contain 12 observations, with 9 successes in group A and 4 in group B. Under equality of success probabilities, the common rate is a nuisance parameter. An unconditional statistic can order all pairs of binomial counts, while each nuisance value supplies a different null probability for that tail set.
For a teaching ledger, take gamma = 0.001 and a 99.9% null confidence set C-gamma = [0.30, 0.72]. The tail values below are illustrative outputs from a declared grid, not a universal table for these counts.
| Common success rate eta | Tail probability p(x; eta) | Included in C-gamma? |
|---|---|---|
| 0.30 | 0.0318 | Yes |
| 0.45 | 0.0407 | Yes |
| 0.58 | 0.0415 | Yes |
| 0.72 | 0.0296 | Yes |
The calculation is only reproducible if the tail ordering, confidence-set algorithm, grid resolution and optimisation check are recorded. A coarse grid can miss an interior maximum and make a supposedly valid result too small.
CHAPTER 4 OF 12 · WORK THE NUISANCE LEDGER
4. Confidence sets are part of the test
Different confidence sets with the same nominal coverage can produce different p-values. A central interval may include nuisance values that a likelihood-based set excludes, while a conservative exact interval may be wider than an asymptotic interval. Wider sets tend to move the result toward the full-space supremum.
Gamma also creates a trade-off. A smaller gamma costs less when added to the p-value but usually requires a wider confidence set. A larger gamma narrows the set but adds more directly. The final value can therefore have an interior optimum, yet choosing gamma after inspecting outcomes turns one test into an unrecorded search.
Use a coverage method appropriate to the design. Exact binomial coverage, likelihood inversion and asymptotic normal intervals do not make identical finite-sample promises.
CHAPTER 5 OF 12 · WORK THE NUISANCE LEDGER
5. Changed context: two nuisance parameters
Now consider matched binary pairs or a stratified design with two nuisance coordinates. A rectangular confidence region may be easy to construct but can include implausible corners. A joint likelihood region may be smaller, yet its claimed coverage must still hold under the null.
| Nuisance structure | Confidence set | Verification burden |
|---|---|---|
| One scalar | Exact interval | Check endpoints and interior maximum |
| Two coordinates | Joint region | Check coverage and boundary search |
| Many strata | Projected or profiled set | Check numerical optimisation and conservatism |
Berger and Sidik used confidence-interval p-values in a matched-pairs problem with two nuisance parameters. The general idea transfers, but the region and optimiser must match the new sampling model.
CHAPTER 6 OF 12 · TRANSFER THE VALIDITY CHECK
6. Validity is not the same as optimality
A Berger–Boos p-value can be smaller than the full-space supremum while retaining finite-sample validity. It is not guaranteed to be the smallest valid p-value, the most powerful test for every alternative, or compatible with every confidence interval.
Power depends on the ordering statistic, discreteness and the alternative. Two valid procedures may spend the Type I error budget differently. Comparing only their observed p-values after seeing the data is method shopping, not a power study.
The useful comparison is planned: enumerate size over the null space, compute power over relevant alternatives, and record whether the matching confidence interval rejects exactly when the p-value does.
CHAPTER 7 OF 12 · TRANSFER THE VALIDITY CHECK
7. Numerical maximisation needs an audit trail
The supremum may lie at an endpoint, at a stationary point or on a discontinuity created by a discrete tail set. A general optimiser can stop too early; a regular grid can step over a narrow peak.
A defensible workflow evaluates endpoints, uses a fine exploratory grid, brackets candidate maxima, refines them, and repeats at tighter tolerances. If the statistic changes its rejection set as eta moves, those change points also deserve attention.
Report enough digits to show the result is stable, but do not imply precision beyond the underlying calculation. When the final p-value is near the decision threshold, numerical error should be much smaller than the reported margin.
CHAPTER 8 OF 12 · TRANSFER THE VALIDITY CHECK
8. Connection to conditioning and exact tests
The conditional-tests guide explains how conditioning can remove a nuisance parameter by fixing a sufficient statistic. Berger–Boos keeps an unconditional reference experiment and controls the nuisance through a restricted maximisation.
The mid-p guide changes how observed probability mass is counted. The randomised exact-test guide mixes at a rejection boundary. Neither operation is the same as restricting the nuisance space.
Keep sample-space choice, tail ordering, nuisance handling and boundary treatment as separate lines in the analysis record.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Limits and misconceptions
The gamma addition cannot be omitted. Without it, the procedure ignores the event that the confidence set missed the true nuisance value. Calling the set “99.9%” does not make that event impossible.
An estimated nuisance value is not a confidence set. Plugging in one estimate can be accurate asymptotically, but it does not inherit the finite-sample guarantee of the Berger–Boos construction.
The method does not repair a misspecified model, selective endpoints, optional stopping or biased sampling. Its validity is only as relevant as the null family and sampling design being maximised over.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Evidence and verification route
The foundational source is Berger and Boos, P Values Maximized Over a Confidence Set for the Nuisance Parameter. A modern non-asymptotic review is Fay and Hunsberger, Practical Valid Inferences for the Two-Sample Binomial Problem.
For a design with two nuisance parameters, see Berger and Sidik, Exact unconditional tests for a 2 × 2 matched-pairs design. Verification should reproduce confidence-set coverage, full and restricted suprema, endpoint checks, numerical tolerances, null size and planned power.
Archive the code and nuisance grid rather than only the final p-value. A second implementation should reach the same maximum to within the declared tolerance.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Why add gamma after maximising?
Gamma bounds the chance that the nuisance confidence set excluded the true null value. It is the price of restricting the maximisation.
Can gamma be zero?
Only if the confidence set covers the entire nuisance space, which returns the full supremum. In practice a small positive gamma is selected in advance.
Is the confidence set for the main effect?
No. It is a confidence set for the nuisance parameter under the null parameterisation.
Does a smaller p-value prove greater power?
No. Power is a repeated-sampling property over specified alternatives and must be compared at controlled size.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Final transfer check and connected reading
Repeat the worked ledger with gamma = 0.01 and a narrower confidence set whose largest tail value is 0.036. The result is 0.046, not 0.036. Explain why narrowing the set did not automatically lower the final p-value.
Then imagine the maximum occurs between two grid points. Describe the endpoint, grid and refinement checks needed before claiming finite-sample validity.
Continue with the Probability root, conditional tests, mid-p tests, the Gold Standard of Education and How X Works.
Top · Contents · Previous · How X Works