VIEW THIS AS

Auto mode follows the Route Engine until you choose a viewpoint.

YOU ARE HERE

ROUTE CHECK

CONNECTED TO

WHAT NEXT

Use the canonical route for this room, or HELP if you are unsure.

Why Mathematics? | Survey Domain Estimation: Subpopulations and Correct Standard Errors

Why is mathematics important when a survey question concerns one subgroup but the sample design covers everyone? Domain estimation keeps the full survey design in the variance calculation and uses an indicator to define the subgroup. This preserves uncertainty in how many domain members each primary sampling unit would contain across repeated samples.

Did You Know? Deleting every out-of-domain row before running complex-survey software can produce the same subgroup point estimate yet a different, often too small, standard error.

Full chapter index

Chapters 1–7 · Understand the question
  1. The subgroup question
  2. What a domain is
  3. The official NHANES guidance
  4. A domain total
  5. A domain mean
  6. A worked domain mean
  7. A worked Taylor linearisation
Chapters 8–14 · Reproduce the example
  1. Why deleting rows changes variance
  2. Subpopulation commands
  3. Replicate-weight domains
  4. Small domains
  5. Empty cells
  6. Cross-classified domains
  7. Conditional models
Chapters 15–21 · Check assumptions
  1. Missing outcomes
  2. Comparison with post-stratification
  3. Comparison with separate surveys
  4. Connection to effective sample size
  5. Diagnostics
  6. Validation plan
  7. Limits
Chapters 22–28 · Explore applications
  1. A spreadsheet workflow
  2. A coding workflow
  3. Applications and careers
  4. A student investigation
  5. Common misconceptions
  6. Connection to recent mathematics articles
  7. Guidance for students and parents
Chapters 29–34 · Study and verify
  1. Practice questions
  2. Frequently asked questions
  3. A reporting checklist
  4. Zero is information
  5. Final domain audit
  6. Useful next reading

CHAPTER 1 OF 34 · Understand the question

1. The subgroup question

Back to contents

Analysts frequently need a mean for adults over 65, a proportion for one region or a model for students in a programme. The subgroup membership count within each sampled PSU is itself random under the design.

Contents · Next chapter

CHAPTER 2 OF 34 · Understand the question

2. What a domain is

Back to contents

A domain, subpopulation or subdomain is defined by an indicator D_i. Point estimates use D_i to include the target records while variance estimation retains the design information of all eligible sampled records.

Contents · Previous chapter · Next chapter

CHAPTER 3 OF 34 · Understand the question

3. The official NHANES guidance

Back to contents

CDC states that software requires all observations with nonzero analysis weight plus a subpopulation indicator; physically subsetting can treat domain sample size as fixed and underestimate variance. Read the NHANES module.

Contents · Previous chapter · Next chapter

CHAPTER 4 OF 34 · Understand the question

4. A domain total

Back to contents

Estimate a domain total as the weighted sum of D_i y_i. Records outside the domain contribute zero to the total but still help preserve PSU and stratum structure for variance estimation.

Contents · Previous chapter · Next chapter

CHAPTER 5 OF 34 · Understand the question

5. A domain mean

Back to contents

Estimate the mean as sum(w_i D_i y_i) divided by sum(w_i D_i). Both numerator and denominator are random weighted totals.

Contents · Previous chapter · Next chapter

CHAPTER 6 OF 34 · Understand the question

6. A worked domain mean

Back to contents

Suppose weighted domain outcome total is 360 and weighted domain population total is 120. The domain mean is 360/120=3.

Weighted outcome totalWeighted domain totalMean
3601203
Domain mean.

Contents · Previous chapter · Next chapter

CHAPTER 7 OF 34 · Understand the question

7. A worked Taylor linearisation

Back to contents

For ratio X/Y at X=360 and Y=120, derivatives are 1/120≈0.008333 and −360/120²=−0.025. These combine numerator and domain-size uncertainty.

dXdY
0.008333-0.025
Domain ratio gradients.

Contents · Previous chapter · Next chapter

CHAPTER 8 OF 34 · Reproduce the example

8. Why deleting rows changes variance

Back to contents

A stratum may appear to have one domain PSU after subsetting even though the full sample has two PSUs. The deleted PSU's zero domain contribution is part of the repeated-sampling contrast, not missing design metadata.

Contents · Previous chapter · Next chapter

CHAPTER 9 OF 34 · Reproduce the example

9. Subpopulation commands

Back to contents

Survey software usually offers a domain or subpopulation option. It keeps the design object intact while restricting the estimand, which differs from filtering the data first.

Contents · Previous chapter · Next chapter

CHAPTER 10 OF 34 · Reproduce the example

10. Replicate-weight domains

Back to contents

For each replicate, recompute the domain numerator and denominator using the domain indicator and replicate weights. Empty-domain replicates require a documented rule and stability warning.

Contents · Previous chapter · Next chapter

CHAPTER 11 OF 34 · Reproduce the example

11. Small domains

Back to contents

A domain may have many respondent rows but few contributing PSUs. Precision and degrees of freedom depend on the design support, not the raw subgroup count alone.

Contents · Previous chapter · Next chapter

CHAPTER 12 OF 34 · Reproduce the example

12. Empty cells

Back to contents

If no sampled domain member appears in a stratum or replicate, that is a data and design fact. Adding arbitrary pseudo-observations changes the estimand and requires a formal model.

Contents · Previous chapter · Next chapter

CHAPTER 13 OF 34 · Reproduce the example

13. Cross-classified domains

Back to contents

Combining age, region and programme indicators can create thin cells quickly. Pre-specify important domains and report multiplicity and reliability limits.

Contents · Previous chapter · Next chapter

CHAPTER 14 OF 34 · Reproduce the example

14. Conditional models

Back to contents

A model fitted only to a domain can still use full-design variance information through domain-aware software. The scientific target must clarify whether effects are descriptive within the domain or causal.

Contents · Previous chapter · Next chapter

CHAPTER 15 OF 34 · Check assumptions

15. Missing outcomes

Back to contents

Domain membership and outcome availability are different indicators. Missing-data treatment can change weights, denominators and the target population.

Contents · Previous chapter · Next chapter

CHAPTER 16 OF 34 · Check assumptions

16. Comparison with post-stratification

Back to contents

Domain estimation asks about a subgroup under an existing design. Post-stratification or calibration changes weights to match control totals; it is a weighting adjustment, not merely a subgroup filter.

Contents · Previous chapter · Next chapter

CHAPTER 17 OF 34 · Check assumptions

17. Comparison with separate surveys

Back to contents

A domain estimate from one national survey shares the original design and weights. It is not equivalent to a new simple random survey conducted only inside the subgroup.

Contents · Previous chapter · Next chapter

CHAPTER 18 OF 34 · Check assumptions

18. Connection to effective sample size

Back to contents

A domain's nominal n can overstate information when members cluster in few PSUs or carry unequal weights. Domain-specific design effects provide a more relevant precision summary.

Contents · Previous chapter · Next chapter

CHAPTER 19 OF 34 · Check assumptions

19. Diagnostics

Back to contents

Count domain respondents, contributing PSUs and strata; inspect empty and lonely domain cells; compare domain-command and naive-filter standard errors; verify the correct analysis weight.

Contents · Previous chapter · Next chapter

CHAPTER 20 OF 34 · Check assumptions

20. Validation plan

Back to contents

Reproduce the domain mean 3, compare full-design indicator analysis with physical filtering, test Taylor and replicate methods and simulate random domain counts within PSUs.

Contents · Previous chapter · Next chapter

CHAPTER 21 OF 34 · Check assumptions

21. Limits

Back to contents

Design-correct variance does not solve sparse-data bias, multiple testing, nonresponse, misclassification of domain membership or unsupported causal conclusions.

Contents · Previous chapter · Next chapter

CHAPTER 22 OF 34 · Explore applications

22. A spreadsheet workflow

Back to contents

Keep all rows, create a zero-one domain indicator and calculate weighted domain numerator, denominator, mean, contributing PSUs and design-based standard error.

Contents · Previous chapter · Next chapter

CHAPTER 23 OF 34 · Explore applications

23. A coding workflow

Back to contents

Build the full survey design first, call a domain estimator, unit-test 360/120=3 and compare with a deliberately filtered analysis to expose variance differences.

Contents · Previous chapter · Next chapter

CHAPTER 24 OF 34 · Explore applications

24. Applications and careers

Back to contents

Health, education, labour and social-policy analysts use domain estimation for trustworthy subgroup reporting.

Contents · Previous chapter · Next chapter

CHAPTER 25 OF 34 · Explore applications

25. A student investigation

Back to contents

Simulate two PSUs per stratum with random domain membership. Compare standard errors from full-design indicators and row filtering over many samples.

Contents · Previous chapter · Next chapter

CHAPTER 26 OF 34 · Explore applications

26. Common misconceptions

Back to contents

Out-of-domain rows are not useless to variance estimation, a large subgroup row count need not mean many PSUs, and a domain result is not automatically causal.

Contents · Previous chapter · Next chapter

CHAPTER 27 OF 34 · Explore applications

27. Connection to recent mathematics articles

Back to contents

The design-effect guide explains domain-specific information. The comparing-percentages guide helps keep subgroup denominators visible.

Contents · Previous chapter · Next chapter

CHAPTER 28 OF 34 · Explore applications

28. Guidance for students and parents

Back to contents

Students should write the indicator inside both numerator and denominator. Parents can ask how many schools, clinics or neighbourhoods actually support a subgroup estimate.

Contents · Previous chapter · Next chapter

CHAPTER 29 OF 34 · Study and verify

29. Practice questions

Back to contents

  • Calculate the domain mean 3.
  • Derive two ratio gradients.
  • Explain zero contributions.
  • Compare filtering and domain commands.
  • Count contributing PSUs.

Contents · Previous chapter · Next chapter

CHAPTER 30 OF 34 · Study and verify

30. Frequently asked questions

Back to contents

### Should out-of-domain rows be deleted? Usually not before design-based variance estimation.

### Is domain n fixed? Not under repeated sampling.

### Can a replicate domain be empty? Yes.

### Does correct variance remove bias? No.

Contents · Previous chapter · Next chapter

CHAPTER 31 OF 34 · Study and verify

31. A reporting checklist

Back to contents

Report domain definition, universe, weight, nominal n, contributing PSUs and strata, estimator, variance method, degrees of freedom, empty replicates and reliability rule.

Contents · Previous chapter · Next chapter

CHAPTER 32 OF 34 · Study and verify

32. Zero is information

Back to contents

An out-of-domain record contributes zero to the domain total while its PSU membership preserves the structure needed to measure how domain composition varies across samples.

Contents · Previous chapter · Next chapter

CHAPTER 33 OF 34 · Study and verify

33. Final domain audit

Back to contents

Confirm full-design retention, indicator-based estimation, numerator and denominator recomputation, contributing PSU counts, empty-cell handling and no causal overreach.

Contents · Previous chapter · Next chapter

CHAPTER 34 OF 34 · Study and verify

34. Useful next reading

Back to contents

Continue through the Mathematics Learning Hub. Recalculate one variance or effective-sample-size example, then explain which design feature changes the answer.

Contents · Previous chapter · Mathematics Learning Hub

Extended practice laboratory

Practice case 1: Define universe

Setup: Start from the article’s survey-domain-estimation mechanism for case 1. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 2: Define indicator

Setup: Start from the article’s survey-domain-estimation mechanism for case 2. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 3: Keep full design

Setup: Start from the article’s survey-domain-estimation mechanism for case 3. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 4: Calculate domain numerator

Setup: Start from the article’s survey-domain-estimation mechanism for case 4. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 5: Calculate denominator

Setup: Start from the article’s survey-domain-estimation mechanism for case 5. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 6: Calculate mean

Setup: Start from the article’s survey-domain-estimation mechanism for case 6. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 7: Derive gradients

Setup: Start from the article’s survey-domain-estimation mechanism for case 7. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 8: Count PSUs

Setup: Start from the article’s survey-domain-estimation mechanism for case 8. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 9: Count strata

Setup: Start from the article’s survey-domain-estimation mechanism for case 9. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 10: Use Taylor method

Setup: Start from the article’s survey-domain-estimation mechanism for case 10. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 11: Use replicate weights

Setup: Start from the article’s survey-domain-estimation mechanism for case 11. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 12: Check empty replicate

Setup: Start from the article’s survey-domain-estimation mechanism for case 12. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 13: Use percentage

Setup: Start from the article’s survey-domain-estimation mechanism for case 13. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 14: Use ratio

Setup: Start from the article’s survey-domain-estimation mechanism for case 14. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 15: Use regression

Setup: Start from the article’s survey-domain-estimation mechanism for case 15. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 16: Add missing outcomes

Setup: Start from the article’s survey-domain-estimation mechanism for case 16. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 17: Cross-classify domain

Setup: Start from the article’s survey-domain-estimation mechanism for case 17. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 18: Compare filtering

Setup: Start from the article’s survey-domain-estimation mechanism for case 18. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 19: Compare separate survey

Setup: Start from the article’s survey-domain-estimation mechanism for case 19. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 20: Calculate domain DEFF

Setup: Start from the article’s survey-domain-estimation mechanism for case 20. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 21: Audit weight

Setup: Start from the article’s survey-domain-estimation mechanism for case 21. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 22: Audit support

Setup: Start from the article’s survey-domain-estimation mechanism for case 22. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 23: Audit failures

Setup: Start from the article’s survey-domain-estimation mechanism for case 23. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 24: Name limits

Setup: Start from the article’s survey-domain-estimation mechanism for case 24. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.

Practice case 25: Write conclusion

Setup: Start from the article’s survey-domain-estimation mechanism for case 25. Task: Change the named quantity, recompute the dependent result and compare with the original.

Check: Preserve the sampling design, use the documented variance rule, state one diagnostic, record units and compare one defensible alternative with clear reasoning. Show the arithmetic, name the response and predictor units, and separate the numerical result from the claim it supports. Then vary one defensible assumption, calculate again and explain whether the conclusion changed because of the data, the fitted model or the validation purpose. Finish with one compact table or graph that could reveal a failure hidden by the summary.