Find your next learning step
A zero-truncated count regression can predict visible positive counts well while its implied unseen population remains fragile. This guide separates covariate choice, conditional residual checking and population-total sensitivity.
Open the chapter contents · Probability root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The teaching chapters remain open; these grouped routes form the index.
Route 1 · Frame the conditional question
Route 2 · Select and diagnose
Route 3 · Stress-test the total
Route 4 · Validate and challenge
Route 5 · Check and transfer
CHAPTER 1 OF 12 · FRAME THE CONDITIONAL QUESTION
1. Positive-only regression has two targets
When a dataset contains only units with at least one event, regression begins with a conditional target: how the positive count distribution changes with covariates. A second target may then be derived: how many zero-count units were absent. These targets share parameters, but they do not have the same evidential strength.
For a covariate pattern x, let p0(x) be the untruncated probability of zero. The observed likelihood uses probabilities conditional on Y being greater than zero. A covariate can improve the relative fit among ones, twos and larger counts while also changing p0(x), and therefore changing an expanded population total. Model selection must inspect both consequences.
The preceding zero-truncated GIGP guide separates visible-count modelling from unseen-zero recovery. Here the focus is narrower: selecting covariates without letting a good positive fit disguise a fragile total.
CHAPTER 2 OF 12 · FRAME THE CONDITIONAL QUESTION
2. Conditional effects are not ordinary count effects
In an ordinary count regression, a coefficient may describe a change in an untruncated mean or latent rate. In a zero-truncated model, the data directly reveal a change in the positive conditional distribution. Because selection removes zeros, the conditional mean equals the untruncated mean divided by one minus p0 when that identity applies. Its covariate response can differ sharply from the latent-rate response.
Report predictions on three scales for representative covariate patterns: the untruncated mean, p0 and the mean conditional on being positive. An exposure offset should represent opportunities defined before inclusion, such as observation time or sampled area. Exposure measured only after a unit enters the positive register may itself be selected and should not be treated as a neutral offset.
The same discipline applies across neighbouring mixtures. Compare the fitted scales with Poisson–inverse-Gaussian regression and Poisson–lognormal regression rather than assuming that one mixing law fixes the meaning of a coefficient.
CHAPTER 3 OF 12 · FRAME THE CONDITIONAL QUESTION
3. Worked contrast: similar positive fit, different totals
Suppose a complete register contains 600 positive units in group A and 400 in group B. Model 1 estimates zero probabilities of 0.15 and 0.35. Under complete capture of positive units, the expanded group totals are 600 divided by 0.85, about 706, and 400 divided by 0.65, about 615. The combined estimate is 1,321.
Model 2 fits the observed positive frequencies almost as well but estimates p0 values of 0.20 and 0.45. Its totals are 750 and about 727, or 1,477 combined. The visible sample is unchanged, yet the total is about 12 percent larger.
| Group | Visible positives | Model 1 p0 | Model 1 total | Model 2 p0 | Model 2 total |
|---|---|---|---|---|---|
| A | 600 | 0.15 | 706 | 0.20 | 750 |
| B | 400 | 0.35 | 615 | 0.45 | 727 |
| Combined | 1,000 | Mixed | 1,321 | Mixed | 1,477 |
This is the practical selection warning. A small improvement in conditional likelihood is not automatically worth a large and unstable change in the derived total.
CHAPTER 4 OF 12 · SELECT AND DIAGNOSE
4. Randomised quantile residuals for truncated counts
Raw residuals from discrete counts form bands, so an apparent pattern can be a consequence of discreteness rather than misspecification. Randomised quantile residuals place each observation uniformly within the fitted conditional probability jump, then transform that value through a standard normal quantile. For a well-fitted conditional distribution, the residuals should be approximately normal without covariate trends.
The cumulative distribution used here must be the zero-truncated conditional distribution, not the untruncated count distribution. Plot residuals against fitted positive means, every candidate covariate, exposure, time and group. Also inspect binned observed-versus-expected frequencies because a normal-looking residual plot can hide a local shortage of ones or an excess in the far tail.
Dunn and Smyth’s original residual paper supplies the general randomisation principle. The residual checks diagnose the visible conditional law; they do not validate the number of unseen zeros.
CHAPTER 5 OF 12 · SELECT AND DIAGNOSE
5. Covariate selection should protect interpretation
Begin with variables justified by the observation process and subject mechanism. Keep exposure, known design variables and confounders separate from optional predictors. For each candidate, compare conditional predictive performance, residual structure, coefficient stability and the change in derived p0 across the covariate range.
Blind stepwise selection is especially risky because small likelihood gains may be amplified through 1 divided by one minus p0. Prefer a small prespecified set, penalised or shrinkage fits when the set is large, and cross-validation that preserves clusters or time ordering. Interactions should have a reason: a group-by-exposure interaction can represent a real change in opportunity, while a freely searched interaction may merely chase sparse high counts.
The 2023 work on zero-truncated Poisson regression illustrates that estimation and variable selection must respect truncation. A selected model remains a conditional model until the observation-frame assumptions are supplied.
CHAPTER 6 OF 12 · STRESS-TEST THE TOTAL
6. Population totals are nonlinear predictions
For covariate cells j with observed positive counts n sub j and complete positive capture, a model-based total is the sum of n sub j divided by one minus p0(x sub j). This is a nonlinear prediction. Cells with large p0 receive large expansion weights, so sparse cells can dominate the total even when they have little influence on the positive-count fit.
Always display cell contributions. A total built from thousands of stable low-p0 cells and one tiny high-p0 cell is not uniformly precise. If positive capture is c(x), the denominator becomes c(x) times one minus p0(x). Unknown capture cannot be estimated from the truncated count likelihood merely by adding more count covariates.
CHAPTER 7 OF 12 · STRESS-TEST THE TOTAL
7. Sensitivity belongs beside the point estimate
Return to the 1,000 visible units. Vary both the modelled zero fraction and positive-capture probability. The exercise is not a pessimistic afterthought; it states exactly which assumptions turn a positive register into a population estimate.
| Average p0 | Positive capture | Overall inclusion | Implied total from 1,000 positives |
|---|---|---|---|
| 0.20 | 1.00 | 0.800 | 1,250 |
| 0.30 | 1.00 | 0.700 | 1,429 |
| 0.20 | 0.90 | 0.720 | 1,389 |
| 0.30 | 0.90 | 0.630 | 1,587 |
Repeat the table by important covariate cells rather than relying only on an average p0. A total that appears stable in aggregate may conceal offsetting changes between groups.
CHAPTER 8 OF 12 · STRESS-TEST THE TOTAL
8. Profile and bootstrap the quantity readers use
Standard errors for coefficients are not enough when the decision uses a population total. Profile the likelihood over the derived total or over parameters that control p0, and use a bootstrap that repeats the entire workflow: resampling, covariate selection where applicable, model fitting and total calculation. If clusters or periods are the independent units, resample those units.
Separate model uncertainty from capture uncertainty. A parametric bootstrap conditional on complete capture quantifies variation under that assumption; it cannot discover an unknown capture rate. Add external capture scenarios or a probability model supported by a known frame, repeated lists or audit data. Label these components so readers can see what comes from observed positive counts and what comes from design information.
CHAPTER 9 OF 12 · VALIDATE AND CHALLENGE
9. Validation needs changed contexts
Hold out whole groups, sites or periods and assess the positive conditional distribution. Record log score, calibration of positive-count intervals, residual patterns and tail frequencies. Then compare the p0 and total contributions predicted for held-out covariate patterns. Because zeros remain absent, stability across held-out positive data is supportive but not direct validation of p0.
A useful transfer check is to refit plausible mixture families. Compare Poisson–GIG count mixtures with Poisson–Tweedie regression. If conditional predictions agree but totals separate, report the spread as structural uncertainty rather than choosing the largest likelihood and hiding the consequence.
CHAPTER 10 OF 12 · VALIDATE AND CHALLENGE
10. Evidence boundaries and common mistakes
The zero-truncated GIGP implementation of Burrell and Fenton supports conditional maximum-likelihood modelling for positive counts. It does not make a missing population frame observable. Dunn and Smyth support conditional residual construction, not an unseen-zero census.
Common mistakes are using the untruncated CDF for residuals, selecting covariates only by coefficient significance, omitting exposure, treating a selected p0 as observed, and reporting a narrow coefficient interval beside an unexamined total. Another mistake is to run selection once and bootstrap only the final model; that omits selection instability.
The limit is clear: positive-only data can reject poor conditional models. They usually cannot, by themselves, distinguish every pair of models that extrapolate differently to zero.
CHAPTER 11 OF 12 · CHECK AND TRANSFER
11. Questions readers usually ask
Should a covariate be retained if it improves residuals but changes the total greatly?
Possibly, but the change must be explained and stress-tested. Check whether the variable was defined before inclusion, whether the improvement transfers to held-out positive data, and whether the total is stable across plausible count families.
Can ordinary Pearson residuals be used?
They can be supplementary, but their discrete bands and unequal variance are hard to read. Randomised quantile residuals based on the conditional truncated CDF provide a clearer common scale.
Is the lowest information criterion the best population model?
No. It ranks conditional fits under its assumptions. Population-total stability, capture assumptions and structural comparisons remain separate checks.
What if positive capture is unknown?
Report totals across explicit capture scenarios or stop at conditional positive-count inference. A precise total without a capture argument is false precision.
CHAPTER 12 OF 12 · CHECK AND TRANSFER
12. A reusable reporting routine
Write the inclusion rule first. Fit the zero-truncated likelihood, report predictions on latent, zero-probability and positive-conditional scales, and examine conditional randomised quantile residuals. Select covariates with design knowledge and changed-context validation. Then calculate cell-level population contributions, profile or bootstrap the total, and vary capture assumptions.
Keep the reading route connected to the Probability root, the Gold Standard of Education and the How X Works hub. The transferable lesson is simple: choose a conditional model for what the data show, and challenge every derived total at the boundary where the data stop.
Top · Contents · Previous · How X Works