Find your next learning step
Positive counts can reveal a count process while hiding every zero. This guide separates the conditional model you can fit from the population total you can only recover with an explicit observation frame.
Open the chapter contents · Probability root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The teaching chapters remain open; these grouped routes form the index.
Route 1 · See what truncation changes
Route 2 · Build the GIGP regression
Route 3 · Separate missing zeros
Route 4 · Estimate and challenge
Route 5 · Check and transfer
CHAPTER 1 OF 12 · SEE WHAT TRUNCATION CHANGES
1. Positive-only data changes the question
Suppose a register contains the number of events produced by every unit that produced at least one event. The register is not merely a count sample with a few missing values. It is a sample conditional on Y being greater than zero. Units with Y equals zero have no rows, so the observed distribution answers “how are positive counts distributed?” rather than “how many units exist?”
That distinction matters in ecology, insurance, informetrics, reliability and service logs. A flexible count model can describe the visible positive tail very well and still leave the number of unseen zeros weakly identified. The first discipline is therefore to write down the observation rule before choosing a likelihood.
A useful notation
Let p sub y denote the probability of count y under an untruncated model. The zero-truncated probability for y at least one is p sub y divided by one minus p sub zero. The denominator renormalises the positive support; it does not create information about how many zero-count units were omitted.
CHAPTER 2 OF 12 · SEE WHAT TRUNCATION CHANGES
2. The conditional likelihood and what it contains
For observed positive counts y1 through yn, the zero-truncated log-likelihood adds the ordinary log probabilities and subtracts n times the logarithm of one minus p0. This correction is essential. Fitting the ordinary likelihood to positive-only data silently treats the missing zero class as impossible and usually distorts the mean, dispersion and tail parameters.
The conditional likelihood can identify parameters that alter the relative frequencies of ones, twos, threes and larger counts. It can also contain information about p0 through the model’s structural link between zero probability and the positive distribution. That is model-based information, not direct observation of zero units. If different parameter combinations produce nearly the same positive probabilities but different p0 values, the population-size estimate will be unstable.
CHAPTER 3 OF 12 · SEE WHAT TRUNCATION CHANGES
3. Worked example: renormalising the positive counts
Consider a teaching distribution with untruncated probabilities 0.20, 0.25, 0.20, 0.15 and 0.20 for zero, one, two, three and four-or-more events. Conditioning on a positive count divides each positive probability by 0.80. In a positive-only sample of 800 units, the expected visible counts are therefore 250, 200, 150 and 200.
| Count class | Untruncated probability | Positive-only probability | Expected among 800 positives |
|---|---|---|---|
| 0 | 0.2000 | Not observed | 0 |
| 1 | 0.2500 | 0.3125 | 250 |
| 2 | 0.2000 | 0.2500 | 200 |
| 3 | 0.1500 | 0.1875 | 150 |
| 4 or more | 0.2000 | 0.2500 | 200 |
The arithmetic exposes the boundary. The 800 positive units tell us their conditional composition. Converting 800 into 1,000 total units uses the additional assumption that all positive units were captured and that p0 really is 0.20.
CHAPTER 4 OF 12 · BUILD THE GIGP REGRESSION
4. The GIGP mixing structure
The generalized inverse Gaussian–Poisson model starts with a Poisson count conditional on a latent rate, then lets that rate follow a generalized inverse Gaussian distribution. Mixing turns simple Poisson variation into a wider family with overdispersion and flexible right tails. Special parameter limits connect the family to familiar count mixtures, including the Poisson–inverse-Gaussian case.
The model is useful when the positive counts show more heterogeneity than a single Poisson rate can carry. It is not a licence to assign every heavy positive tail to one mechanism. A long tail may reflect latent exposure, aggregation, reporting behaviour or a genuine mixture of subpopulations. The fitted mixing law should be challenged against those possibilities.
The preceding Poisson–GIG count-mixture guide develops the mixing family and its boundary cases. Here the extra difficulty is that truncation removes the zero class before estimation.
CHAPTER 5 OF 12 · BUILD THE GIGP REGRESSION
5. Regression, exposure and the visible mean
In regression, covariates usually enter a parameter governing the latent or marginal count rate. An exposure offset may represent time observed, area sampled or number of opportunities. The conditional positive mean is not the same as the untruncated marginal mean: conditioning lifts it because zeros have been removed.
A good model therefore reports both scales. For each covariate pattern, show the untruncated mean, the modelled p0 and the conditional positive mean. A coefficient can look modest on the latent-rate scale yet imply a large change in the chance of being observed. Conversely, an exposure variable recorded only for visible units cannot repair an unknown sampling frame by itself.
Compare the structure with Poisson–inverse-Gaussian regression and Poisson–lognormal regression. Agreement across plausible mixing families is evidence of robustness; a large spread in p0 is evidence that unseen-zero recovery is model-sensitive.
CHAPTER 6 OF 12 · SEPARATE MISSING ZEROS
6. Truncation is not zero inflation or censoring
Zero truncation means zeros cannot appear in the observed dataset because of the inclusion rule. Zero inflation means zeros are observed and occur more often than a baseline count model expects. A hurdle model separately models whether the count is zero and, if positive, its size. Censoring records that a value lies below or above a limit, while truncation removes the unit entirely.
These mechanisms can coexist, but they answer different questions. A list of customers with purchases cannot reveal noncustomers unless a customer frame exists. A sensor that records “below detection” does retain a row and is therefore closer to censoring. Calling both situations “many zeros” hides the decisive design information.
CHAPTER 7 OF 12 · SEPARATE MISSING ZEROS
7. Recovering unseen zeros requires an observation frame
If every positive unit is captured, a fitted p0 suggests a Horvitz-like expansion N hat equals n positive divided by one minus p0. The formula is transparent, but its uncertainty has two parts: uncertainty in the fitted p0 and uncertainty about completeness of positive ascertainment. The second part is not estimated by the zero-truncated count likelihood.
Population recovery becomes more credible when there is a known list of eligible units, repeated independent lists, a documented search effort, or an external estimate of positive-capture probability. Without such information, present the total as a sensitivity range indexed by p0 and capture completeness, not as a discovered census.
CHAPTER 8 OF 12 · SEPARATE MISSING ZEROS
8. Changed context: the same 800 rows, different totals
Keep the fitted p0 at 0.20 and suppose the file contains 800 observed positive units. Under complete ascertainment, 800 represents 80 percent of the population and the implied total is 1,000. If only 80 percent of positive units enter the file, the inclusion probability is 0.80 times 0.80 equals 0.64, and the implied total is 1,250. If p0 is instead 0.35 with 80 percent capture, the implied total rises to about 1,538.
| Modelled zero probability | Positive-capture probability | Overall inclusion probability | Implied population from 800 rows |
|---|---|---|---|
| 0.20 | 1.00 | 0.800 | 1,000 |
| 0.20 | 0.80 | 0.640 | 1,250 |
| 0.35 | 0.80 | 0.520 | 1,538 |
| 0.35 | Unknown | Unknown | Not identified from this file alone |
The transfer check is simple: whenever a total is reported, ask which factor estimates zero absence and which factor estimates missed positive units. If only the first appears, the total assumes complete positive ascertainment.
CHAPTER 9 OF 12 · ESTIMATE AND CHALLENGE
9. Estimation, computation and boundary checks
The GIGP likelihood involves special functions and can be numerically delicate. Optimisation should use stable log-scale evaluations, multiple starting points and explicit parameter constraints. Profile the zero probability or derived population total rather than relying only on a local Hessian. A flat or one-sided profile is a boundary warning, not a software inconvenience.
Check fitted and observed positive frequencies, especially ones and the far tail. Refit after grouping the sparsest tail cells, vary the mixing family, and examine whether influential high counts control the estimate. The original zero-truncated GIGP implementation by Burrell and Fenton demonstrates both the feasibility of maximum likelihood and the importance of recognising data for which the motivating model is inappropriate.
CHAPTER 10 OF 12 · ESTIMATE AND CHALLENGE
10. Evidence checks, limits and misconceptions
The primary sources are Burrell and Fenton’s zero-truncated GIGP implementation, Sichel’s GIGP formulation, and later work on the Sichel model’s mixing and truncation order. They support the conditional modelling framework; they do not guarantee that a population total is identified in every application.
Common mistakes are to fit an untruncated likelihood, treat p0 as observed, equate zero truncation with zero inflation, omit exposure, or report a single population estimate without a capture assumption. Another mistake is to select the mixing family only by in-sample fit. Tail and zero extrapolations deserve out-of-sample or sensitivity checks because the unobserved class cannot validate itself.
CHAPTER 11 OF 12 · CHECK AND TRANSFER
11. Questions readers usually ask
Can positive counts identify p0?
Only through restrictions imposed by the chosen count family. Strong curvature in the positive frequencies may constrain p0, but this is indirect and can be sensitive to the mixing law.
Does a better tail fit guarantee a better population estimate?
No. Two models can fit visible tail counts similarly and imply different zero probabilities. Compare derived p0 and total-size profiles, not just a goodness-of-fit statistic.
Should zero-truncated regression include an offset?
Yes when units have different opportunities to produce counts, provided exposure is defined for the target population. An offset observed only after inclusion may itself be selected.
What should be reported when capture completeness is unknown?
Report the positive-count model, a range over plausible capture probabilities and the assumptions needed to turn that range into a total.
CHAPTER 12 OF 12 · CHECK AND TRANSFER
12. A transfer routine and connected reading
Start by drawing the inclusion rule. Write the untruncated model, derive the positive conditional likelihood, and inspect whether the positive frequencies constrain the zero probability. Then separate the modelled zero fraction from the probability that a positive unit reaches the dataset. Profile both quantities and carry their uncertainty into any population total.
For neighbouring count families, read Poisson–Tweedie regression, generalized-Poisson regression and conditional Poisson rate-ratio inference. The reusable principle is broader than GIGP: conditional data can support a careful conditional model while leaving the size of the missing frame unresolved.
Top · Contents · Previous · How X Works