Find your next learning step
Count data can vary less than a Poisson model expects. Generalized-Poisson regression can represent that underdispersion, but only when its parameterization, support and mean model are made explicit.
Open the chapter contents · Probability root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The teaching chapters remain open; these grouped routes form the index.
Route 1 · See the variance gap
Route 2 · Work the model
Route 3 · Diagnose and transfer
Route 4 · Check limits and evidence
Route 5 · Ask and continue
CHAPTER 1 OF 12 · SEE THE VARIANCE GAP
1. When counts vary less than their mean
A Poisson model ties the conditional variance to the conditional mean. If a fitted mean is five events, the model also expects a variance of five. Some processes are more regular: appointment systems cap daily arrivals, manufacturing cycles impose spacing, or a fixed number of opportunities restricts how events can accumulate. Their counts may be underdispersed, with variance below the mean.
Underdispersion matters because a Poisson likelihood can overstate uncertainty and blur real covariate effects. Yet a small raw variance is not enough to diagnose it. The mean may change across units, repeated observations may be dependent, and rounding or censoring may compress the spread.
This guide extends the Probability root and the earlier guide to negative-binomial rate ratios. Negative-binomial NB2 models extra-Poisson spread; generalized-Poisson models can represent either direction under an explicitly stated parameterization.
CHAPTER 2 OF 12 · SEE THE VARIANCE GAP
2. One useful parameterization separates mean and dispersion
The original generalized-Poisson distribution is often written with parameters lambda and theta. Its mean is lambda divided by (1 minus theta), and its variance is lambda divided by (1 minus theta) cubed. Reparameterizing by the mean mu, with lambda = mu(1 minus theta), gives Var(Y) = mu divided by (1 minus theta) squared.
In this form, theta = 0 recovers Poisson. Positive theta gives overdispersion. Negative theta gives underdispersion, subject to the distribution’s support constraints. A log link can set log(mu_i) = x_i beta, and an exposure offset can be added when the count represents unequal observation time.
Software uses several generalized-Poisson definitions, signs and transformations. Never interpret a fitted dispersion coefficient until the package documentation has been matched to the formula actually used.
CHAPTER 3 OF 12 · WORK THE MODEL
3. Worked ledger: equal means, different spread
Consider two original six-unit datasets, each with mean five.
| Dataset | Counts | Mean | Sample variance | Variance-to-mean ratio |
|---|---|---|---|---|
| Regular process | 2, 4, 5, 6, 7, 6 | 5.0 | 3.2 | 0.64 |
| Bursty process | 0, 2, 3, 7, 8, 10 | 5.0 | 15.2 | 3.04 |
The regular process is less variable than Poisson at this pooled level; the bursty process is more variable. Using the reparameterized variance mu/(1-theta) squared, mu = 5 and theta = -0.25 give variance 5/1.25 squared = 3.2. A positive theta of about 0.426 gives variance near 15.2.
These matching values are teaching calculations, not fitted estimates. Regression should model the conditional mean first and diagnose dispersion from conditional residuals, not choose theta by equating one pooled variance.
CHAPTER 4 OF 12 · WORK THE MODEL
4. Regression makes the mean structure visible
Suppose count regularity changes with shift, machine or clinic. Fit the scientifically required mean predictors before judging dispersion. If omitted groups have different means, their mixture inflates the raw variance; adding the group can restore a near-Poisson conditional distribution.
With unequal exposure T_i, use log(T_i) as a fixed offset when expected count should scale proportionally with exposure. The regression coefficient exp(beta_j) is then a conditional rate ratio for a one-unit predictor change. Generalized-Poisson dispersion changes the conditional count distribution; it does not repair an incorrect exposure clock.
Plot observed and fitted means by important design variables. A model that predicts the wrong mean can produce a dispersion parameter that merely absorbs systematic structure.
CHAPTER 5 OF 12 · WORK THE MODEL
5. A mean–variance ledger compares candidate stories
At theta = -0.25, the generalized-Poisson variance is 0.64 times the mean for every row in this parameterization. At theta = 0.30, it is about 2.04 times the mean.
| Fitted mean mu | Poisson variance | GP variance at theta = -0.25 | GP variance at theta = 0.30 |
|---|---|---|---|
| 2 | 2.00 | 1.28 | 4.08 |
| 5 | 5.00 | 3.20 | 10.20 |
| 10 | 10.00 | 6.40 | 20.41 |
| 20 | 20.00 | 12.80 | 40.82 |
This constant variance-to-mean ratio is not the only possible count model. COM-Poisson regression, binomial mechanisms, hurdle models and correlated-process models imply different shapes. Compare the scientific mechanism as well as likelihood fit.
CHAPTER 6 OF 12 · DIAGNOSE AND TRANSFER
6. Diagnostics should be conditional and graphical
Start with a Poisson regression using the intended mean formula. Inspect Pearson residuals, randomized quantile residuals, fitted-versus-observed count frequencies and residual patterns across fitted mean, exposure, time and group. Pearson chi-square divided by residual degrees of freedom below one is a signal, not a verdict.
Fit the generalized-Poisson alternative and check parameter uncertainty, convergence, boundary proximity and influential observations. Compare predictive calibration using held-out or cross-validated data that respects clustering. A lower information criterion can support relative fit but cannot validate an implausible support or independence assumption.
For small samples, simulation from the fitted model can show whether the diagnostic statistic is unusual under the complete fitted procedure.
CHAPTER 7 OF 12 · DIAGNOSE AND TRANSFER
7. Changed context: a capacity cap creates underdispersion
Imagine a clinic with ten appointment slots and very few walk-ins. Daily attended counts may cluster near eight or nine because the upper capacity truncates variation. A generalized-Poisson model may fit the mean and variance, but the physical mechanism is bounded.
A binomial or beta-binomial attendance model with known slots can make that mechanism clearer. If capacity changes by day, the number of slots belongs in the model. The transfer question is whether the count has an explicit number of opportunities rather than an unbounded event stream.
Statistical flexibility should not hide a design constraint that readers can understand directly.
CHAPTER 8 OF 12 · DIAGNOSE AND TRANSFER
8. Changed context: serial spacing looks like underdispersion
Events that inhibit one another in time—maintenance cycles, refractory periods or scheduled dispatches—can create regular counts within fixed windows. Marginal underdispersion then arises from dependence between event times, not from independent generalized-Poisson draws.
Changing the window length can change the apparent dispersion. Check several defensible aggregation windows and preserve event timestamps where possible. A renewal-process or point-process model may transfer better than a count regression if timing is scientifically central.
The same variance ratio can come from different mechanisms. Diagnosis should follow the process, not only the histogram.
CHAPTER 9 OF 12 · CHECK LIMITS AND EVIDENCE
9. Limits and common misconceptions
Underdispersion does not mean the data are “too good”. It may reflect regulation, dependence, censoring, bounded opportunities or data processing. Generalized Poisson is not a universal correction; negative dispersion can impose finite-support constraints that must contain every observed count.
The sign and scale of the dispersion parameter are software-specific. Do not compare theta values across packages without translating formulas. A likelihood-ratio test against Poisson may involve nonstandard behaviour near a boundary; use the procedure documented for the implementation.
Finally, a dispersion parameter does not turn an associational regression coefficient into a causal effect.
CHAPTER 10 OF 12 · CHECK LIMITS AND EVIDENCE
10. Evidence and reproducibility route
Consul and Famoye introduced the regression framework for both under- and overdispersed counts in Generalized Poisson regression model. For the contrasting overdispersion route, Ver Hoef and Boveng compare quasi-Poisson and negative-binomial variance structures in Quasi-Poisson vs. negative binomial regression.
A reproducible record includes the probability mass function or package parameterization, link, offset, support restrictions, mean formula, dispersion estimate and interval, convergence messages, residual plots, count-frequency calibration, influential-case checks, validation split and software version. Preserve enough information to regenerate predicted probabilities, not only fitted means.
CHAPTER 11 OF 12 · ASK AND CONTINUE
11. Frequently asked questions
Can a negative-binomial model handle underdispersion?
The common NB2 model adds variance above the mean. It does not represent genuine underdispersion.
Is a Pearson ratio below one sufficient?
No. It can be affected by fitted leverage, small samples, an overfitted mean or dependence. Use multiple diagnostics and a process explanation.
Why can negative theta restrict the support?
In the original generalized-Poisson form, probabilities require lambda plus theta times y to remain valid. Negative theta can therefore imply a largest admissible count.
Should I prefer COM-Poisson or generalized Poisson?
Neither dominates universally. Compare parameter interpretation, support, computation, predictive calibration and the mechanism that makes the process regular.
CHAPTER 12 OF 12 · ASK AND CONTINUE
12. Final transfer check and connected reading
Before accepting a generalized-Poisson result, answer six questions: Is dispersion diagnosed after the mean is modelled? Which exact parameterization is used? Does its support contain every observed and plausible count? Is exposure handled correctly? Could capacity or serial dependence explain the regularity? Does the conclusion survive a mechanism-matched alternative?
Continue through the Probability root, negative-binomial rate ratios, exact stratified Poisson tests, the Gold Standard of Education and the How X Works hub. The transferable skill is to separate the mean model, variance law and physical reason for regular counts.
Top · Contents · Previous · How X Works