Find your next learning step
Build the latent-rate mixture, then test numerical integration and the predictive tail before trusting a fitted count model.
Open the chapter contents · Probability root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The teaching chapters remain open; these grouped routes form the index.
Route 1 · Build the mixture
Route 2 · Work the distribution
Route 3 · Fit and transfer
Route 4 · Respect the limits
Route 5 · Verify and continue
CHAPTER 1 OF 12 · BUILD THE MIXTURE
1. Why a lognormal latent rate changes count prediction
A Poisson model gives every observation one rate after its measured predictors and exposure are fixed. A Poisson–lognormal model allows that rate to keep varying through a Gaussian random effect on the log scale. Most observations remain near ordinary rates, while a smaller number can receive much larger positive rates.
This is useful when counts are overdispersed and a multiplicative latent mechanism is credible: sites differ by unmeasured productivity, patients differ by episode intensity, or production runs inherit persistent conditions that were not recorded. The model can also connect repeated or multivariate counts through correlated Gaussian effects.
Read it beside the Probability root, Poisson–inverse-Gaussian regression and negative-binomial regression. Similar variance does not imply the same latent tail or the same numerical fitting problem.
CHAPTER 2 OF 12 · BUILD THE MIXTURE
2. A mean-preserving parameterisation keeps quantities visible
Let Z follow a standard normal distribution. Conditional on Z, let Y be Poisson with rate
Λ=μ exp(σZ−σ²/2).
The subtraction σ²/2 makes E(Λ)=μ, so E(Y)=μ. The latent-rate variance is μ²[exp(σ²)−1], and the marginal count variance is
Var(Y)=μ+μ²[exp(σ²)−1].
Some software instead writes log Λ=xᵀβ+b with b∼N(0,σ²). In that form exp(xᵀβ) is the conditional median rate, while the marginal mean is exp(xᵀβ+σ²/2). State which interpretation is used before comparing coefficients or predictions.
CHAPTER 3 OF 12 · WORK THE DISTRIBUTION
3. Worked ledger: the same mean with a dispersed log rate
Take μ=4 and choose σ²=log(1.5)=0.4055. Then exp(σ²)−1=0.5 and Var(Y)=4+16×0.5=12. The variance-to-mean ratio is three.
The median latent rate is 4 exp(−σ²/2)=3.266. Its 95th percentile is 4 exp[−σ²/2+1.6449σ]=9.309. Thus the marginal mean of four is assembled from many rates below four and a thinner set of much larger rates.
| Quantity | Calculation | Result |
|---|---|---|
| Marginal count mean | μ | 4.000 |
| Log-rate variance | σ² | 0.4055 |
| Latent-rate variance factor | exp(σ²)−1 | 0.500 |
| Marginal count variance | μ+μ²[exp(σ²)−1] | 12.000 |
| Median latent rate | μexp(−σ²/2) | 3.266 |
| 95th percentile latent rate | μexp(−σ²/2+1.6449σ) | 9.309 |
CHAPTER 4 OF 12 · WORK THE DISTRIBUTION
4. The marginal probability needs an integral
The count probability averages a Poisson probability over the normal random effect:
P(Y=y)=∫ P(Y=y | Z=z) φ(z) dz.
There is no elementary closed form for this integral. Gauss–Hermite quadrature, adaptive quadrature, Laplace approximations or Monte Carlo integration are common choices. The method and its settings are part of the fitted model, not an invisible software detail.
For the worked case, 80-point Gauss–Hermite quadrature gives P(Y=0)=0.08321 and P(Y≥10)=0.06815. A Poisson count with mean four has P(Y=0)=0.01832 and a much lighter upper tail. Both extra zeros and large counts arise from rate mixing without a separate zero component.
CHAPTER 5 OF 12 · WORK THE DISTRIBUTION
5. Quadrature stability is an evidence check
The same worked distribution was evaluated with increasing quadrature orders. Ten points already gives a stable zero probability, but the upper tail changes more noticeably before settling.
| Quadrature points | P(Y=0) | P(Y≥10) | Change in upper-tail probability from prior row |
|---|---|---|---|
| 10 | 0.083213 | 0.068129 | — |
| 20 | 0.083212 | 0.068150 | 0.000021 |
| 40 | 0.083212 | 0.068149 | −0.0000002 |
| 80 | 0.083212 | 0.068150 | 0.00000003 |
Repeat this comparison at the fitted extremes of μ and σ². A quadrature order that works near the centre may fail for sparse counts, large random-effect variance or high-dimensional correlated effects.
CHAPTER 6 OF 12 · FIT AND TRANSFER
6. Regression coefficients depend on the scale being reported
Under the mean-preserving form, log μᵢ=xᵢᵀβ+log eᵢ gives a marginal mean model with exposure eᵢ. Then exp(βⱼ) is a marginal mean ratio when σ² is held fixed. Under the more common conditional form log Λᵢ=xᵢᵀβ+bᵢ, the same coefficient is conditional on the random effect.
For a random intercept shared by repeated observations, counts from the same unit become positively associated because they share b. Treating them as independent Poisson–lognormal observations changes both the likelihood and uncertainty.
Report the unit carrying the random effect: person, site, period, batch or another cluster. A label such as “observation-level random effect” is not enough when exposure windows overlap.
CHAPTER 7 OF 12 · FIT AND TRANSFER
7. Changed context: a larger mean amplifies dispersion
Keep σ²=log(1.5) and increase μ from four to eight. The variance becomes 8+64×0.5=40, while quadrature gives P(Y=0)=0.01866. The variance-to-mean ratio rises from three to five even though the log-rate variance is unchanged.
If μ stays four but σ² falls to log(1.2)=0.1823, the variance falls to 7.2 and P(Y=0) to 0.04669. Mean and heterogeneity do different jobs.
The transfer check is therefore two-dimensional: ask whether the mean equation moves correctly and whether the latent spread remains credible. Reusing a dispersion estimate outside the observed predictor or exposure range needs explicit validation.
CHAPTER 8 OF 12 · FIT AND TRANSFER
8. Predictive-tail checks should guide model choice
Compare observed and replicated zeros, central quantiles and prespecified upper-tail exceedances within fitted-mean bands. Use randomized probability-integral-transform residuals for the discrete outcome. Check held-out log scores and interval coverage by site, exposure and time.
Compare Poisson–lognormal predictions with negative-binomial, Poisson–inverse-Gaussian and cluster-count alternatives using identical folds. These models can share a quadratic variance while assigning different probabilities to very large counts.
Inspect posterior or empirical-Bayes random effects only with shrinkage in mind. Extreme fitted effects can reflect a genuine persistent state, one influential count or a misspecified observation process.
CHAPTER 9 OF 12 · RESPECT THE LIMITS
9. Limits and common misconceptions
Overdispersion does not prove a Gaussian log-rate. Omitted covariates, event clustering, serial dependence, measurement error, truncation and structural zeros can all widen a count distribution. The random effect is a modelled latent variable, not a discovered physical measurement.
The lognormal mixture cannot represent underdispersion. A large σ² can also make numerical integration and tail prediction fragile. Boundary inference near σ²=0 needs more care than an ordinary interior Wald interval.
Adding one independent random effect per observation can mimic extra-Poisson variance but does not explain repeated-measure correlation. Conversely, a shared random intercept can create correlation but may miss changing within-unit states.
CHAPTER 10 OF 12 · RESPECT THE LIMITS
10. Evidence, computation and reproducibility
Aitchison and Ho developed the multivariate distribution in The multivariate Poisson-log normal distribution. Lesaffre and Spiessens demonstrated why quadrature order can materially affect mixed-model estimation in On the effect of the number of quadrature points in a logistic random effects model. The numerical warning transfers directly to Poisson random-effects likelihoods.
Archive the exact parameterisation, offset, random-effect level, covariance structure, integration method, quadrature order, optimizer, starting values and convergence diagnostics. Save predictions on both conditional and marginal scales.
Reproduce the μ=4, σ²=log(1.5) moments analytically and the probabilities by two independent integrations, such as high-order quadrature and Monte Carlo simulation.
CHAPTER 11 OF 12 · VERIFY AND CONTINUE
11. Frequently asked questions
Is Poisson–lognormal regression the same as negative-binomial regression?
No. Both can be overdispersed, but they use different latent-rate distributions and give different full predictive tails.
Why subtract σ²/2 from the log rate?
It keeps the marginal mean equal to μ in this article’s parameterisation. Without the correction, exp(xᵀβ) is a conditional median rather than the marginal mean.
How many quadrature points are enough?
There is no universal number. Increase the order until coefficients, likelihood, uncertainty and decision-relevant tail probabilities are stable across the fitted range.
Does a random effect remove the need for covariates?
No. It summarizes remaining heterogeneity. Omitting measured causes can damage interpretation and out-of-sample transfer.
A practical fitting protocol
Define the exposure and random-effect unit. Fit a Poisson baseline, then the Poisson–lognormal model. Increase quadrature order, compare alternative integration methods and evaluate held-out predictive tails. Report where conclusions change.
CHAPTER 12 OF 12 · VERIFY AND CONTINUE
12. Final transfer check and connected reading
Set μ=6 and σ²=log(1.25). Calculate the marginal variance, variance-to-mean ratio and median latent rate. Then explain why the same σ² creates a larger absolute variance contribution when μ grows.
Continue through the Probability root, Poisson–inverse-Gaussian regression, negative-binomial regression, Neyman Type A regression and COM-Poisson regression. Together they turn overdispersion into a testable comparison among latent mechanisms and numerical approximations.
Top · Contents · Previous · How X Works