Find your next learning step
Two hours can have the same planned average arrival rate yet produce very different count variation when the underlying demand level changes from window to window. This guide shows how a Cox process represents that hidden rate variation.
Open the chapter contents · Queueing Theory root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The chapters remain open on the page; these grouped routes are an index.
Route 1 · Build the model
Route 2 · Work the variance
Route 3 · Diagnose the mechanism
Route 4 · Use it in queues
Route 5 · Check and continue
CHAPTER 1 OF 12 · BUILD THE MODEL
1. A random rate is different from a merely changing schedule
A nonhomogeneous Poisson process can have a rate that changes predictably with clock time. Once that rate curve is fixed, a count in any window has Poisson variation: conditional mean and variance are equal. A Cox process adds another layer. Its intensity is itself random, so each day, site or interval may inherit a different latent demand level before individual arrivals are generated.
That distinction matters when repeated one-hour counts vary much more than their average suggests. A scheduled lunch peak is not automatically a Cox process; a latent day-level demand state may be. The model is sometimes called a doubly stochastic Poisson process because arrivals are Poisson only after conditioning on the random intensity.
The question is not “are the counts noisy?” All arrival counts are noisy. It is whether variation in the unobserved rate contributes materially beyond ordinary Poisson variation.
CHAPTER 2 OF 12 · BUILD THE MODEL
2. Conditional Poisson, marginally mixed
Let M be the integrated latent intensity over a chosen window. Conditional on M, the count N follows Poisson(M). The analyst does not usually observe M directly, so the marginal count distribution averages Poisson distributions with different means.
The law of total expectation gives E(N)=E(M). The law of total variance gives Var(N)=E{Var(N|M)}+Var{E(N|M)}=E(M)+Var(M). The first term is arrival randomness at a fixed rate. The second is latent-rate variation across windows.
This identity is the central diagnostic clue. If Var(M)>0, then Var(N)>E(N). Yet overdispersion alone does not prove a Cox mechanism; clustering, batches, retries, omitted exposure and serial dependence can produce similar count summaries.
CHAPTER 3 OF 12 · WORK THE VARIANCE
3. Work a two-state latent demand example
Suppose each hour is quietly in one of two unobserved demand states. With probability 0.75 the rate is 4 arrivals per hour; with probability 0.25 it is 12. Conditional on the state, arrivals are Poisson.
The marginal mean is 0.75×4+0.25×12=6. The latent-rate variance is 0.75(4−6)²+0.25(12−6)²=12. Therefore the count variance is 6+12=18, three times its mean. A plain Poisson model with mean 6 would report variance 6 and materially understate repeated-hour variability.
| Component | Calculation | Result |
|---|---|---|
| Mean count | 0.75×4+0.25×12 | 6 |
| Poisson contribution | E(M) | 6 |
| Latent-rate contribution | Var(M) | 12 |
| Total count variance | 6+12 | 18 |
CHAPTER 4 OF 12 · WORK THE VARIANCE
4. The full mixture changes tail probabilities, not only variance
The two-state model has probability P(N=n)=0.75e⁻⁴4ⁿ/n!+0.25e⁻¹²12ⁿ/n!. It is not Poisson(6). Its distribution places extra mass near the two component regimes and often more probability in the upper tail relevant to staffing.
For example, a manager interested in unusually crowded hours should calculate the mixture tail, not insert the average rate into a single Poisson formula. “Six on average” hides whether six is typical or merely lies between quiet and busy states.
The same principle extends to a continuously varying random intensity. If the integrated intensity follows a gamma distribution, the mixed count is negative binomial under a standard parameterisation. That convenient result is a modelling option, not evidence that every negative-binomial fit represents one physical latent mechanism.
CHAPTER 5 OF 12 · WORK THE VARIANCE
5. Changed context: exposure can mimic or remove overdispersion
Suppose the observation windows have unequal lengths: half last 30 minutes and half last 90 minutes, but the analyst treats them all as one hour. A constant rate of 6 per hour then gives conditional means 3 and 9. The uncorrected counts look overdispersed because exposure, not demand, varies.
If exposure duration T is known, model the integrated intensity as rate×T or include log(T) as an offset in a count model. After that correction, remaining excess variance may shrink sharply. The changed context teaches a practical transfer rule: before inventing a latent process, make the observation opportunity comparable.
Transfer check: repeat the two-state example with 30-minute windows. The conditional means become 2 and 6, the marginal mean becomes 3, and the variance becomes 3+3=6. Both the mean and the latent contribution change with window length.
CHAPTER 6 OF 12 · DIAGNOSE THE MECHANISM
6. Estimate dispersion across genuinely repeated windows
Use windows that share a defensible population, exposure rule and measurement system. Calculate the sample mean and variance of counts, but also inspect their relationship across time, weekday, site and known operating regime.
| Pattern in repeated counts | Plausible explanation | Evidence check |
|---|---|---|
| Variance rises with window length | Persistent latent rate or serial dependence | Plot variance-to-mean ratio by aggregation scale |
| High counts cluster by day | Shared day-level demand state | Add day covariates or a random effect |
| Excess only at zero | Closures, outages or structural zeros | Audit exposure and collection logs |
| Short-lag bursts | Batches, retries or self-excitation | Inspect timestamps and event identity |
A single variance-to-mean ratio loses order and context. Report uncertainty, window count and sensitivity to influential days.
CHAPTER 7 OF 12 · DIAGNOSE THE MECHANISM
7. Separate a Cox process from self-excitation
In a Cox model, a latent environment changes the rate and thereby makes nearby counts associated. In a self-exciting process, an event itself temporarily raises the future event rate. Both may show clustering.
The distinction is causal and operational. Marketing exposure, weather or an unobserved incident load can act as a shared latent driver. Retries, contagion or follow-up calls may be triggered by preceding events. Conditioning on relevant covariates may reduce Cox-type clustering; examining event-triggered lag patterns may reveal self-excitation.
Neither model should be chosen from a count histogram alone. Compare predictive performance on later windows, residual dependence, identifiable mechanisms and whether the fitted structure remains stable when window size changes.
CHAPTER 8 OF 12 · DIAGNOSE THE MECHANISM
8. Latent-rate variation can have more than one scale
A call centre may have a day-level demand state, an hour-level local fluctuation and a known time-of-day curve simultaneously. One random multiplier cannot automatically represent all three. A hierarchical intensity can separate predictable structure from latent variation at chosen scales.
Define the scale before interpreting the variance. A day-level random effect induces association among hours on the same day. A rapidly varying stochastic intensity may affect neighbouring minutes instead. Aggregating to daily totals can conceal one pattern and exaggerate another.
Also distinguish heterogeneity across independent sources from temporal variation within one source. Pooling sites with different stable rates creates a mixed count distribution even if each site is ordinary Poisson.
CHAPTER 9 OF 12 · USE IT IN QUEUES
9. Queue performance depends on arrival correlation, not only count variance
Replacing Poisson arrivals with a Cox input can increase the chance of sustained overload because high-rate periods arrive in clusters. Two models with the same hourly count mean and variance can still create different waits if one concentrates demand into long regimes and the other into short fluctuations.
Queue simulation should preserve the fitted time scale and dependence, service distribution, abandonment, capacity changes and initial backlog. Feeding independent negative-binomial counts into each interval may reproduce marginal dispersion while destroying persistence across intervals.
Report which features the arrival model reproduces: count distribution, autocorrelation, peak duration and event timing. Then test whether the staffing conclusion changes across plausible alternatives.
CHAPTER 10 OF 12 · USE IT IN QUEUES
10. What this model cannot establish
A good Cox-process fit does not reveal the hidden variable automatically. “Latent demand” is a statistical description until linked to measured conditions. Nor does overdispersion imply that every individual arrival is dependent after conditioning on the correct intensity.
Small samples make variance estimates unstable. Timestamp rounding, duplicate events and changing definitions can manufacture dispersion. A flexible latent process may absorb misspecification in seasonality or exposure and still predict poorly outside the fitted period.
Use the model to express uncertainty and dependence honestly, not to turn an unexplained residual into a fictional cause.
CHAPTER 11 OF 12 · CHECK AND CONTINUE
11. Questions readers should ask
Does variance greater than the mean prove a Cox process?
No. It is compatible with latent-rate mixing, but batches, self-excitation, unequal exposure, structural zeros and data errors can also produce it.
Why condition on the intensity?
Conditional modelling separates ordinary Poisson arrival randomness from uncertainty about the rate environment. Marginalising combines both sources.
Can a deterministic busy-hour curve be part of the model?
Yes. The latent intensity can include a predictable baseline multiplied or augmented by stochastic variation, provided it remains nonnegative and identifiable enough for the task.
Should I use smaller windows?
Several window scales are informative. Very small windows are sparse; very large ones can hide local persistence. Choose scales connected to operations and test sensitivity.
CHAPTER 12 OF 12 · CHECK AND CONTINUE
12. Continue through the verified queueing ecosystem
Begin with the Queueing Theory root, the Gold Standard of Education and the How X Works hub. For deterministic variation and model checking, read Time-Varying Arrivals and Capacity That Arrives Too Late and Time-Rescaling Diagnostics for Nonhomogeneous Poisson Arrivals.
For alternative sources of extra variation, use Batch Arrivals, Event Rates and Job Counts and Retries, Attempt Rates and Unique Jobs. For observation bias, continue to Unfinished Jobs, Right Censoring and the Service-Time Mean.
Authoritative technical reading: the open-access Nature Communications treatment of Cox-process representation defines the random-intensity construction, while the peer-reviewed Cox-process inference article states how stochastic intensity generates overdispersion. Use these sources for the mathematics, then validate the mechanism against the actual arrival system.
Top · Contents · Previous · How X Works