Find the question you need
Why averages fail · Build the two samples · Percentiles · SLO deadline · Composed requests · Full chapter index
Two services can report an identical mean response time yet expose users to very different high-percentile latency. This guide uses fully defined request counts, percentile conventions and thresholds.
Chapter index
Advanced companion: distributions and composed requests
Chapters 1–4
1. The two services whose averages look identical · 2. Define latency and its observation boundary · 3. Construct Service A with a constant 100 ms · 4. Construct Service B with a long tail
Chapters 5–8
5. Verify the means with independent arithmetic · 6. Order the observations to calculate percentiles · 7. Compare the ninetieth and ninety-ninth percentiles · 8. Calculate a deadline success rate
Chapters 9–12
9. Quantify the distribution’s variance · 10. A chain of requests can amplify tail exposure · 11. Distinguish sequential from parallel dependencies · 12. Do not silently exclude failed or timed-out requests
Chapters 13–16
13. Request weights and user weights need not agree · 14. The practical monitoring plan · 15. Frequently asked questions and tests · 16. Connected telecommunications and reliability reading
Imagine two fictional telecommunications-supported services, each with one hundred completed request latency observations. A summary dashboard reports an average end-to-end response time of 100 milliseconds for both. On that one statistic, they appear equivalent.
But Service A has exactly 100 ms latency on every one of its hundred requests. Service B finishes ninety requests in 50 ms and ten requests in 550 ms. The mean is also 100 ms, yet a minority of users experiences a substantially longer wait.
The hidden issue is distribution shape. A mean aggregates total waiting time and number of observations; it does not specify how delay is shared among requests. Service-level decisions often need high-percentile latency, deadline violations and an accurate account of requests that never completed.
Our numbers are original invented measurements for a transparent arithmetic example. They are not a live performance result for a real telecommunications carrier or an assurance that any service should meet a particular numerical threshold.
A latency record should say what started and stopped the clock. Possible measures include round-trip network time, time to first response byte, application request completion or end-to-end user-visible rendering. Those are not interchangeable.
For this example, every recorded latency is the elapsed time in milliseconds from a comparable request’s defined start to its successful completion. Each of the hundred records is given equal weight in a request-level distribution.
We initially assume that all hundred attempted requests completed and were observed. We later explain what changes when timeouts, cancellations or failed requests are missing. A completed-request average alone can look artificially good if it excludes the slowest unsuccessful attempts.
Both systems are compared across the same one hundred-request count and consistent metric. Their operating environments, load mix and task complexity are deliberately idealised so that only the timing distribution differs.
Contents · Previous chapter · Next chapter
Service A returns exactly one hundred completed requests, and every observed request takes 100 ms. The sum of the measured delays is therefore 100×100=10,000 milliseconds.
Dividing by the request count gives the arithmetic mean 10,000/100=100 ms. The median is also 100 ms, as are the 90th, 95th and 99th percentiles under common empirical percentile definitions for this constant sample.
Its observed maximum is 100 ms and its population-style variance across these hundred values is zero. This deterministic table is a classroom construction, not a realistic claim that a physical service has no timing jitter.
The purpose is to create a narrow baseline against which another latency distribution can be compared. Equal average delay will not be enough to make their other properties equal.
Contents · Previous chapter · Next chapter
Service B completes ninety of its hundred requests in 50 ms each. The other ten completed requests take 550 ms each. The total latency sum is ninety times fifty plus ten times 550, or 4,500+5,500=10,000 milliseconds.
The mean is therefore 10,000/100=100 ms, precisely matching Service A. The mean equality is exact for the invented sample rather than the result of rounding two noisy averages to the same displayed number.
However, a randomly chosen one of the one hundred completed records has a 90 percent chance of showing 50 ms and a 10 percent chance of showing 550 ms under the empirical distribution. This describes a very different user-experience pattern.
A dashboard that reports only ‘average 100 ms’ discards the ninety fast completions and ten unusually delayed completions as separate groups. That information loss can conceal deadline violations or an operational bottleneck affecting a minority of requests.
Contents · Previous chapter · Next chapter
Service A’s hundred values each equal the mean of 100, so their total is 10,000 ms. Service B’s total is also 10,000 ms, because the ten slow requests contribute 5,500 ms and the ninety fast contribute 4,500 ms.
The ten slow Service B requests therefore consume 55 percent of its cumulative observed waiting time despite representing only ten percent of the request count. That disproportionate contribution is invisible once the record is reduced to a single mean.
This is an important descriptive distinction, but it is not a statement that Service B wasted 55 percent of server capacity. Client elapsed time can include waiting, network transit, queuing and work; it is not automatically additive resource consumption in a concurrent system.
The mean computation is valid. The interpretation fails only when the mean is treated as a substitute for quantiles, deadline probabilities or a causal breakdown of the latency components.
Contents · Previous chapter · Next chapter
For an empirical nearest-rank percentile, sort n observations and select the value at position ceil(pn), where p is expressed between zero and one and positions start at one. Other software percentile conventions interpolate, so the definition should be declared.
In Service A, all sorted values are 100 ms, so every empirical percentile under this method equals 100 ms. In Service B, positions 1 through 90 are 50 ms and positions 91 through 100 are 550 ms.
Therefore Service B’s nearest-rank median and 90th percentile are 50 ms, while its 95th and 99th percentiles are 550 ms. Its typical request is faster than A’s, but its slowest ten percent are much slower.
The Google Site Reliability Engineering guide to service-level objectives explains why averaging latency can hide long tails and why percentile-based indicators can reveal the experience of slower requests.
Contents · Previous chapter · Next chapter
Service A has empirical p90=100 ms and p99=100 ms. Service B has p90=50 ms but p99=550 ms under the declared nearest-rank definition. The services have not only different maxima but different positions of their distribution mass.
Calling Service B ‘faster at the median’ is correct. Calling it ‘faster for all users’ would be false. A high-percentile comparison reveals the ten slow requests that an average or median would conceal.
For a sample of only one hundred observations, p99 is a sensitive extreme statistic. In a real service, many observations and uncertainty monitoring may be needed to estimate a stable 99th-percentile trend. This example uses exact invented counts to demonstrate arithmetic, not estimation precision.
A performance report should identify the observation window, traffic mix, percentile convention and sample count. A raw p99 value without those details can be misleading, particularly in a service with sparse traffic or bursty requests.
| Quantity | Service A | Service B |
|---|---|---|
| Request observations | 100 | 100 |
| Latency distribution | 100 at 100 ms | 90 at 50 ms; 10 at 550 ms |
| Arithmetic mean | 100 ms | 100 ms |
| Median | 100 ms | 50 ms |
| Nearest-rank p95 | 100 ms | 550 ms |
| Nearest-rank p99 | 100 ms | 550 ms |
| At or below 200 ms | 100% | 90% |
| Descriptive SD | 0 ms | 150 ms |
Contents · Previous chapter · Next chapter
Suppose a fictional service-level objective requires at least 99 percent of requests to complete within 200 ms. Under the specified inclusive threshold, Service A has 100 of 100 observations at or below 200 ms and meets the threshold in this sample.
Service B has ninety completions at 50 ms and ten at 550 ms. Only ninety of one hundred are at or below 200 ms. Its observed deadline success rate is 90 percent, so it fails the hypothetical 99-percent requirement despite sharing the same 100 ms arithmetic mean as A.
This illustrates why a mean alone cannot tell whether an explicit deadline-based SLO was met. A meaningful SLO requires a declared eligible-request denominator and a definition for successes, failures and timeouts.
The 200 ms target is invented for teaching, not a general telecommunications or accessibility standard. Different workloads and user groups may have different acceptable thresholds.
Contents · Previous chapter · Next chapter
Service A has sample values all equal to its mean, so the squared deviations from 100 are zero. Its descriptive variance and standard deviation across these values are zero.
For Service B, the ninety 50 ms observations each differ from the mean by −50 ms and the ten 550 ms observations differ by +450 ms. Using a denominator of one hundred, its descriptive variance is [90×50²+10×450²]/100=22,500 ms².
The corresponding standard deviation is sqrt(22,500)=150 ms. We state the denominator because an unbiased sample variance estimate would use n−1 and yield a slightly different figure.
Equal means can therefore coexist with very different dispersion. A variance is useful, but it still does not identify a specific high percentile or the full shape of a multimodal distribution.
Contents · Previous chapter · Next chapter
Suppose a hypothetical user action requires ten sequential requests drawn independently from the empirical Service B distribution. The probability that a single request is not one of the 550 ms slow outcomes is 0.9.
Under that independence assumption, the probability all ten avoid a slow result is 0.9¹⁰≈0.348678. Therefore the probability at least one request experiences 550 ms is 1−0.9¹⁰≈65.13 percent.
The expected sum of ten individual latencies is still ten times 100 ms, or 1,000 ms, by linearity of expectation. The average-chain duration alone does not show how frequently a user action contains a conspicuously slow component.
Real latency events can be correlated through shared congestion, backend failures and traffic bursts, invalidating simple independent multiplication. The example is a conditional calculation, not a prediction that exactly 65 percent of real user sessions encounter the slow tail.
Contents · Previous chapter · Next chapter
Ten sequential requests add their elapsed times to a workflow under a simple no-overlap model. A parallel fan-out that waits for all ten responses is instead governed by the maximum response time among the requests, plus orchestration effects.
Under independent draws from our two-point Service B distribution, the chance a ten-way fan-out contains at least one 550 ms response is the same 1−0.9¹⁰. If the workflow must wait for every response, that slow component can determine the user-visible completion time.
But actual fan-out operations may allow cancellation, fallbacks or partial results, and request latencies are often correlated. A model that assumes independence or waits-for-all semantics must make those assumptions explicit.
Service-level engineering needs the structure of the user journey, not merely an aggregate backend mean. The Google SRE guidance on monitoring distributed systems discusses why latency distributions and long-tail request behaviour can affect composed services.
Contents · Previous chapter · Next chapter
Our initial tables contain only successfully completed requests and deliberately no timeouts. A real monitoring pipeline that logs only completed successful transactions can omit the slowest and most consequential requests if those eventually time out or are cancelled.
For example, if a client times out after 500 ms, a 550 ms task may never appear as an ordinary successful 550 ms completion. Averaging only completed responses would shift the data set and potentially make the service look better while more users receive errors.
A meaningful service-level indicator needs rules for counting attempted requests, errors, cancellations, retries and timeouts. It should specify what happens when a request never has a recorded completion time.
A correct average over an incomplete denominator can be misleading about the user’s experience. Monitoring must preserve enough event status data to separate latency, availability and failures.
Contents · Previous chapter · Next chapter
Our empirical sample weights each request equally. A user who submits many requests contributes more entries than a user who submits one. Therefore a request-weighted latency statistic need not represent an average over distinct users.
Some workflows may contain a small number of high-cost requests and many lightweight requests. A single aggregate percentile can hide differences by operation, region, client network or device class if those populations are mixed indiscriminately.
A trustworthy performance review can segment by meaningful endpoints and disclose whether the metric is request-weighted, session-weighted or user-experience-weighted. The appropriate choice depends on the decision being made.
The general principle is to name the statistical population behind the number. Equal ‘mean latency’ values cannot be interpreted reliably if the two means describe different traffic mixes or different completion boundaries.
Contents · Previous chapter · Next chapter
Begin with a clear latency definition: request start, finish, eligible population, and treatment of cancelled or failed requests. Select a sampling interval and instrument the service without imposing excessive measurement overhead or losing rare slow events.
Record distributions or latency histograms rather than only arithmetic averages. Preserve enough resolution to estimate p50, p95 and p99 and to count deadline violations, while acknowledging instability when sample counts are small.
Connect the monitored metrics to the user journey. A user action that waits for multiple sequential or parallel requests may have a tail experience quite unlike any one backend’s typical request.
Finally investigate causes only after the measurement is sound. Queuing, server work, network latency, retries and client-side delays require separate tracing. A histogram identifies a tail; it does not automatically prove its cause.
Contents · Previous chapter · Next chapter
Does lower median latency prove a better service? Not in every sense. Service B has a lower median but worse high percentiles and many failures against the hypothetical 200 ms deadline.
Can two systems have equal means and unequal p99? Yes; the hundred-request tables here are an exact counterexample. Does a p99 of 550 ms mean one percent of requests are slower than 550 ms? No; percentile wording depends on a rank convention and sample distribution, including ties.
Practice: change Service B to ninety-five requests at 60 ms and five requests at 860 ms. Verify its mean, median and p99. Determine whether the hypothetical 99-percent-within-200-ms objective is met.
Practice: introduce two timeouts and decide how your service-level indicator will count them. State whether the latency distribution includes completed requests only and whether a separate availability indicator includes all eligible attempts.
Contents · Previous chapter · Next chapter
The exact teaching result is that both services average 100 ms across one hundred completed requests, while Service A has p99=100 ms and Service B has p99=550 ms. A single mean does not determine the experience of slow requests.
Read The Gold Standard of Telecommunications, The Gold Standard of Systems Reliability, and Equal Means and Different Queueing Waits for connected system concerns.
External foundations include Google SRE: Service Level Objectives and Monitoring Distributed Systems. These describe performance measurement principles; the latency values here are invented teaching observations, not Google service measurements.
Continue through the Gold Standard hub and How X Works. A useful latency report says who waited, how long, which requests were counted and how much of the distribution lay beyond the acceptable threshold.
Advanced worked companion: long-tail distribution and user journeys
Calculate an entire ten-request sequential distribution
Return to the invented Service B, with an independent request taking 50 ms with probability 0.9 or 550 ms with probability 0.1. Let K be the number of slow requests among ten sequential independent draws. Under those assumptions K follows a binomial distribution with ten trials and slow-event probability 0.1.
Each fast request contributes 50 ms. A slow request contributes an additional 500 ms above that fast baseline. Thus ten sequential request times sum to T=10×50+500K=500+500K milliseconds. This representation makes the effect of one or more tail events directly auditable.
The expected number of slow requests is E[K]=10×0.1=1, so E[T]=500+500×1=1,000 ms. That matches ten requests times the 100 ms marginal mean. Linearity of expectation therefore gives the same average-chain duration even though the distribution of completion times can be highly skewed.
The variance of K is 10×0.1×0.9=0.9. Consequently Var(T)=500²×0.9=225,000 ms² and SD(T)=sqrt(225,000)≈474.34 ms. A user journey with the same expected total time can therefore experience substantial variation when it depends on independent long-tail requests.
Find the chance of crossing a multi-request deadline
The chance no slow request appears is P(K=0)=0.9¹⁰≈0.348678. The chance at least one slow request appears is 1−0.9¹⁰≈0.651322, or 65.13%. When K≥1, the sequential total is at least 1,000 ms; when K=0, it is 500 ms.
If a fictional workflow deadline requires completion strictly below 1,500 ms, the workflow misses that strict deadline when K≥2 because T=500+500K. The probability of at least two slow requests is 1−P(K=0)−P(K=1)=1−0.9¹⁰−10×0.1×0.9⁹≈26.39%.
The inequality convention matters. At exactly 1,500 ms, a workflow with K=2 is on the boundary. A requirement of at most 1,500 ms differs from one requiring strictly less than 1,500 ms, and both differ from a per-request 200 ms objective.
All these probabilities rely on independent latency draws. In real systems a traffic burst, shared queue, common backend fault or congested link can cause correlated slow requests. Such dependence may increase the chance of several slow requests together or create a different pattern. Reusing the independent binomial calculation without checking the assumption would overstate the evidence.
Why the 99th percentile needs a definition and a sample count
Under the nearest-rank convention with n=100, the 99th percentile is the 99th sorted observation. For Service B, positions 91 to 100 are all 550 ms, so its nearest-rank p99 is 550 ms. A software package using interpolation may report different values for some percentile definitions and sample configurations.
Now consider a new hypothetical sample of one thousand requests: 990 at 90 ms and ten at 1,090 ms. Its mean is [990×90+10×1,090]/1,000=100 ms. But under nearest-rank order statistics, position 990 is still 90 ms, so p99=90 ms, whereas a higher percentile such as p99.9 corresponds to a tail request.
This illustrates a different limitation: one chosen percentile does not capture the entire distribution. An SLO defined at p99 can conceal the slowest one percent when the sample has a sharp tail boundary. A high percentile is useful only with a clear aggregation window, count, convention and complementary indicators for errors or timeouts.
Percentiles are not linear averages. Averaging daily p99 values does not generally produce the p99 of all requests across the combined days. A complete aggregate should preserve request counts and sufficiently detailed histogram information, then calculate the percentile for the intended combined population.
Avoid the denominator traps of performance reports
A completed-request latency sample may exclude requests that timed out or were cancelled. A request-attempt SLO may instead count all eligible attempts and treat timeout outcomes as failures. The choice should be stated explicitly so a reported improvement in mean latency does not merely reflect dropping the slowest requests from the data.
Similarly, request-level averages are not user-level averages. One automated client issuing thousands of small requests can dominate a global request count, while a smaller number of ordinary users perform complicated multi-request tasks. The observed service population must match the user experience the metric is meant to represent.
Consider segmenting by endpoint and region to understand whether one particular operation produces the long tail. The 90-fast/10-slow example does not identify the cause of the slow requests, and it would be improper to infer congestion, server processing or customer geography without diagnostic evidence.
A useful SLO record reports the start and stop events, request eligibility, observation period, successful completions, timeouts, failure counts, relevant percentiles and the sample population. These details turn a headline number into a reproducible measurement specification.
An operational checklist for measuring the long tail
First validate timestamps, clock synchronisation and aggregation windows. Then capture histogram buckets with boundaries fine enough to distinguish the expected service target from the unacceptable tail. Ensure the slowest requests remain counted even when their result status differs from a successful response.
Next compare mean, median, p95, p99 and deadline-violation proportions without pretending that any one statistic completely characterises the distribution. Assess the number of eligible requests behind each percentile and be cautious about high-percentile estimates in tiny samples.
Finally investigate the performance path rather than guessing from the dashboard. Network transit, service queuing, retries, storage, downstream dependencies and client rendering can contribute separately. The relevant user-visible SLO may include several of these stages.
The Google SRE guidance on SLOs and distributed-systems monitoring provide the underlying measurement principles. Our request counts, latency levels and ten-request chain are mathematical teaching examples, not operational claims about any provider.