How Superintelligence Works | Latency and Throughput — Queues, Tail Delays and Real User Experience

Latency is how long a specified AI task takes; throughput is how much work a service completes per unit time. Neither figure tells the whole story alone. An inference engine can serve many requests while some users wait in long queues, and a fast first token is not proof that the full checked answer has arrived.

This chapter uses an original ten-request timing sample to calculate mean, median, nearest-rank p90 and p95, then compares a fictional streamed answer’s first-token time with its complete output. Changed-context exercises test a skewed latency distribution and a service whose arrivals exceed sustainable processing capacity.

It fills topic 085 of How Super Intelligence Works. The series is an educational account of current AI system mechanisms, not a claim that existing systems are artificial superintelligence.

Contents and fast learning routes

Start with latency · Read the ten observations · Check percentiles · Practise transfer · Sources

1. Latency belongs to a specific user and a precise interval

Latency means elapsed time between a defined beginning and end. For an AI application, that could be time from the user’s click until the first visible token, until the last token, until a file is stored, or until a consequential change is verified. Those are different intervals. A claim of ‘fast AI’ is not testable until the measurement boundary is clear.

An inference server can complete numerical execution while a retrieval step, network transfer or human approval remains unfinished. Measuring the inner computation does not measure the completed user job. A realistic timing diagram shows arrival, admission, queue wait, preprocessing, model operation, delivery and any independent checking.

For education, imagine a student asking for a verified Mathematics solution. The first word appearing may be reassuring, but the useful outcome is a complete, correct explanation that supports independent learning. Speed is valuable; speed without the required quality or final result is a misleading optimisation target.

Back to contents

2. Throughput counts work over time

Throughput is a quantity of completed work divided by elapsed time. Depending on the system, it may be completed requests per second, output tokens per second, records processed per hour or other meaningful units. Do not divide an unrelated count by a convenient interval and label it model capacity.

Two requests can differ dramatically in cost. One might classify a short sentence, another may process a long report and generate pages of feedback. A service that handles ten short requests per second may handle far fewer long ones. Aggregate request counts can obscure input and output token volumes.

Use output tokens per second when a generating workload makes that measure helpful, but still record task accuracy, time to first token and completion time. The fastest service by one metric may be slower or less useful for a particular reader.

Back to contents

3. The fictional Study Desk timing packet

Our central teaching fixture is a fictional study-guide service called Study Desk. It completes ten requests during a two-second observed window in an invented measurement record. Their end-to-end completion latencies in milliseconds, ordered from smallest to largest, are 120, 130, 140, 150, 160, 170, 180, 190, 400 and 700.

These figures are not real performance data from a device or provider. They are printed so a learner can recompute the mean, median and selected tail percentiles and understand why a single average can hide slow user experiences.

The service’s completion count and each request’s measured latency refer to different dimensions. We will calculate both without claiming that one determines the other. The window records ten completed requests; it does not tell us how many were waiting at the end or what service capacity exists under continuous load.

Back to contents

4. Compute mean latency from all ten requests

The first eight entries form an arithmetic sequence from 120 to 190 in ten-millisecond increments. Their sum is 8×(120+190)/2 = 1,240 milliseconds. Add the two long cases: 400+700 = 1,100. The total across all ten requests is 2,340 milliseconds.

Mean latency is 2,340/10 = 234 milliseconds. This is the exact arithmetic mean of the invented sample. It does not imply that every user waited approximately 234 milliseconds, and it is not necessarily the mean of the next hundred requests.

A report should identify the number of requests and the time interval. A mean reported with three decimal places would not become more representative simply because it contains more digits.

Back to contents

5. Compute the median and see the different story

The ten latencies are sorted. With an even number of observations, the conventional sample median is the average of the fifth and sixth values: (160+170)/2 = 165 milliseconds. Half the observed requests are at or below 160 and half at or above 170.

The mean of 234 milliseconds is considerably higher than the median of 165 because the 400- and 700-millisecond cases pull the average upward. Neither statistic is wrong. They describe different aspects of the distribution.

Students can learn a general mathematical habit from this: describe the distribution before interpreting its centre. An apparently satisfactory mean can conceal a slow tail, while an extreme maximum may exaggerate what most users experience.

Back to contents

6. Define percentiles before making a tail claim

Percentiles can be calculated using different statistical conventions, especially for small samples. For this exercise we explicitly use the nearest-rank method: sort n observations and select the value at position ceil(p×n), where p is the fraction such as 0.90.

At p=0.90 with ten observations, the selected position is ceil(9)=9, giving a nearest-rank p90 of 400 milliseconds. At p=0.95, ceil(9.5)=10, giving a nearest-rank p95 of 700 milliseconds. A tool using interpolation may return different sample percentile estimates.

The computation method must be labelled in the report. A p95 number without population, observation window and convention is not precise evidence of a guaranteed response threshold. On small samples, one unusual request can dominate these tail estimates.

Back to contents

7. Calculate throughput without inventing the arrival pattern

Ten completed requests over the stated two-second observation window yield 10/2 = 5 completed requests per second. That is the observed-within-fixture completion rate for the interval. It is not necessarily the machine’s maximum sustained service rate.

The packet does not say whether twenty requests were submitted and ten remained queued, whether users arrived evenly or whether the service was idle part of the time. The same completion count could correspond to very different operational situations.

For a real service, record arrivals, completions, failures and waiting requests alongside the workload mix. Only then can an operator distinguish low demand from insufficient capacity.

Back to contents

8. Mean, median and tail need a common interpretation

The invented Study Desk summary is therefore: ten completed requests, five completions per second over two seconds, mean completion latency 234 ms, median 165 ms, nearest-rank p90 400 ms and p95 700 ms. These values belong together with the method and sample size.

A misleading report might publish ‘average response 165 ms’ by relabelling the median as the mean, or ‘95% of all future users receive an answer within 700 ms’ by turning one sample percentile into an absolute promise.

An accurate explanation preserves the measured result and its limitations. The service can be investigated for its slow cases without pretending that ten observations prove a stable worldwide performance guarantee.

Back to contents

9. Time to first token and total completion

Text-generation systems may stream output. Time to first token, often called TTFT, measures how long the first generated token takes to become available under the tool’s stated convention. Time per output token and total request latency describe different parts of the experience.

Consider a fictional streamed response whose first token arrives at 300 ms and whose next token arrives every 40 ms, for a total of twenty tokens. There are nineteen intervals after the first token, so the final token appears at 300+19×40 = 1,060 ms.

Calling the response ‘300 milliseconds’ because the first token appeared then would be misleading for a reader who needs the complete answer. The difference becomes larger for long generated documents.

Back to contents

10. Changed output length, unchanged first-token time

Keep TTFT at 300 ms and each later token interval at 40 ms, but ask for fifty tokens rather than twenty. Now there are forty-nine intervals after the first token. Completion occurs at 300+49×40 = 2,260 milliseconds.

The first-token metric did not change, yet the total completion time more than doubled. This illustrates why the number of generated tokens matters to user experience and cost. Real inter-token timings vary, so the constant-interval model is a teaching assumption only.

An honest performance card includes the input and output length distribution. A comparison where one model writes a short paragraph and another writes a long checked explanation cannot be interpreted from total response time alone.

Back to contents

11. The distinction between prefill and decode

For many transformer-based text systems, prefill processes incoming context, while decode repeatedly generates output tokens using the relevant retained attention state. Their computational profiles can differ. Long input documents can affect time before output begins; long responses affect the generation phase.

A service might handle short conversation turns well but struggle with long uploaded material because of context processing or memory limits. Another configuration might respond promptly with a first token but generate long passages slowly.

Google Cloud’s inference metrics documentation distinguishes TTFT, time per output token, total request latency and output-token throughput. These definitions are useful anchors for interpreting a real measurement.

Back to contents

12. A queue is where latency can grow without slower computation

Suppose two identical workers each finish a request in 100 ms. With continuous full utilisation and zero overhead, the simplified service capacity is 20 requests per second. If thirty requests arrive every second for an extended period while the service can complete only twenty, the waiting backlog grows.

Under that simplified deterministic rate difference, ten net requests per second accumulate. After five seconds from an empty queue, the backlog could be fifty requests if no admission limits, cancellations or service changes intervene. That is an illustrative capacity calculation, not a forecast for a real inference engine.

The model code has not slowed. User latency grows because the queue contains more work than the workers can drain. Adding an accelerator might help if execution capacity is the actual constraint, but not if the delay is in an external database or shared approval stage.

Back to contents

13. Queue length is not the same as latency

A queue contains pending jobs; its length is a count at a given moment. Latency is elapsed time for an individual job. They are related through arrival and service processes but cannot be equated without more information.

A queue of twenty tasks could drain quickly with many independent workers or slowly with one overloaded device. Long requests may block shorter ones under some scheduling policies. A dashboard needs queue depth, wait-time distribution and workload composition rather than a single unqualified queue indicator.

When measuring a queue, specify whether requests executing on workers are included in the count. Some metrics report pending work, others report total in-system work. That distinction matters when applying queueing formulas.

Back to contents

14. Little’s law with the required conditions

Little’s law relates the long-run average number of jobs in a stable system, the long-run average arrival or completion rate and the average time in that system: L = λW. It is powerful because it applies under broad conditions when the underlying system is appropriately stable and measurement boundaries align.

For a fictional stable service completing six requests per second and holding an average end-to-end in-system time of 0.25 second, the average number in the system is 6×0.25 = 1.5 jobs. A fractional average is normal: it describes a time average rather than a snapshot of half a request.

Do not use this relation to legitimise a system whose backlog grows without bound. Stable conditions and coherent averaging intervals matter. The earlier overloaded example with thirty arrivals and twenty completions per second cannot sustain a finite steady-state waiting average indefinitely.

Back to contents

15. Utilisation and variability both matter

A service operating near full resource capacity has little slack to absorb bursts. Even if its mean arrival rate remains below mean service capacity, variability can create occasional long waits. The arrival distribution and service-time distribution influence the tail.

Two systems with the same average workload can behave differently when one receives requests evenly and the other receives concentrated bursts. Likewise, two models may have the same average generation length but very different proportions of exceptionally long responses.

Evaluation should therefore include representative arrival patterns and difficult length outliers. An average-rate calculation is a useful starting point, not a guarantee that no user will experience delay.

Back to contents

16. Batching trades one kind of time for another

Combining requests into batches can improve hardware utilisation, reducing average computation cost per item. But waiting to accumulate a batch can delay the earliest request. NVIDIA’s Triton batcher documentation discusses queue delays, batch size and scheduling controls.

A large batch might increase output throughput while worsening time to first result for users whose requests arrive just after the previous batch. Continuous batching can help certain autoregressive workloads, yet memory and scheduling limits remain.

Choose the batching policy from the service’s actual deadline and expected load. A grade-appropriate tutoring chat may prefer responsiveness; an overnight document-processing job may accept longer individual waits for greater throughput.

Back to contents

17. Tail latency matters to real people

A percentile is a description of a distribution under a specified method, not a personal promise to any future user. However, high-percentile latency can identify experiences that a low mean conceals. A few very slow requests can matter disproportionately when pupils or staff face fixed schedules.

Look at which tasks occupy the tail: long uploads, cold starts, retries, large output jobs, overloaded queues or unavailable tool dependencies. Grouping slow cases by cause is more useful than simply celebrating a lower p50.

Testing only short requests in ideal conditions can create a falsely reassuring number. A full service study includes representative tasks, interruptions and clear measurement windows.

Back to contents

18. Correctness and timing must be measured together

A service that returns an unsupported answer instantly has not delivered a useful completion for a high-stakes user. A long answer that is accurate but arrives after a time-critical deadline may also fail the intended task.

Define a joint success criterion such as ‘correct, supported answer delivered within the stated response target’. Record timeouts, refusals and incomplete responses explicitly instead of excluding them from the sample as inconvenient observations.

Performance optimisation can change model quality through reduced context, numerical precision or aggressive time limits. Test the effect of any change on both output usefulness and user latency.

Back to contents

19. Cold start, caching and warm performance

A newly started model service can incur image loading, weight transfer and memory-allocation delays. A warm service may answer repeated requests faster. Prefix or other caches can improve some workloads by reusing compatible computation or data, but cached content has its own validity and isolation requirements.

A benchmark performed only on warm, repeated prompts does not describe every cold or unfamiliar request. Similarly, a newly deployed model may have different latency for its first calls than during steady operation.

Document the test conditions, including whether the model was loaded, what caches existed, and whether prompts were unique. Avoid comparing a cached run of one system against a cold run of another while presenting the result as a fair hardware comparison.

Back to contents

20. Multi-tenant services and shared resources

A school might share an inference service with many unrelated applications. Heavy activity from another tenant can affect queues, resource access and tail latency unless the system isolates or allocates capacity appropriately. Performance is therefore not always a property of the model alone.

Security matters as much as scheduling: one user’s context or cached state must not leak into another user’s answer. Batch efficiency is never permission to merge private request ownership or cross authorisation boundaries.

An operator should investigate failures with suitable telemetry while minimising personal content in logs. Queue identifiers, model versions and aggregate timings may be sufficient for many diagnostics without retaining entire private prompts.

Back to contents

21. Timeouts and cancellations affect the denominator

A client may abandon a request after a deadline even while the server continues working. A retry could submit another copy of the same computation. For read-only generation this may waste resources; for tools that change external state, duplicate attempts can cause real consequences.

Define how the benchmark counts submitted, accepted, completed, failed and cancelled requests. A throughput figure based only on successes should not hide the fraction that timed out. Some operations may need a separate verified-action status after text generation has finished.

Responsible service design attaches identifiers and uses reconciliation before repeating consequential writes. An assistant must not say ‘the document was published’ because a model predicted that the publishing tool would probably succeed.

Back to contents

22. Load tests should not require private user data

One can design a useful performance experiment with synthetic inputs whose lengths and arrival patterns resemble the intended workload. Measure the service under permitted conditions, avoiding disclosure of real student documents or overloading an unauthorised external service.

Test light load, typical load, peaks, bursts, long contexts, long outputs and a recovery period. Inspect how measured latency changes as throughput approaches saturation and whether errors or memory pressure appear.

A published benchmark should state whether it used fabricated data, whether the actual system was tested and which capacity rules applied. A simulation illustrates mechanisms but is not itself real service-performance evidence.

Back to contents

23. Worked changed-context packet: six different durations

Now use a second invented sample of six end-to-end latencies, in milliseconds: 100, 110, 120, 130, 600 and 900. First calculate the arithmetic mean and the sample median. Then find nearest-rank p90 using position ceil(0.90×6).

The total is 100+110+120+130+600+900 = 1,960 milliseconds. The mean is 1,960/6 ≈ 326.7 ms. The median is (120+130)/2 = 125 ms. Nearest-rank p90 has position ceil(5.4)=6, so it is 900 ms under this explicitly chosen convention.

Explain why the mean and median tell different stories, and why six observations are too few to claim a stable service p90. The exercise checks whether the method transfers to a more heavily skewed tail.

Back to contents

24. A second transfer: overload after a capacity change

Imagine a simplified service with twenty requests per second of maximum sustainable throughput under fixed conditions. Demand rises to twenty-five requests per second for four seconds, with no cancellations, drops or rejections. If it begins with an empty queue and runs at full capacity, the backlog grows by five each second, to twenty waiting requests.

Now add five requests per second of genuine additional processing capacity, keeping every other assumption unchanged. In the simplified deterministic calculation, arrivals and completions match at twenty-five per second and the backlog no longer grows from zero. This does not guarantee low real tail latency, because arrivals and service times can vary.

The important transfer is to separate capacity arithmetic from observed performance. Only an actual, appropriately safe load test could show how the specific serving engine behaves with real request lengths and system overhead.

Back to contents

25. A practical latency-and-throughput audit

Define which user outcome must be complete and the relevant response deadline. Capture arrival, queue start, execution, first token, final token, downstream checking and verified delivery as separate timestamps where possible. Count cancellations and errors rather than making them vanish from the denominator.

Show mean, median and meaningful tail percentiles with sample size, measurement interval and calculation convention. Segment by input length, output length and task type, and identify model and configuration version. Measure requests and token throughput separately when needed.

Then investigate the real bottleneck rather than blindly increasing model capacity. A useful performance improvement is one that makes the intended task more reliable or responsive under representative conditions without worsening correctness, privacy or approved costs.

Back to contents

26. Frequently asked questions

Is one second of first-token latency the same as a one-second answer? No. Streaming can start before completion. Record both first-token and final verified-result time.

Does higher throughput always mean better user experience? No. A large batch can improve total output per second while increasing some users’ waiting times. Evaluate the actual task and latency target.

Why can a server be slow when its model computation is fast? Queueing, retrieval, network, cold start or output checking may dominate. Measure all stages.

Does p95 guarantee ninety-five percent of future requests will meet a threshold? No. It describes a sample or estimated distribution using a specified convention and conditions, not a universal guarantee.

Back to contents

27. Source notes and connected curriculum

Official performance definitions come from Google Cloud’s model inference documentation, which distinguishes TTFT, token timing, request latency and throughput. NVIDIA’s Triton batcher guide documents scheduling and queue-delay controls. These sources explain mechanisms, not the fictional Study Desk measurements.

Follow the adjacent Inference Infrastructure chapter to inspect serving architecture, and GPUs and Accelerators for hardware bottlenecks. Return to How Super Intelligence Works, How X Works and The Gold Standard of Education.

All request durations, arrival rates, worker capacities and distributions here are invented learning examples. No public service was stress-tested and no real user data were accessed. Reviewed on 22 October 2026.

Back to contents