Find your next learning step
An autoscaler acts on news from the recent past. Measurement windows, decision intervals, startup time and cooldown rules can turn a sensible queue response into overshoot or repeated scale changes. Stability belongs to the whole feedback loop, not to a threshold alone.
Open the chapter contents · Queueing Theory root · Gold Standard of Education · How X Works
Chapter contents
Choose the route closest to the question, or read from Chapter 1 for the complete argument. The chapters remain open on the page; these grouped routes are an index.
Route 1 · Map the delayed loop
Route 2 · Work the interval model
Route 3 · Read stability and oscillation
Route 4 · Test evidence and limits
Route 5 · Check and continue
CHAPTER 1 OF 12 · MAP THE DELAYED LOOP
1. An autoscaler closes a delayed feedback loop
A queue-based autoscaler observes a signal, chooses a capacity target and waits for that target to become effective. The observation may already be an average over several minutes. The controller may reconcile only at fixed intervals. New workers may then need time to start, warm caches or receive work. By the time completions increase, the queue that caused the decision has changed.
Write the loop as four separate times: event time, measurement time, decision time and effective-capacity time. Treating them as one timestamp makes a delayed controller look instantaneous and can assign the wrong queue to the wrong action.
| Loop element | Example record | Question it answers |
|---|---|---|
| Plant state | queue, arrivals, completions | What work actually remained? |
| Sensor | metric window and sample time | What did the controller know? |
| Controller | recommendation and reason | What capacity was requested? |
| Actuator | ready workers and effective time | When did service change? |
CHAPTER 2 OF 12 · MAP THE DELAYED LOOP
2. Stability is different from a good-looking average
For interval t, a basic conservation model is q(t+1) = max{0, q(t) + a(t) − c(t)}, where q is queued work, a is admitted arrivals and c is completed work. Capacity may depend on a recommendation calculated from q(t−d), with delay d. A stable operating claim concerns how the state responds through time, not whether average offered capacity exceeds average arrivals in one report.
Queue feedback can overshoot even when every chosen capacity would be adequate if applied immediately. Delayed scale-up lets work accumulate. Delayed scale-down can leave excess capacity after the queue clears. If another burst arrives after that excess is removed, the cycle can repeat.
Stability also needs a boundary. A finite queue can stay bounded by rejecting jobs. A capped replica count can stay fixed while backlog diverges. Report queue, loss, admitted throughput and capacity together.
CHAPTER 3 OF 12 · WORK THE INTERVAL MODEL
3. Work a delayed interval controller
Consider one-minute intervals. Each ready worker can complete four jobs per interval. Begin with one worker and no queue. Arrivals over six intervals are 10, 10, 2, 2, 2 and 2. A recommendation uses a queue observation two intervals old and sets desired workers to max{1, ceiling(observed queue/4)}. For this teaching example, the recommended count is effective in the next listed interval.
The queue balance gives end-of-interval queues 6, 12, 10, 4, 0 and 0 while ready workers are 1, 1, 1, 2, 3 and 3. The controller finally reaches three workers after the arrival burst has already ended. Capacity responds in the correct direction but late enough to overshoot the remaining work.
| Interval | Arrivals | Ready workers | Capacity | Ending queue |
|---|---|---|---|---|
| 1 | 10 | 1 | 4 | 6 |
| 2 | 10 | 1 | 4 | 12 |
| 3 | 2 | 1 | 4 | 10 |
| 4 | 2 | 2 | 8 | 4 |
| 5 | 2 | 3 | 12 | 0 |
| 6 | 2 | 3 | 12 | 0 |
CHAPTER 4 OF 12 · WORK THE INTERVAL MODEL
4. Separate measurement delay from actuation delay
The worked table compresses several delays into one rule. In practice, a five-minute metric average can hide a rising edge even before the controller evaluates it. Reconciliation delay follows. Provisioning, readiness checks, routing and application warm-up add actuation delay. A worker counted as created is not necessarily a worker completing useful jobs.
These delays affect different evidence. Measurement timestamps show what was visible. Recommendation logs show controller behaviour. readiness and first-completion timestamps show the actuator and plant. Subtracting creation time from queue peak time mixes them.
When delay varies, use its distribution rather than one nominal number. Rare slow starts may dominate the largest queues even if the median scale-up looks fast.
CHAPTER 5 OF 12 · WORK THE INTERVAL MODEL
5. Changed context: shorten the delay, not the threshold
Keep the same arrivals and per-worker capacity, but let the controller use the queue from one interval earlier. The scale-up recommendation arrives sooner, so more of the 12-job peak is served while the burst is still recent. That can reduce both the maximum queue and the later idle-capacity area without changing the target threshold.
This transfer check separates responsiveness from aggressiveness. Lowering a threshold requests more capacity for the same observation. Shortening delay makes an existing request effective sooner. They can produce similar plots in one experiment but behave differently under noise and repeated bursts.
The conclusion is not “shorter is always better.” Very frequent decisions made from noisy metrics can increase switching. Controller period, averaging window, startup time and minimum run time must be tested together.
CHAPTER 6 OF 12 · READ STABILITY AND OSCILLATION
6. Oscillation needs amplitude, period and phase
Repeated replica changes are not automatically an unstable oscillation. A periodic workload can produce a periodic but well-tracked capacity response. Diagnose the relationship: does capacity consistently lag the queue, does each correction cross the target, and do swings persist after the disturbance stops?
Measure peak-to-trough queue amplitude, replica amplitude, cycle period and phase lag. Also count changes, because a small-amplitude chatter can be costly without creating a large queue. A controller may be bounded yet operationally poor.
In a linearised model, delay adds phase lag and high feedback gain can turn correction into over-correction. In a real autoscaler, integer replicas, saturation, dead bands and warm-up create nonlinear behaviour. Simulation or replay is often safer than pretending those features are negligible.
CHAPTER 7 OF 12 · READ STABILITY AND OSCILLATION
7. Dead bands, hysteresis and stabilisation windows
A dead band ignores small deviations around the target. Hysteresis uses different conditions for scaling up and scaling down. A stabilisation window considers recent recommendations before removing capacity. Each adds memory to prevent rapid reversals, so current queue alone no longer determines the action.
Kubernetes documents configurable scaling behaviour, tolerance and stabilisation windows for the Horizontal Pod Autoscaler. Its documentation explains that recent recommendations can be retained to smooth scale-down and reduce flapping. Those settings are implementation facts, not proof that a particular workload is stable.
State the exact rule and software version used. “Cooldown enabled” is not enough to reproduce a controller whose direction-specific windows, rate limits and readiness treatment differ.
CHAPTER 8 OF 12 · READ STABILITY AND OSCILLATION
8. Queue metrics and utilisation metrics answer different questions
Utilisation measures how busy current capacity is. Queue length measures unfinished admitted work. High utilisation can occur with no backlog, while a queue can persist during low reported utilisation if the metric omits a bottleneck or workers are not ready. Autoscaling on one signal changes the dynamics of the other.
For queue targets, define whether the signal counts waiting jobs, jobs in service, retries or unique requests. For utilisation targets, retain denominator changes when replicas are added. An average utilisation can fall instantly because capacity increased even though no additional job has completed.
Multi-metric controllers need an explicit arbitration rule. Choosing the largest recommendation protects different constraints from averaging each other away, but missing metrics and scale-down handling still matter.
CHAPTER 9 OF 12 · TEST EVIDENCE AND LIMITS
9. What the model does not establish automatically
The interval example does not prove a universal autoscaling policy. It assumes identical workers, immediate routing once ready, divisible demand only at interval boundaries and known service capacity. Stateful services, heterogeneous jobs, downstream limits and correlated failures can break those assumptions.
A stable queue model also does not prove a reliable service. Tail latency, abandonment, rejection, cost and error rates may worsen while mean queue falls. Nor does a controller log establish causality if a scheduler, rate limiter or human operator changed capacity at the same time.
Finally, a short test cannot demonstrate long-run stability under every admissible input. Use bounded claims tied to the workload range, delays and constraints that were actually exercised.
CHAPTER 10 OF 12 · TEST EVIDENCE AND LIMITS
10. A practical stability evidence check
Reconstruct every interval from event-level admitted arrivals, completions, abandonments and rejections. Join metric windows to recommendations and join recommendations to ready-capacity changes. Verify q(t+1) against the ledger before fitting controller parameters.
Replay a quiet-to-burst-to-quiet input and a repeated-burst input. Perturb measurement delay, startup delay and per-worker service rate. Record maximum queue, clearing time, rejected work, replica-time, switch count and whether swings decay after arrivals return to baseline.
Compare against fixed capacity as a control. An autoscaler that looks active may add no benefit if capacity becomes available only after each workload episode ends.
CHAPTER 11 OF 12 · CHECK AND CONTINUE
11. Questions readers should ask
Does average capacity above average demand guarantee a stable autoscaler?
No. Time variation, delay, caps, losses and feedback policy determine the path. An average can hide an interval in which backlog grows faster than the controller can respond.
Is every repeated scale change an oscillation?
No. It may track periodic demand. Check phase, overshoot and whether the motion decays after a disturbance stops.
Why keep recommendation and ready-capacity timestamps?
They separate controller delay from actuation delay. Without both, a slow startup can be misdiagnosed as a poor decision rule.
Can a stabilisation window remove all instability?
No. It can reduce rapid reversals, but a poor target, long scale-up delay, capacity cap or downstream bottleneck can still produce large queues.
CHAPTER 12 OF 12 · CHECK AND CONTINUE
12. Continue through the verified queueing ecosystem
Begin with the Queueing Theory root, the Gold Standard of Education and the How X Works hub. Build the immediate prerequisite with Environment Feedback, Endogenous Capacity and Queue-Dependent Switching, then compare Time-Varying Arrivals and Capacity That Arrives Too Late and Retries, Attempt Rates and Unique Jobs.
For implementation evidence, use the official Kubernetes Horizontal Pod Autoscaling documentation and HorizontalPodAutoscaler v2 API reference. For a primary systems study of delayed and noisy performance control, read Black-Box Performance Control for High-Volume Non-Interactive Systems. These sources describe particular controllers and experiments; they do not replace workload-specific validation.
Top · Contents · Previous · How X Works