How Mathematics Improves The World | Spreading Internet Traffic Across Servers Before One of Them Collapses
A website can have ten healthy servers and still fail.
Send every request to Server 1.
Server 1 fills its CPU.
Queues grow.
Response time rises.
Requests time out.
The other nine machines sit partly idle.
The infrastructure did not run out of total capacity.
It failed to distribute the work.
Load balancing solves that coordination problem.
A load balancer receives traffic and chooses a healthy target.
Round robin?
Least connections?
Weighted capacity?
Lowest latency?
Same-user affinity?
Nearest region?
The answer depends on what kind of work is arriving and what “balanced” actually means.
Mathematics turns a river of requests into a controlled queueing network.
Quick Read
Modern load balancers distribute incoming traffic across multiple targets such as servers, containers or IP addresses. AWS Elastic Load Balancing describes the basic operational contract clearly: accept traffic, monitor registered targets, and route requests only to healthy targets. Cloudflare similarly describes traffic distribution, health checks, automatic failover, weighted policies, latency-based steering and geographic routing.
The mathematical problem is broader than “split requests equally”. Servers differ in capacity. Requests differ in cost. Connections can persist for minutes. Network latency differs by user location. Caches and sessions create state. Failure changes the feasible set without warning. Queueing theory explains why utilisation near 100% is dangerous: as arrival rate approaches service capacity, expected waiting time can grow dramatically. Good load balancing therefore keeps slack, senses health and redistributes work before congestion becomes collapse.
Common algorithms include round robin, weighted round robin, least connections, least response time, randomised choice and hash-based routing. Large distributed systems also use consistent hashing to map keys or sessions to nodes while limiting how much work moves when servers join or leave.
One-sentence answer: Mathematics improves the world by turning incoming internet traffic into a dynamic resource-allocation problem, continuously matching requests to healthy capacity so no single server becomes overloaded while useful computing power sits unused elsewhere.
A Request Is Work Arriving in Time
Users do not arrive at perfectly even intervals.
Traffic bursts.
A news story breaks.
A concert ticket sale opens.
A school portal receives thousands of logins at 8 a.m.
Arrival rate λ is therefore a stochastic quantity.
Servers process requests at some service rate μ.
When λ exceeds sustainable μ for long enough, queues grow without bound unless traffic is rejected, delayed or moved elsewhere.
Utilisation: Why 99% Busy Can Be Bad
In the simplest M/M/1 queue, utilisation is:
ρ=λ/μ.
Expected time in system is:
W=1/(μ−λ).
As λ approaches μ, W rises sharply.
This is the queueing reason “keep every server 100% utilised” can be disastrous for latency-sensitive systems.
Slack is not waste.
Slack absorbs randomness.
Round Robin: The Simplest Fair Rotation
Requests arrive:
1,2,3,4,5,6…
Servers A, B, C receive:
A,B,C,A,B,C…
Round robin is easy and has low overhead.
It works well when requests and servers are roughly homogeneous.
It fails when one request costs 1 ms and another 30 seconds, or one server has twice the capacity of another.
Weighted Round Robin: Equal Turns Are Not Equal Capacity
Server A has 32 cores.
Server B has 16.
Server C has 8.
Assign weights 4:2:1.
Over time, A receives about four times as many requests as C.
Cloud load-balancing systems commonly expose weighted traffic distribution for exactly this reason.
Fairness should be proportional to service capacity, not machine count.
Least Connections: Count Ongoing Work
Server A has 200 active connections.
B has 80.
C has 25.
Send the next connection to C.
This adapts better when connection duration varies.
But connection count is only a proxy.
One connection could be idle.
Another could be streaming gigabytes.
Good metrics should track the resource that actually binds.
Least Response Time: Let Performance Feed Back Into Routing
Measure recent latency per target.
Send more requests toward faster healthy servers.
This creates feedback.
Too aggressive a controller can herd traffic to yesterday’s fastest server, overload it, then swing elsewhere.
Smoothing and hysteresis can prevent oscillation.
Load balancing becomes control theory.
Power of Two Choices: Tiny Randomness, Big Queue Improvement
Choose two servers at random.
Send the request to the less loaded one.
This “power of two choices” strategy can dramatically reduce maximum load compared with choosing one random server while keeping measurement overhead low.
The insight is beautiful:
a very small amount of choice can produce a very large improvement in balance.
Health Checks: Capacity Exists Only If It Is Usable
A server is registered.
Does that mean it should receive traffic?
No.
AWS documents active health checks that repeatedly test registered targets and remove them from service after configured failure thresholds, returning them after enough successful checks.
Cloudflare similarly uses health monitoring and automatic failover.
The feasible routing set changes as health state changes.
False Health: The Server Answers “OK” While the Application Is Broken
A TCP port is open.
The database dependency is dead.
A shallow health check says healthy.
A real user request fails.
Health checks should test the layer whose availability matters.
But deep checks are more expensive and can amplify outages if every monitor performs heavy work.
Observability itself needs design.
Failure Thresholds: Avoid Flapping
One failed health check could be packet loss.
Remove the server immediately and capacity may bounce unnecessarily.
Wait for several consecutive failures and detection is slower.
AWS exposes unhealthy and healthy threshold counts for this reason.
Failure detection balances sensitivity and stability.
Failover: The Best Balance Changes Instantly After Failure
Four servers each handle 25%.
One fails.
The remaining three must suddenly handle roughly 33% each.
If they were already at 80% utilisation, this can push them past sustainable capacity.
Redundancy without spare capacity is fragile redundancy.
N+1 design reserves enough headroom to survive at least one expected failure mode.
Cross-Zone Load Balancing: Geography Is Part of Capacity
A cloud service spans several availability zones.
Traffic can be distributed only within a local zone or across zones depending on architecture.
AWS documents cross-zone options for its load-balancing services.
Cross-zone routing can improve utilisation and resilience.
It may add inter-zone traffic cost or latency.
Topology enters the objective.
Geographic Steering: Nearest Is Often Better, Not Always
A Singapore user reaches a Singapore origin faster than a distant one under ordinary conditions.
Cloudflare supports geographic and latency-based steering to direct users toward suitable origins.
But the closest server may be overloaded.
A slightly farther healthy region may produce lower end-to-end latency.
Routing should optimise measured outcome, not map distance alone.
Latency Is a Sum of Layers
User response time includes:
- DNS;
- network propagation;
- TLS handshake;
- queueing;
- server processing;
- database calls;
- response transfer.
Load balancing controls only part of the sum.
A “fast server” can still feel slow through a congested network path.
End-to-end optimisation needs end-to-end measurement.
Sticky Sessions: Sometimes Balance Must Preserve Identity
A user logs in and server-local memory stores session state.
Next request reaches another server.
The session disappears.
One solution is shared external session storage.
Another is session affinity: route the same user consistently to the same target for some period.
Affinity improves state locality and reduces balancing freedom.
State creates routing constraints.
Hashing: Turn a Key Into a Server Choice
For key k, compute hash h(k).
A simple scheme maps:
server = h(k) mod N.
This gives deterministic routing.
Add or remove a server and N changes.
Most keys remap.
That can destroy cache locality.
Consistent hashing was invented to reduce that remapping.
Consistent Hashing: Move Only a Fraction When Membership Changes
Place servers and keys on a hash ring.
A key maps to the next server clockwise.
Add one server.
Only keys in part of the ring move.
Remove one.
Its keys move to neighbours.
This property is valuable for distributed caches, object stores and partitioned services where moving state is expensive.
Virtual Nodes: Smooth the Hash Ring
One hash position per server can create uneven ranges.
Represent each physical server by many virtual positions.
Load spreads more smoothly.
Heterogeneous machines can receive different numbers of virtual nodes.
Randomisation plus replication approximates proportional balance.
Caching: The Fastest Request Is the One the Origin Never Sees
Content delivery networks serve cached objects near users.
This reduces origin traffic before load balancing even begins.
Cache-hit ratio becomes capacity multiplication.
If 90% of requests are satisfied at the edge, origin servers see only the remaining 10%.
Reducing demand can be more powerful than balancing demand.
Autoscaling: Add Capacity When Demand Rises
Load balancing redistributes existing capacity.
Autoscaling changes capacity itself.
Monitor CPU, queue depth, request rate or custom metrics.
When thresholds or predictive models indicate sustained load, add instances.
Remove them when demand falls.
Balancing and scaling form a coupled feedback system.
Scaling Delay: New Servers Do Not Appear Instantly
A new virtual machine may take minutes to boot.
A container may start faster but still needs warm caches and connections.
If demand spikes faster than capacity arrives, queues form.
Predictive scaling estimates future traffic before thresholds are crossed.
Forecasting buys lead time.
Cold Starts: Equal Servers May Not Be Equally Ready
A newly started server has empty caches.
Just-in-time code may not be warmed.
Database connection pools are empty.
Sending full traffic immediately can make the new server look slow.
Slow-start algorithms ramp traffic gradually.
Capacity is not binary; readiness can be a trajectory.
Long-Tail Latency: The Average Can Look Fine While Users Suffer
Average latency: 100 ms.
99th percentile: 4 seconds.
One in one hundred users sees a terrible experience.
Large distributed systems monitor p95, p99 and p99.9 latencies because tail behaviour often dominates perceived reliability.
Load balancers can route away from unhealthy or slow targets, but the metric should include tail performance rather than only mean response time.
Hedged Requests: Ask Twice When Tail Risk Is Expensive
A read request is unusually slow.
After a short delay, issue the same idempotent request to another replica.
Use whichever response arrives first.
This reduces tail latency and increases total load.
Hedging is useful only when the extra resource cost is worth the tail reduction and semantics make duplication safe.
Redundancy trades capacity for latency insurance.
Backpressure: Sometimes the Right Answer Is “Slow Down”
Downstream database capacity is saturated.
Sending more requests into application servers only builds larger queues.
Backpressure propagates overload information upstream.
Clients retry more slowly.
Producers reduce rate.
Queues remain bounded.
Load balancing cannot solve a system whose total demand exceeds every downstream bottleneck.
Retry Storms: Recovery Logic Can Become the Outage
A server fails.
Ten thousand clients retry instantly.
Remaining servers overload.
They fail.
More retries arrive.
Exponential backoff and random jitter spread retries through time.
Randomness can prevent synchronized recovery from becoming synchronized collapse.
Circuit Breakers: Stop Sending Work to a Failing Dependency
A downstream service starts timing out.
Instead of allowing every request to wait for the full timeout, a circuit breaker temporarily fails fast.
After a cooling period, test recovery.
This protects upstream thread pools and queues.
Load balancing and circuit breaking together keep failures from consuming all capacity.
Admission Control: Not Every Request Can Be Accepted During Crisis
If demand exceeds capacity, pretending otherwise creates unbounded queues and universal failure.
Admission control rejects or defers some work deliberately.
Priority queues may protect critical operations.
Rate limits protect shared infrastructure from one tenant or client.
A controlled partial service can be better than uncontrolled total collapse.
Fairness Across Tenants
One customer sends 90% of traffic.
Should they consume 90% of every shared bottleneck?
Weighted fair queueing and token-bucket policies allocate service according to contracts or priorities.
Load balancing across servers and fairness across customers are related but different control planes.
Efficiency does not automatically produce fairness.
Observability: The Balancer Needs a Model of What “Loaded” Means
CPU can be low while a server waits on disk.
Connection count can be low while one query consumes all database capacity.
Queue depth can be high because downstream is slow.
Good routing metrics combine several signals.
Measurement should reflect the bottleneck, not merely the easiest number to collect.
Control Loops Can Interact Badly
Load balancer shifts traffic away from Server A.
Autoscaler sees lower utilisation and removes capacity.
Traffic shifts back.
Another controller reacts.
Independent feedback loops can oscillate.
Distributed systems need coordinated timescales, smoothing and hysteresis so control layers do not fight each other.
Chaos Testing: Break Servers on Purpose Before Reality Does
Terminate one server.
Increase latency in one zone.
Fail a health endpoint.
Observe whether traffic redistributes safely.
Resilience is not proven by a diagram.
It is demonstrated by controlled failure experiments and production evidence.
A Classroom Thought Experiment: Four Checkout Counters
Imagine four checkout queues.
Round robin sends one shopper to each counter in rotation.
Now one shopper has a full trolley and another one item.
Round robin can become unbalanced.
Try least-queue length.
Then make one cashier twice as fast.
Students discover why real load balancing needs state and weights.
Primary Mathematics: Load Balancing Begins With Sharing
Primary students can understand:
- division;
- ratios;
- queues;
- averages;
- rates;
- graphs.
100 requests across four equal servers suggests 25 each.
Then change the capacities.
The fair share changes.
Secondary Mathematics: The Website Becomes a Queueing Network
Secondary students add:
- probability;
- expected value;
- functions;
- rates;
- graphs and networks;
- feedback;
- optimisation.
Arrival rate becomes λ.
Service rate becomes μ.
Health becomes a changing feasible set.
Routing becomes a decision rule.
Advanced Mathematics: Load Balancing as Stochastic Control
Modern traffic distribution draws on:
- queueing theory;
- probability;
- randomised algorithms;
- hashing;
- graph and network theory;
- control theory;
- distributed systems;
- optimisation.
Requests arrive randomly.
Servers fail.
Capacity changes.
The controller keeps routing anyway.
Why This Improves the World
1. It uses existing computing capacity better
Traffic is spread across healthy resources instead of overloading one machine while others idle.
2. It improves availability
Health checks and failover remove failed targets from the routing set automatically.
3. It reduces latency
Queue-aware and latency-aware policies steer work away from congestion.
4. It makes scaling useful
New servers create capacity only when traffic distribution can discover and use them safely.
5. It supports global services
Geographic and latency steering can match users with healthy regional infrastructure.
6. It prevents one failure from becoming many
Headroom, backpressure and controlled retries reduce cascading overload.
What Mathematics Does Not Do
Load balancing cannot create capacity when total demand exceeds the entire system indefinitely.
A healthy TCP port does not guarantee a healthy application.
Equal request counts do not imply equal work.
Nearest geography does not guarantee lowest latency.
Autoscaling does not arrive with zero delay.
Retries do not fix a permanently overloaded dependency.
And a perfect routing algorithm cannot rescue an architecture with a single downstream bottleneck that every request must still pass through.
Frequently Asked Questions
What does a load balancer do?
It accepts incoming traffic and forwards requests or connections to one of several available targets according to a routing policy, commonly using health information to avoid failed targets.
What is round-robin load balancing?
It cycles through servers in order, sending successive requests to successive targets. It is simple and works best when servers and requests are similar.
Why use least connections?
When connection durations vary, the server with the fewest active connections may have more spare capacity than one that merely received fewer recent requests.
What is consistent hashing?
It is a hash-based assignment method designed so adding or removing servers remaps only part of the key space rather than nearly every key, preserving cache or state locality.
Why do load balancers need health checks?
A registered server can fail while remaining configured. Health checks provide current evidence about whether it should remain eligible to receive traffic.
Sources and Further Reading
- AWS, How Elastic Load Balancing Works, on routing, target groups and health monitoring.
- AWS, What Is Elastic Load Balancing?, on distributing traffic, health checks and automatic scaling of load-balancer capacity.
- Cloudflare, Balance Traffic Across Origins, updated 2026, on weighted and latency-based traffic distribution, geographic steering, health checks and failover.
Continue Through eduKateSG
Continue with How Mathematics Works. Compare this article with Keeping Electricity Flowing Through a Changing Grid, where demand is again matched continuously to distributed capacity, and with Making Elevators Decide Who to Pick Up Next, where queueing and changing state also make simple nearest-resource rules insufficient.
Final Thought: A Website Is a Crowd Asking for Tiny Pieces of Time
A request arrives.
Then a thousand.
Then a million.
One server slows.
A health check changes the eligible set.
Traffic shifts.
New capacity starts.
Queues shorten.
The user sees one page load normally.
Mathematics improves the world here by turning many machines into one service that knows how not to ask too much of any one of them.