Why is mathematics important in numerical stability? Because a formula can be mathematically correct and still produce an unreliable computer answer. Computers store finite approximations, round after operations and sometimes subtract nearly equal quantities. A stable method controls how those small errors grow.
This matters in spreadsheets, scientific calculators, simulations, finance, engineering, machine learning and navigation. Numerical stability does not mean “use more decimal places” or “trust the calculator.” It means separating the sensitivity of the problem from the behaviour of the algorithm, then choosing a computation whose errors remain proportionate.
Exact Mathematics Meets Finite Representation
Real numbers can require infinitely many digits. A computer format stores a finite sign, significand and exponent, so most real values are rounded.
Binary cannot represent every decimal exactly
The decimal fraction 0.1 has a repeating expansion in binary. A binary floating-point system stores a nearby value. Printing may hide the difference, but later operations can reveal it.
IEEE 754 defines shared behaviour
The IEEE 754-2019 standard specifies floating-point formats, operations, rounding and exceptional values. It makes behaviour far more predictable; it does not turn finite arithmetic into exact real arithmetic.
Rounding is necessary
When the exact result lies between representable numbers, the system chooses according to a rounding rule. Round-to-nearest is common because it limits typical bias.
Quick Reading Routes
- Understand floating-point spacing and rounding.
- Separate problem conditioning from algorithm stability.
- Work through cancellation, overflow and summation examples.
- Compare forward, backward and relative error.
- Build reliable testing and communication habits.
- Finish with student investigations, guidance and FAQs.
Absolute and Relative Error Answer Different Questions
If the true value is x and the computed value is x-hat, absolute error is |x-hat − x|. Relative error divides by |x| when x is non-zero.
A measurement example
An answer 1000.2 instead of 1000 has absolute error 0.2 and relative error 0.0002, or 0.02%. An answer 0.3 instead of 0.1 also has absolute error 0.2 but relative error 200%.
Scale matters
Absolute tolerances suit quantities with a meaningful fixed unit. Relative tolerances suit values spanning many magnitudes. Reliable tests often combine both.
Near zero needs care
Relative error becomes huge or undefined near zero. A test such as |a − b| ≤ atol + rtol|b| uses an absolute floor plus relative scaling.
Floating-Point Spacing Changes With Magnitude
Floating-point numbers are denser near zero and farther apart at large magnitudes.
One unit in the last place
An ulp is the gap between adjacent representable values near a number under a stated format. The same decimal increment can be visible at one scale and disappear at another.
Adding small to large
If a very large accumulated total has spacing greater than a tiny new term, adding that term may leave the stored total unchanged. The exact sum changed; the representable rounded sum did not.
Precision is not decimal places everywhere
A binary format provides a roughly fixed number of significant binary digits, not a fixed number of digits after the decimal point.
Conditioning Belongs to the Problem
A problem is well-conditioned when small changes in input cause small changes in output. It is ill-conditioned when tiny input changes can cause large output changes.
A simple function
For y = 1/(1 − x), values near x = 1 are sensitive. If x moves from 0.999 to 0.9991, the denominator changes from 0.001 to 0.0009 and y changes from 1000 to about 1111.11.
The input may already be uncertain
If x was measured only to three decimal places, no algorithm can manufacture reliable extra information near the singularity.
A condition number
A relative condition number estimates output-relative-change divided by input-relative-change for small perturbations. A large number warns that input error can be strongly amplified.
Conditioning is not an algorithm bug
Changing code cannot make the underlying question insensitive. It can avoid adding unnecessary error and can report uncertainty honestly.
Stability Belongs to the Algorithm
A stable algorithm produces the exact answer to a nearby problem or keeps its forward error close to what conditioning predicts.
Same formula, different route
Algebraically equivalent expressions can behave differently in finite arithmetic. Rearrangement can avoid cancellation, overflow or repeated rounding.
Backward stability
If the computed result is exactly correct for inputs only slightly perturbed from the stated inputs, the method is backward stable.
Forward accuracy
Forward error compares computed output with the exact output for the original input. A backward-stable method can still have large forward error when the problem is ill-conditioned.
The central separation
Conditioning asks, “How sensitive is the mathematical problem?” Stability asks, “How much extra error does this computational method introduce?”
Catastrophic Cancellation
Subtracting nearly equal numbers can discard significant digits.
A decimal toy model
Suppose a machine keeps six significant digits. Values 1.23456 and 1.23455 are each known to six digits. Their difference is 0.00001. Small rounding in the original values becomes large relative uncertainty in the difference.
The quadratic formula
For ax² + bx + c = 0, the expression −b + sqrt(b² − 4ac) can subtract nearly equal values for one root. An alternative computes one root with the sign that avoids cancellation, then uses x1x2 = c/a for the other.
Cancellation is not always bad
Exact cancellation can be meaningful. The danger is subtracting approximations that share leading digits while their small difference carries the answer.
Reformulation protects information
Identities such as log(1 + x) through a specialised `log1p` function or exp(x) − 1 through `expm1` preserve accuracy for small x better than naive evaluation.
A Worked Quadratic Example
Consider x² − 100000x + 1 = 0. The exact roots are widely separated.
Naive small root
The small root uses [100000 − sqrt(100000² − 4)]/2. The square root is very close to 100000, so subtraction loses leading digits.
Stable route
Compute the large root with addition: [100000 + sqrt(100000² − 4)]/2. Then compute the small root as 1 divided by the large root because the product is c/a = 1.
Check with an invariant
The sum of roots should be 100000 and product should be 1. Invariants expose a numerically damaged result without requiring every hidden digit of the exact answer.
Precision is not the only fix
More precision delays the problem but does not remove the unstable structure. Reformulating the algorithm is usually more portable.
Summation Accumulates Rounding
Adding a long list left to right can repeatedly round the running total.
Order changes the answer
Adding small magnitudes first can preserve them before they meet a large total. Pairwise summation combines values in a tree and often reduces error growth.
Kahan compensated summation
Compensated summation carries a correction for low-order information lost in the previous addition. It costs extra operations but can greatly improve sums with varied magnitudes.
A practical example
Add one million values of 0.01 to a large baseline. A format whose spacing near the baseline exceeds 0.01 may ignore individual additions. Summing cents separately and combining later preserves more information.
Exact domains may be better
Money with fixed decimal subdivisions may use integer cents or decimal arithmetic under defined rounding rules. Choosing representation is part of correctness.
Overflow and Underflow
Overflow occurs when magnitude exceeds the format's range. Underflow concerns values near the smallest representable magnitudes.
Intermediate overflow
The expression sqrt(x² + y²) can overflow in x² even when the final hypotenuse is representable. Scaling by the larger magnitude avoids squaring huge numbers directly.
Log-sum-exp
Computing log(exp(a) + exp(b)) can overflow when a or b is large. Let m = max(a,b), then use m + log(exp(a−m) + exp(b−m)). The exponentials are at most 1.
Gradual underflow
IEEE formats include subnormal values that extend representation toward zero with reduced precision. They support smoother behaviour but do not guarantee every tiny result is meaningful.
Units can prevent range problems
Representing nanoseconds in one context and seconds in another changes numeric magnitude. Unit conversion should be explicit and tested.
Iterative Methods Have Several Errors
An iterative solver approaches an answer over repeated steps.
Truncation error
The method approximates a continuous or infinite process with finite steps.
Rounding error
Each step uses finite arithmetic. Smaller steps can reduce truncation while increasing the number of rounded operations.
Stopping error
Stopping too early leaves iteration error. Stopping based only on successive values can be misleading if progress stalls.
Residual is not always solution error
A small residual means the computed value nearly satisfies the equation. For an ill-conditioned system, the actual solution may still be far from the true one.
Solving Linear Systems
Matrix equations Ax = b appear in science, graphics, statistics and engineering.
Do not compute the inverse by default
Solving through factorisation is usually more efficient and stable than explicitly forming A inverse.
Pivoting matters
Gaussian elimination with pivoting avoids dividing by dangerously small pivots when a better row is available.
Scaling affects interpretation
Variables measured in metres, micrometres and gigapascals can create badly scaled matrices. Nondimensionalisation or sensible scaling improves algorithms and human reasoning.
Condition estimates guide trust
A large condition estimate warns that input or rounding errors may amplify. Report it with residuals when decisions depend on the solution.
Variance Is a Stability Example
The identity variance = mean(x²) − mean(x)² is mathematically correct. It can subtract two large nearly equal quantities when variance is small relative to the mean.
One-pass danger
Large observations near 1,000,000 with tiny spread create two terms near 10¹². Their difference may lose meaningful digits.
Stable updates
Algorithms such as Welford's update maintain a running mean and centred sum of squares, avoiding the most severe subtraction.
Distributed combination
Partial means and centred sums can be combined with formulas that account for group-size and mean differences. Simply averaging variances is generally wrong.
Statistics needs definitions
Population and sample variance use different denominators. Numerical stability does not decide which estimand is appropriate.
Differentiation Near a Limit
A finite-difference derivative uses [f(x+h) − f(x)]/h.
Large h
Large h creates truncation error because the secant is not close enough to the tangent.
Tiny h
Tiny h makes f(x+h) and f(x) nearly equal, causing cancellation; dividing by h magnifies the difference error.
There is a useful middle
Error often has a U-shaped relationship with step size. The best h balances truncation and rounding for the function, scale and precision.
Automatic differentiation is different
Automatic differentiation applies chain rules to program operations; it is not the same as symbolic differentiation or finite differences. It has its own memory and conditioning considerations.
Reproducibility and Operation Order
Floating-point addition is not mathematically associative after rounding.
Parallel reductions change grouping
A multi-threaded sum can group terms differently between runs or machine counts. Small last-digit differences may be expected.
Compiler transformations
Fused multiply-add can round once instead of twice. Fast-math options may assume away NaNs, infinities or signed zero. Performance flags can change semantics.
Determinism is a requirement
If exact reproducibility matters, specify format, rounding, operation order, libraries and hardware expectations. “Same formula” is not enough.
Tolerance needs meaning
Do not choose a loose tolerance solely to make a test pass. Tie it to error analysis and user consequences.
Interval and Higher-Precision Methods
Some applications need explicit bounds or more digits.
Interval arithmetic
Intervals enclose possible values and use outward rounding. Dependency between repeated variables can make bounds wider than necessary.
Arbitrary precision
More significand bits reduce rounding but cost time and memory. They do not repair ill-conditioning or a wrong model.
Exact rational arithmetic
Fractions can remain exact for some algebraic tasks, but numerator and denominator sizes can grow quickly.
Mixed precision
Systems may store or multiply in lower precision and accumulate or refine in higher precision. Error analysis must cover the actual path.
A Reliability Case Study
A sensor pipeline subtracts a baseline of 10,000.000 from readings near 10,000.005, then averages the residuals.
Risk
If readings are stored with insufficient significant precision, the 0.005 signal can disappear before subtraction.
Better design
Store adequate precision, calibrate units, subtract a representable reference early, and sum centred residuals with a stable method.
Validation
Test synthetic signals of known amplitude, values just around rounding boundaries and a higher-precision reference calculation.
Communication
Report sensor uncertainty and numerical error separately. A stable computation cannot make the physical sensor more accurate.
Error Budgets
Reliable work assigns a budget to input uncertainty, modelling, discretisation, iteration and rounding.
Combine thoughtfully
Worst-case bounds may add absolute contributions. Independent random errors may sometimes combine quadratically, but independence must be justified.
Focus on dominant terms
Reducing rounding from 10⁻¹² to 10⁻¹⁵ is irrelevant if measurement uncertainty is 10⁻³.
Revisit after scale changes
A model that was stable for hundreds of records may behave differently for billions. Magnitudes, iteration counts and reduction orders change.
Keep units attached
An error of 0.01 means little without metres, dollars, probability or another scale.
Decimal, Binary and Integer Representations
Not every quantity should use the same number system. Binary floating point is efficient for scientific computation, graphics and machine learning, but money, counters and identifiers may need different representations.
Fixed-point arithmetic
If a price has two decimal places, a program can store cents as an integer. S$12.34 becomes 1234 cents. Addition is exact while values remain in range. Multiplication and division still require a rounding rule, so fixed point moves the decision to an explicit boundary rather than eliminating it.
Decimal floating point
Decimal formats can represent many everyday decimal fractions exactly, including 0.1. They are useful when decimal rules are part of the specification. They do not make all fractions exact: one third still repeats, and operations can still overflow, underflow or round.
Integers also have limits
An integer type has a largest and smallest value. Overflow behaviour depends on the language and type. A calculation that converts from floating point to integer may also truncate or round. Reliable software chooses a representation from the problem's rules, range and performance needs.
Unit Roundoff and a Simple Error Bound
A floating-point operation is often modelled as fl(a op b) = (a op b)(1 + δ), where the relative error δ is bounded by a small unit roundoff u, provided the exact result is in the normal range. This model is an approximation, but it gives a disciplined way to reason about accumulated error.
Repeated operations
For n operations, the worst-case bound is not simply “n digits are lost”. Under standard assumptions, analysis often uses γₙ = nu/(1 – nu) when nu is less than 1. The expression shows why both precision and operation count matter.
Worst-case bounds can be conservative because rounding errors may partly cancel. They remain valuable when failure has a high cost. Experimental averages cannot replace a required safety bound.
Fused multiply-add
A fused multiply-add computes a × b + c with one final rounding rather than rounding the product and then the sum separately. IEEE 754 specifies fused operations. They can improve both accuracy and speed, but they also mean that two apparently similar execution paths may produce slightly different last bits.
This is another reason to specify acceptable error and reproducibility requirements instead of assuming every platform must follow the same sequence of roundings.
Polynomial Evaluation and Horner's Method
Consider p(x) = a₀ + a₁x + a₂x² + a₃x³. A direct program calculates powers and several products. Horner's method rewrites it as p(x) = a₀ + x(a₁ + x(a₂ + xa₃)).
The nested form usually needs fewer multiplications. Fewer operations can mean less accumulated rounding and lower computational cost. It also avoids explicitly forming large powers that may overflow before terms cancel.
A worked count
For a cubic polynomial, direct evaluation may require computing x² and x³ plus multiplying three coefficients by powers. Horner's form uses three multiplications and three additions in a regular chain. The exact operation count depends on what is reused, but the nested structure is simpler and often more stable.
Stability still depends on the polynomial
Near a root, the true value may be tiny compared with the individual terms. Cancellation can remain unavoidable. Changing the evaluation scheme cannot make an ill-conditioned question well conditioned. It can, however, avoid adding unnecessary computational error.
Trigonometric and Exponential Reformulations
Mathematical identities can protect significant information.
Small angles
For small x, evaluating 1 – cos(x) subtracts two numbers near 1. The identity 1 – cos(x) = 2 sin²(x/2) avoids that particular cancellation. A numerical library may use specialised functions and range reduction for similar reasons.
Logarithms near one
Computing log(1 + x) for tiny x can lose x when 1 + x rounds to 1. Many libraries provide `log1p(x)`, designed to retain accuracy in this region. Likewise, `expm1(x)` computes exp(x) – 1 accurately for small x.
Range reduction
Trigonometric functions reduce a large argument to a smaller interval before approximating it. If the input is enormous, accurately determining the correct multiple of π can itself be difficult. The function's mathematics and the representation of constants both matter.
These examples teach a useful habit: look for a library function whose contract matches the delicate expression rather than assembling it from general functions.
Testing Numerical Software
Ordinary unit tests often compare one output with one expected number. Numerical tests need a richer design.
Use invariants
An invariant is a property that should remain true. Probabilities should sum close to 1, a rotation should approximately preserve length, and roots of a polynomial should reproduce its coefficients within a justified tolerance. Invariants test structure rather than one chosen example.
Compare against a reference
For small cases, use higher precision, exact arithmetic or a trusted implementation as ground truth. The reference must be independent enough to avoid repeating the same bug.
Test difficult regions
Include values near zero, very large and very small magnitudes, nearly equal operands, repeated values, singular or almost singular matrices, and thresholds where branches change. Random testing over an easy range can miss precisely the cases that expose instability.
Define tolerances from the problem
An assertion such as `abs(actual – expected) < 0.001` is meaningful only if 0.001 has a reason. Combine an absolute tolerance for values near zero with a relative tolerance for larger values, and connect both to measurement uncertainty or downstream requirements.
Property-based and metamorphic tests
When exact answers are hard to list, test relationships. Scaling all inputs by a known factor may predict how the output scales. Reordering an operation may be expected to change only within a bound. Such tests explore many cases while retaining mathematical meaning.
Numerical Stability in Machine Learning
Machine-learning systems perform large chains of matrix operations, reductions and nonlinear functions. Their scale makes stable formulations important even when individual roundoff errors look tiny.
Softmax
The expression exp(zᵢ)/Σ exp(zⱼ) can overflow when a logit is large. Subtracting the maximum logit from every z leaves the exact ratios unchanged while keeping exponentials at or below 1. This is the same reformulation principle used in log-sum-exp.
Loss scaling and mixed precision
Lower-precision formats save memory and increase throughput, but small gradients can underflow. Loss scaling multiplies the loss before backpropagation and compensates later. Dynamic schemes adjust the scale when overflow appears. This is an engineering response to a numerical range constraint, not a change to the learning objective.
Stable statistics
Normalisation layers and optimisers rely on means, variances and accumulated moments. Stable update formulas and suitable accumulator precision help prevent drift. A model's accuracy evaluation should also include repeat runs because nondeterministic operation order can create small differences.
Accuracy is not model validity
A numerically stable training run can still use biased data, the wrong objective or an unsuitable evaluation set. Numerical reliability protects the calculation; it does not establish that the social or scientific conclusion is sound.
Communicating a Computed Result
A trustworthy report does not print every stored digit as if each were known.
Match significant figures to evidence
If an input is measured to the nearest 0.1 cm, reporting a derived length to twelve decimal places creates false precision. Keep extra digits during calculation when helpful, then round the communicated result according to uncertainty and purpose.
Report method and version
For consequential work, record the algorithm, library, precision, tolerance, stopping rule and software version. These details make a result reproducible and explain why a later implementation may differ slightly.
Separate warning from failure
A condition estimate or residual may indicate reduced confidence without proving the answer useless. Systems should distinguish a successful accurate result, a result with a warning, and a failed computation. Hiding all three behind one number makes review harder.
Give a decision-relevant bound
If the result controls a threshold, state whether uncertainty could change the decision. A value of 9.9998 against a limit of 10 is not comfortably below the limit when total error could be ±0.001. Mathematics supports the honest statement: the computation is too close to call at the available precision.
Stopping Rules Need Scale
Iterative algorithms need a rule for deciding when to stop. A fixed rule such as “stop when the change is below 0.001” can be too strict for values near a million and too loose for values near 0.0001.
Absolute and relative criteria
A combined test can stop when |xₙ₊₁ – xₙ| is less than atol + rtol × |xₙ₊₁|. The absolute term protects the region near zero; the relative term scales with the answer. The constants must come from the problem's required accuracy, not convenience alone.
Small step does not prove convergence
An iteration can stagnate because rounding prevents further change even though the answer is inaccurate. Check a residual or another problem-specific condition as well. Conversely, a residual can be small while an ill-conditioned problem still has a large solution error.
Maximum iterations are a guardrail
A maximum count prevents an unexpected case from running forever. Reaching it should produce a clear status, not silently return the last number as fully converged.
Cross-Platform Reproducibility
Two compliant systems may choose different instruction sequences, reduction orders or library algorithms. The last bits can differ without either system being defective.
Bitwise reproducibility may be necessary for audits, debugging or scientific archives. Achieving it can require fixed algorithms, controlled compiler options, deterministic parallel reductions and recorded library versions. Numerical reproducibility is a different goal: results agree within a justified tolerance or lead to the same scientific conclusion.
Teams should name which goal they need. Demanding bitwise identity everywhere can sacrifice performance, while accepting an undefined tolerance can hide a real error. A clear contract states the representation, allowed variation, validation method and action when the bound is exceeded.
Reproduce the environment as well as the formula
A saved input file is not always enough. Thread count can change reduction order; processor features can enable fused operations; a library update can choose a different kernel. Record these conditions when they could affect a consequential result.
For classroom work, students can repeat one calculation with different operation orders and precisions. They should identify which digits remain stable and whether any decision changes. The exercise shows that reproducibility is a designed property with a stated purpose, not an assumption attached to every displayed digit.
Did You Know? More Digits Can Hide a Bad Method
A spreadsheet may display fifteen digits even when the input data supports only three significant digits. Formatting reveals stored digits; it does not certify knowledge.
David Goldberg's classic article, What Every Computer Scientist Should Know About Floating-Point Arithmetic, explains why representation, rounding and exceptions belong in software reasoning rather than being dismissed as calculator quirks.
Common Misconceptions
“Floating point is random”
Operations follow defined rules, but finite rounding and order can produce surprises.
“More precision guarantees accuracy”
It reduces one error source. Bad data, ill-conditioning, unstable formulas and wrong models remain.
“A tiny residual proves the solution is correct”
Not when the problem is ill-conditioned.
“Equivalent algebra gives identical computer results”
Finite intermediate values can overflow, underflow or cancel differently.
“Round only at the end solves everything”
Hardware rounds intermediate operations; the algorithm must account for that.
“Every last-digit difference is a bug”
Parallel grouping and permitted operations can create small differences. Judge them against requirements.
How Students Can Investigate
Reveal 0.1
Print 0.1 with many binary or decimal digits, then add it repeatedly and compare with multiplication.
Change summation order
Add a list containing one large value and many small ones forward, reverse and pairwise. Compare with a high-precision reference.
Explore cancellation
Evaluate sqrt(x+1) − sqrt(x) for large x, then rationalise it as 1/[sqrt(x+1)+sqrt(x)].
Find the finite-difference minimum
Approximate a derivative over decreasing h and plot error. Identify where rounding overtakes truncation.
Write assumptions
For every result, state format, reference value, absolute error and relative error when meaningful.
Guidance for Parents and Teachers
Treat calculator disagreement as an invitation to inspect representation, brackets, angle mode and rounding—not as proof that technology is unreliable.
Ask learners to estimate before calculating. A result near 10⁸ when the expected scale is 10⁻³ should trigger investigation even if the display looks precise.
This topic connects significant figures, functions, algebraic identities, sequences, logarithms and error bounds. It builds useful habits for science, computing, engineering and finance without guaranteeing a pathway.
For current Singapore examination requirements, refer to MOE secondary education information and SEAB. Use the terminology and calculator rules for the learner's actual cohort and subject.
A Responsible Computation Checklist
- State units and input uncertainty.
- Choose a suitable numeric representation.
- Estimate the expected magnitude.
- Inspect conditioning or sensitivity.
- Avoid avoidable cancellation.
- Scale to prevent overflow and underflow.
- Use stable library functions.
- Set justified stopping rules.
- Compare against a trusted reference on test cases.
- Test boundary, extreme and adversarial inputs.
- Report residuals and condition information where useful.
- Do not print more certainty than the model supports.
Frequently Asked Questions
What is numerical stability?
It describes whether a computational method controls the growth of rounding and related errors.
What is conditioning?
It describes how sensitive the mathematical problem's answer is to small input changes.
Why is 0.1 not exact in binary floating point?
Its binary expansion repeats, so a finite format stores a nearby representable number.
What is catastrophic cancellation?
It is severe loss of relative information when nearly equal approximations are subtracted.
Does IEEE 754 guarantee identical results everywhere?
It standardises formats and operations, but expression order, compiler choices, libraries and permitted precision differences can still matter.
When should decimal arithmetic be used?
It can suit domains with decimal rules, such as fixed monetary subdivisions, but rounding policy still must be specified.
Is higher precision always slower?
It usually costs resources, but the exact effect depends on hardware and software. Measure the actual workload.
Why can a parallel sum differ?
Threads may group additions differently, and rounded addition is not associative.
Can a stable algorithm fix uncertain data?
No. It avoids adding excessive numerical error but cannot recover information never measured.
Does learning this guarantee an engineering career?
No. It develops valuable reasoning alongside domain knowledge, programming, experimentation and communication.
Useful Next Reading
- Compare device calculations in Scientific Calculators, Significant Figures, Numerical Methods and Rounding.
- Review formula auditing in Spreadsheets, Cell References, Formulas, Floating-Point Error and Checking.
- Connect estimates to Database Query Planners, Cardinality Estimates, Join Costs and Execution Plans.
- Read the official IEEE 754-2019 standard page.
Final Perspective
Numerical stability teaches that a correct symbolic formula is only the beginning. Representation, conditioning, operation order and algorithm design determine whether a finite machine preserves the information that matters.
The mature question is not “How many digits did the calculator show?” It is “How sensitive was the problem, what errors entered, and did the method amplify them?” That question turns computation from button pressing into mathematical judgement.