Chapter map

1. 1. System
2. 2. Start
3. 3. First basis
4. 4. Multiply
5. 5. Project
6. 6. Orthogonalise
7. 7. Next basis
8. 8. Reduced solve
9. 9. Iterate
10. 10. Residual
11. 11. Second space
12. 12. Finish
13. 13. Restart
14. 14. Conclusion

Visit the How Mathematics Works hub for the wider series and its connections to education, reasoning, science, computing, design and careers.

---

1. System

The central mathematical idea is concrete. Use A=[[1,1],[0,2]], b=(0,1).

A concrete situation

Exact x=(-1/2,1/2).

What the mathematics does not license

A is nonsymmetric. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record system. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

2. Start

Here the geometry becomes operational. Set x0=0.

A concrete situation

r0=b and beta=1.

What the mathematics does not license

Use true residual. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record start. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

3. First basis

This step deserves a careful pause. Normalize r0.

A concrete situation

q1=(0,1).

What the mathematics does not license

Keep unit norm. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record q1. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

4. Multiply

The useful question is not only how to calculate, but what the calculation assumes. Compute Aq1.

A concrete situation

It is (1,2).

What the mathematics does not license

Use original A. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record vector. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

5. Project

A small example makes the mechanism visible. Compute h11.

A concrete situation

h11=2.

What the mathematics does not license

Dot with q1. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record entry. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

6. Orthogonalise

This is where modelling choices begin to affect evidence. Subtract 2q1.

A concrete situation

Remainder=(1,0).

What the mathematics does not license

Check orthogonality. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record w. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

7. Next basis

The central mathematical idea is concrete. Normalize remainder.

A concrete situation

h21=1,q2=(1,0).

What the mathematics does not license

No breakdown. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record basis. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

8. Reduced solve

Here the geometry becomes operational. Minimize ||(1,0)-y(2,1)||.

A concrete situation

y=2/5.

What the mathematics does not license

Use least squares. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record y. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

9. Iterate

This step deserves a careful pause. Set x1=q1 y.

A concrete situation

x1=(0,2/5).

What the mathematics does not license

Map back correctly. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record x1. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

10. Residual

The useful question is not only how to calculate, but what the calculation assumes. Compute b-Ax1.

A concrete situation

(-2/5,1/5), norm 1/sqrt5.

What the mathematics does not license

Check directly. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record residual. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

11. Second space

A small example makes the mechanism visible. Use q1,q2.

A concrete situation

It spans R2.

What the mathematics does not license

Exact arithmetic. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record dimension. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

12. Finish

This is where modelling choices begin to affect evidence. Solve within K2.

A concrete situation

x2=(-1/2,1/2).

What the mathematics does not license

Verify Ax=b. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record solution. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

13. Restart

The central mathematical idea is concrete. Discard basis after m steps.

A concrete situation

Memory falls.

What the mathematics does not license

Convergence can change. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record m. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

14. Conclusion

Here the geometry becomes operational. State Arnoldi,restart,preconditioner,tolerance.

A concrete situation

Residual minimized each full step.

What the mathematics does not license

Keep true-residual check. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Write conclusion. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

A seven-day practice lab

1. Day 1: define the decision, observation unit, X variables and Y or reference condition.
2. Day 2: reproduce the smallest worked example and check every dimension, unit and sign.
3. Day 3: build a transparent baseline and write what it assumes.
4. Day 4: fit the main method inside a training-only pipeline; record every parameter.
5. Day 5: challenge one assumption with a deliberately difficult dataset or operating regime.
6. Day 6: compare validation metrics with the practical workload created by errors.
7. Day 7: present one supported conclusion, one unresolved uncertainty and one safe next experiment.

Questions for a student, parent or mentor

• Which part of the model is a mathematical definition, and which part is a choice?
• What small calculation would catch a sign, scale or dimension error?
• What evidence comes from training, validation and the untouched test?
• What benign condition could look like a fault or prediction failure?
• How would the answer change for another response, asset, school project or time period?
• What should a human review before acting on the score?

A method-specific audit before deployment

GMRES solves a linear system Ax=b by choosing, at each iteration, the vector in the current Krylov affine space whose Euclidean residual norm is smallest. This makes residual minimisation the defining contract rather than an informal hope.

The Arnoldi process builds orthonormal vectors spanning K_k(A,r0). Multiplication by A produces a small upper-Hessenberg matrix, so the large residual problem becomes a compact least-squares problem.

Use A=[[1,1],[0,2]], b=(0,1) and x0=(0,0). Then r0=b, beta=1 and q1=(0,1). Multiplying gives A q1=(1,2).

Arnoldi records h11=q1 dot A q1=2, subtracts 2q1 and leaves (1,0). Thus h21=1 and q2=(1,0). The first reduced matrix column is (2,1).

One-step GMRES minimizes ||(1,0)-y(2,1)||. The minimizer is y=2/5, so x1=(0,2/5), residual=(-2/5,1/5) and residual norm is 1/sqrt5.

The second Krylov space fills R2. Solving the system gives x=(-1/2,1/2), and unrestarted GMRES reaches that exact solution in exact arithmetic.

Did You Know? GMRES residual norms cannot increase in exact arithmetic while the Krylov space expands, because every new minimisation includes all earlier candidates. Restarting changes that nesting.

Memory grows because Arnoldi stores and orthogonalises against prior basis vectors. Restarted GMRES limits storage but can slow or stagnate because it discards accumulated subspace information.

Nonsymmetric matrices can be nonnormal: their eigenvectors may be far from orthogonal. Eigenvalues alone can then predict convergence poorly, so field-of-values, pseudospectral and empirical residual evidence can matter.

Modified Gram-Schmidt, Householder Arnoldi and reorthogonalisation make different stability and cost choices. Saying only “orthogonalise” omits an important numerical decision.

Preconditioning changes the system seen by the Krylov method. Left and right preconditioning give different reported residuals and iterate formulas even when both are legitimate.

A small recursive residual should be checked occasionally against the true residual b-Ax. Floating-point recurrences can drift from the original equation.

Students can plot residual norm after each iteration, then restart every one or two steps. The experiment shows that a local memory limit changes the global approximation path.

Engineering applications include fluid flow, circuits and discretised transport. Solver convergence does not validate boundary conditions, mesh resolution or physical assumptions.

For parents and students, the transferable lesson is to search inside a growing space of combined past directions and choose the combination that best satisfies the whole system.

A responsible report states matrix size and properties, start, Arnoldi variant, restart length, preconditioner side, tolerance, true residual, iterations, precision, memory and breakdown handling.

---

Frequently asked questions

What is GMRES?

A Krylov method that minimizes Euclidean residual norm at each unrestarted step.

Why Arnoldi iteration?

It builds an orthonormal basis and a small Hessenberg relation.

Can GMRES solve nonsymmetric systems?

Yes, subject to numerical and modelling conditions.

Why restart?

To limit storage and orthogonalisation cost, with possible convergence loss.

Is a small recurrence residual enough?

Check the true residual b-Ax as well.

What must be reported?

Matrix, start, Arnoldi variant, restart, preconditioner, tolerance, precision, memory and true residual.

---

Primary source and fact-check note

Youcef Saad and Martin H. Schultz, “GMRES: A Generalized Minimal Residual Algorithm for Solving Nonsymmetric Linear Systems,” SIAM Journal on Scientific and Statistical Computing 7, 1986. Definitions and core claims in this article were checked against this primary publication. Calculations shown here were independently recomputed. Examples are educational illustrations; they are not reported industrial performance results.

Useful next reading

• Conjugate gradient method
• Incomplete Cholesky preconditioning
• Randomized Kaczmarz
• How Mathematics Works
• Study and learning methods

Keep the options open. Mathematics education supports clearer questions, transferable problem-solving skills and informed choices across science, engineering, computing, AI, statistics, design and many other careers. It does not guarantee intelligence, income, admission or employment; those outcomes depend on far more than one subject or model.

Method audit: structure, recurrence and numerical evidence

Structure audit

Verify the matrix class and eligible method.

Recurrence audit

Recompute every scalar and vector.

Residual audit

Check b-Ax or the original least-squares residual.

Cost audit

Count products, storage, orthogonalisation and preconditioner work.

Mechanism workshop: calculate, challenge and transfer

Manual reconstruction

Build the smallest matrix example and label dimensions, norms, subspaces and assumptions.

Assumption map

Place symmetry, nonsymmetry, definiteness, consistency and adjoint conditions beside the step they permit.

Counterexample

Remove one condition and locate the first failed conclusion.

Sensitivity

Change one matrix entry, tolerance, restart or damping parameter.

Independent verification

Recompute the iterate and true residual directly.

Precision experiment

Round intermediate scalars and measure drift.

Cost experiment

Count products, stored vectors and orthogonalisation.

Engineering transfer

Connect the mechanism to a sparse physical system.

Student explanation

Explain the search space without promising success.

Parent explanation

Describe iterative correction with a whole-problem check.

Evidence notebook: reconstruct every step

Evidence notebook 1: System

Step 1 isolates system. Use A=[[1,1],[0,2]], b=(0,1). The checked result is: Exact x=(-1/2,1/2). Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: A is nonsymmetric. Record system. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Evidence notebook 2: Start

Step 2 isolates start. Set x0=0. The checked result is: r0=b and beta=1. Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: Use true residual. Record start. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Evidence notebook 3: First basis

Step 3 isolates first basis. Normalize r0. The checked result is: q1=(0,1). Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: Keep unit norm. Record q1. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Evidence notebook 4: Multiply

Step 4 isolates multiply. Compute Aq1. The checked result is: It is (1,2). Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: Use original A. Record vector. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Evidence notebook 5: Project

Step 5 isolates project. Compute h11. The checked result is: h11=2. Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: Dot with q1. Record entry. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Evidence notebook 6: Orthogonalise

Step 6 isolates orthogonalise. Subtract 2q1. The checked result is: Remainder=(1,0). Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: Check orthogonality. Record w. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Evidence notebook 7: Next basis

Step 7 isolates next basis. Normalize remainder. The checked result is: h21=1,q2=(1,0). Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: No breakdown. Record basis. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Evidence notebook 8: Reduced solve

Step 8 isolates reduced solve. Minimize ||(1,0)-y(2,1)||. The checked result is: y=2/5. Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: Use least squares. Record y. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Evidence notebook 9: Iterate

Step 9 isolates iterate. Set x1=q1 y. The checked result is: x1=(0,2/5). Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: Map back correctly. Record x1. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Evidence notebook 10: Residual

Step 10 isolates residual. Compute b-Ax1. The checked result is: (-2/5,1/5), norm 1/sqrt5. Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: Check directly. Record residual. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Evidence notebook 11: Second space

Step 11 isolates second space. Use q1,q2. The checked result is: It spans R2. Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: Exact arithmetic. Record dimension. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Evidence notebook 12: Finish

Step 12 isolates finish. Solve within K2. The checked result is: x2=(-1/2,1/2). Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: Verify Ax=b. Record solution. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Evidence notebook 13: Restart

Step 13 isolates restart. Discard basis after m steps. The checked result is: Memory falls. Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: Convergence can change. Record m. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Evidence notebook 14: Conclusion

Step 14 isolates conclusion. State Arnoldi,restart,preconditioner,tolerance. The checked result is: Residual minimized each full step. Preserve dimensions, matrix class, norms, preconditioning, scaling and units beside every number. The safeguard is: Keep true-residual check. Write conclusion. Change one input, recompute the reduced problem, iterate and true residual, and explain whether the change comes from subspace geometry, nonnormality, indefiniteness, biorthogonality, damping, conditioning or rounding. Ask a second reader to reproduce the arithmetic. Finish with the narrow conclusion and remaining evidence.

Failure cases and responsible recovery

Eigenvalue-only forecast

Nonnormality matters. Build a counterexample, restore the missing condition and recalculate before making the claim.

Restart-is-free

Discarded bases can slow convergence. Build a counterexample, restore the missing condition and recalculate before making the claim.

Recursive-only residual

Check b-Ax. Build a counterexample, restore the missing condition and recalculate before making the claim.

Hidden preconditioner side

Residual meaning changes. Build a counterexample, restore the missing condition and recalculate before making the claim.

No orthogonality audit

Arnoldi can lose orthogonality. Build a counterexample, restore the missing condition and recalculate before making the claim.

Exact n-step claim

Roundoff matters. Build a counterexample, restore the missing condition and recalculate before making the claim.

Memory omission

Full GMRES storage grows. Build a counterexample, restore the missing condition and recalculate before making the claim.

Model leap

A solved system can encode wrong physics. Build a counterexample, restore the missing condition and recalculate before making the claim.

Transfer checks: from worked example to trustworthy use

Definition check

Name the mathematical object, update and target norm without software vocabulary.

Units check

Attach units and rescale one variable.

Baseline check

Compare with a direct solve or simpler eligible iteration using the same tolerance.

Stopping check

Test the stopping rule on a misleading local state.

Reproducibility check

Record ordering, seed where relevant, precision, preconditioner and transformations.

Boundary check

Construct a nearly singular, nearly breakdown or badly scaled case and choose a safeguard.

Decision check

State when this method fits and what evidence would make you switch.

Communication check

Write student, parent and technical explanations with the same limit statement.

Verified case study: ten steps from matrix to decision

1. Define the decision

State why GMRES is being considered, what matrix or operator is available, which residual norm matters, and what downstream decision uses the solution. Keep solver accuracy separate from model accuracy and policy judgement.

2. Verify the operator

Check dimensions, sparsity, symmetry claims, transpose or adjoint consistency, finite entries and scaling. Use small probe vectors and inner-product identities. A file name, software type or documentation label is not sufficient evidence.

3. Establish a baseline

Solve a small representative instance with a trusted direct method or high-accuracy reference. Compare the iterative result, true residual and application quantity. The baseline tests implementation logic; it does not predict production runtime by itself.

4. Choose the starting state

Record the initial vector and calculate its true residual before iteration. A warm start can reduce work, but it can also import stale assumptions. Explain where it came from and why it is eligible.

5. Configure the method

Name restart length, shadow residual, damping, Lanczos convention, orthogonalisation and preconditioner as applicable. Put each choice beside the mathematical effect it creates instead of presenting a collection of unexplained tuning values.

6. Monitor the path

Log recursive and periodically recomputed true residuals, iteration counts, matrix-vector products, detected breakdown quantities and elapsed time. A smooth plot is useful evidence, but it is not a theorem or a certificate of correct modelling.

7. Challenge the solution

Perturb data within credible measurement or rounding limits, change the start and test a second tolerance. Explain whether the final application conclusion is stable. Numerical convergence without sensitivity analysis can create false confidence.

8. Compare total cost

Include setup, preprocessing, factor construction, storage, communication, orthogonalisation, transpose products and final verification. The method with fewer iterations can take longer, while a method with a larger basis can be more robust.

9. Review interpretation

Translate residual size into the units and risk of the application. Ask a subject expert to review boundary conditions, noise assumptions and acceptable error. Mathematics supports the decision without replacing domain responsibility.

10. Preserve the evidence

Save the matrix or operator version, right-hand side, code version, precision, parameters, logs and verification calculations. Write one sentence stating what the result supports and one stating what it cannot establish.

11. Test a failure mode

Create a controlled example that violates one assumption: lose symmetry, make a denominator tiny, supply a wrong adjoint or restart too aggressively. Observe the diagnostic that changes first and document the recovery route.

12. Check numerical precision

Repeat a small run in higher precision and compare residual histories, orthogonality and final application quantities. A visible difference does not automatically make the lower-precision run unusable, but it identifies evidence that must be judged.

13. Explain the tolerance

Convert the stopping tolerance into an absolute discrepancy and, where possible, a bound or sensitivity statement for the desired output. Avoid choosing a tiny number only because it looks rigorous; unattainable tolerances can waste work or expose rounding limits.

14. Close the loop

Recompute the original equations or least-squares objective from the saved final vector, independently of the recurrence. Confirm the public conclusion against this check and keep the complete evidence available for later reproduction.