Chapter map

1. 1. Model
2. 2. Target
3. 3. Ordinary derivative
4. 4. Probability Fisher
5. 5. Inverse metric
6. 6. Natural direction p
7. 7. Logit coordinate
8. 8. Logit derivative
9. 9. Logit Fisher
10. 10. Natural direction η
11. 11. Map back
12. 12. Damping
13. 13. Approximation
14. 14. Empirical Fisher
15. 15. Validation
16. 16. Claim

Visit the How Mathematics Works hub for the wider series and its connections to education, reasoning, science, computing, design and careers.

---

1. Model

This step deserves a careful pause. Use a Bernoulli probability p=.8.

A concrete situation

The model assigns probabilities p and 1−p.

What the mathematics does not license

Keep p inside (0,1). A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record parameterization. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

2. Target

The useful question is not only how to calculate, but what the calculation assumes. Use q=.5 in L=½(p−q)².

A concrete situation

The teaching objective has a transparent derivative.

What the mathematics does not license

This is illustrative, not Bernoulli log-likelihood training. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

State purpose. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

3. Ordinary derivative

A small example makes the mechanism visible. Compute p−q.

A concrete situation

g_p=.3.

What the mathematics does not license

Gradient magnitude depends on coordinate choice. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record g_p. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

4. Probability Fisher

This is where modelling choices begin to affect evidence. Compute 1/[p(1−p)].

A concrete situation

F_p=6.25.

What the mathematics does not license

Use expected model Fisher for one observation. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record definition. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

5. Inverse metric

The central mathematical idea is concrete. Take F_p⁻¹=.16.

A concrete situation

The metric rescales the coordinate gradient.

What the mathematics does not license

A singular Fisher needs regularization or a pseudoinverse. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record inverse. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

6. Natural direction p

Here the geometry becomes operational. Multiply .16 by .3.

A concrete situation

Direction=.048.

What the mathematics does not license

The update uses a separate learning rate and sign. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record direction. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

7. Logit coordinate

This step deserves a careful pause. Set η=log[p/(1−p)].

A concrete situation

η=log4≈1.386294.

What the mathematics does not license

Finite steps are not exactly coordinate invariant. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record η. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

8. Logit derivative

The useful question is not only how to calculate, but what the calculation assumes. Use dp/dη=p(1−p)=.16.

A concrete situation

g_η=.3×.16=.048.

What the mathematics does not license

Apply the chain rule once. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record derivative. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

9. Logit Fisher

A small example makes the mechanism visible. Transform the Fisher metric.

A concrete situation

F_η=.16.

What the mathematics does not license

Do not reuse F_p numerically. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record metric. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

10. Natural direction η

This is where modelling choices begin to affect evidence. Divide .048 by .16.

A concrete situation

Direction=.3.

What the mathematics does not license

This is a tangent-coordinate component. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record value. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

11. Map back

The central mathematical idea is concrete. Multiply by dp/dη=.16.

A concrete situation

Infinitesimal p direction=.048.

What the mathematics does not license

Compare tangent changes, not raw coordinate numbers. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Verify agreement. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

12. Damping

Here the geometry becomes operational. Replace F by F+λI when necessary.

A concrete situation

The step becomes smaller and numerically stable.

What the mathematics does not license

Damping weakens exact invariance. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Report λ. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

13. Approximation

This step deserves a careful pause. Use diagonal or block structure for large models.

A concrete situation

Memory and solves become practical.

What the mathematics does not license

Approximation changes geometry. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Name blocks. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

14. Empirical Fisher

The useful question is not only how to calculate, but what the calculation assumes. Compare gradient outer products with expected Fisher.

A concrete situation

They need not be equal.

What the mathematics does not license

Do not substitute silently. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Report estimator. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

15. Validation

A small example makes the mechanism visible. Evaluate frozen optimizer settings on held-out data.

A concrete situation

Training speed and generalization are separated.

What the mathematics does not license

One loss curve is insufficient. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Repeat seeds. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

16. Claim

This is where modelling choices begin to affect evidence. State model, Fisher, approximation and damping.

A concrete situation

Both coordinates give the same infinitesimal model direction in the example.

What the mathematics does not license

Do not promise universal superiority. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Bound conclusion. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

A seven-day practice lab

1. Day 1: define the decision, observation unit, X variables and Y or reference condition.
2. Day 2: reproduce the smallest worked example and check every dimension, unit and sign.
3. Day 3: build a transparent baseline and write what it assumes.
4. Day 4: fit the main method inside a training-only pipeline; record every parameter.
5. Day 5: challenge one assumption with a deliberately difficult dataset or operating regime.
6. Day 6: compare validation metrics with the practical workload created by errors.
7. Day 7: present one supported conclusion, one unresolved uncertainty and one safe next experiment.

Questions for a student, parent or mentor

• Which part of the model is a mathematical definition, and which part is a choice?
• What small calculation would catch a sign, scale or dimension error?
• What evidence comes from training, validation and the untouched test?
• What benign condition could look like a fault or prediction failure?
• How would the answer change for another response, asset, school project or time period?
• What should a human review before acting on the score?

A method-specific audit before deployment

Natural gradient changes the geometry of an optimization step. An ordinary gradient depends on the numerical coordinates used to describe a statistical model. The natural gradient measures a step with the Fisher information metric, so two parameterizations that describe the same nearby probability distribution produce the same infinitesimal model change under the ideal theory.

Take a Bernoulli model with probability p=.8 and a teaching objective L(p)=½(p−q)² with q=.5. The ordinary derivative is g_p=.3. Fisher information in the probability coordinate is F_p=1/[p(1−p)]=1/.16=6.25, so the natural direction is F_p⁻¹g_p=.16×.3=.048.

Now use the logit η, where p=σ(η). The derivative is g_η=(p−q)p(1−p)=.3×.16=.048, while Fisher information is F_η=p(1−p)=.16. The natural direction in η is .048/.16=.3. Mapping that infinitesimal direction back through dp/dη=.16 gives .16×.3=.048, agreeing with the probability-coordinate calculation.

This exact local example does not remove practical difficulty. Real Fisher matrices can be huge, singular or poorly conditioned; damping, minibatch estimates, empirical-Fisher substitutions and block approximations change the update. Report the statistical model, Fisher definition, coordinate system, approximation, damping, solver, learning rate, sample split and held-out performance. Natural gradient can improve conditioning without guaranteeing the best optimizer or generalization.

---

Frequently asked questions

What is natural gradient?

A gradient measured with the Fisher information metric on a statistical model family.

Why is the ordinary gradient coordinate-dependent?

Its components and Euclidean length change when the same model is described by new parameters.

What does Fisher information do?

It measures local distinguishability of nearby model distributions under regularity conditions.

Is natural gradient exactly invariant for finite steps?

Its ideal property is infinitesimal; finite steps, damping and approximations weaken it.

Is the empirical Fisher always the Fisher?

No. They are different estimators and can behave differently.

What must be reported?

Model, Fisher definition, coordinate system, approximation, damping, solver, learning rate, batch, seeds and held-out evidence.

---

Primary source and fact-check note

Shun-ichi Amari, “Natural Gradient Works Efficiently in Learning,” Neural Computation 10(2), 251–276 (1998), DOI 10.1162/089976698300017746. Definitions and core claims in this article were checked against this primary publication. Calculations shown here were independently recomputed. Examples are educational illustrations; they are not reported industrial performance results.

Useful next reading

• K-FAC Fisher blocks and damping
• Kernel–target alignment
• How Mathematical Stability Works
• How Mathematics Works
• eduKateSingapore learning routes

Keep the options open. Mathematics education supports clearer questions, transferable problem-solving skills and informed choices across science, engineering, computing, AI, statistics, design and many other careers. It does not guarantee intelligence, income, admission or employment; those outcomes depend on far more than one subject or model.

Extended laboratory: geometry, curvature and trainability evidence

Model audit

Specify the probability model and confirm the score expectation and Fisher regularity assumptions.

Coordinate audit

Repeat one infinitesimal calculation under a second parameterization and compare model-space change.

Approximation audit

Separate exact, empirical, diagonal, block and Kronecker-factored Fisher matrices.

Optimizer audit

Publish damping, solver residuals, learning rates, seeds and held-out curves.

Mechanism workshop: calculate, challenge and transfer

Manual reconstruction

Rebuild the smallest example for Fisher-metric learning directions. Label parameters, matrices, spectra, units and every approximation.

Coordinate experiment

Reparameterize the same model and predict which raw numbers change and which ideal geometric quantity should agree.

Damping experiment

Sweep a small set of declared ridge values. Record condition number, step norm, predicted reduction and actual reduction.

Direction experiment

Apply the matrix or Jacobian to basis vectors and random probes. Connect the outputs to eigenvectors or singular vectors.

Depth experiment

Compose several simple layers or steps and calculate how mild anisotropy accumulates.

Estimator experiment

Compare an exact tiny matrix with matrix-vector, Lanczos or block approximations and quantify error.

Student transfer

Use a two-parameter classifier, sensor calibration model or small neural network with held-out rows.

Parent explanation

Explain why a learning step should be measured by the model change it causes, not only the number written in one coordinate.

Limit statement

Write one supported claim, one attractive overclaim and one repaired claim naming model, sample and approximation.

Next measurement

Choose a complementary predictive, robustness or causal check and explain its separate question.

Evidence notebook: reconstruct every mathematical step

Evidence notebook 1: Model

Step 1 isolates model. Use a Bernoulli probability p=.8. The reproducible result is: The model assigns probabilities p and 1−p. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: Keep p inside (0,1). Record parameterization. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 2: Target

Step 2 isolates target. Use q=.5 in L=½(p−q)². The reproducible result is: The teaching objective has a transparent derivative. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: This is illustrative, not Bernoulli log-likelihood training. State purpose. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 3: Ordinary derivative

Step 3 isolates ordinary derivative. Compute p−q. The reproducible result is: g_p=.3. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: Gradient magnitude depends on coordinate choice. Record g_p. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 4: Probability Fisher

Step 4 isolates probability fisher. Compute 1/[p(1−p)]. The reproducible result is: F_p=6.25. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: Use expected model Fisher for one observation. Record definition. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 5: Inverse metric

Step 5 isolates inverse metric. Take F_p⁻¹=.16. The reproducible result is: The metric rescales the coordinate gradient. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: A singular Fisher needs regularization or a pseudoinverse. Record inverse. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 6: Natural direction p

Step 6 isolates natural direction p. Multiply .16 by .3. The reproducible result is: Direction=.048. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: The update uses a separate learning rate and sign. Record direction. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 7: Logit coordinate

Step 7 isolates logit coordinate. Set η=log[p/(1−p)]. The reproducible result is: η=log4≈1.386294. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: Finite steps are not exactly coordinate invariant. Record η. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 8: Logit derivative

Step 8 isolates logit derivative. Use dp/dη=p(1−p)=.16. The reproducible result is: g_η=.3×.16=.048. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: Apply the chain rule once. Record derivative. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 9: Logit Fisher

Step 9 isolates logit fisher. Transform the Fisher metric. The reproducible result is: F_η=.16. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: Do not reuse F_p numerically. Record metric. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 10: Natural direction η

Step 10 isolates natural direction η. Divide .048 by .16. The reproducible result is: Direction=.3. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: This is a tangent-coordinate component. Record value. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 11: Map back

Step 11 isolates map back. Multiply by dp/dη=.16. The reproducible result is: Infinitesimal p direction=.048. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: Compare tangent changes, not raw coordinate numbers. Verify agreement. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 12: Damping

Step 12 isolates damping. Replace F by F+λI when necessary. The reproducible result is: The step becomes smaller and numerically stable. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: Damping weakens exact invariance. Report λ. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 13: Approximation

Step 13 isolates approximation. Use diagonal or block structure for large models. The reproducible result is: Memory and solves become practical. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: Approximation changes geometry. Name blocks. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 14: Empirical Fisher

Step 14 isolates empirical fisher. Compare gradient outer products with expected Fisher. The reproducible result is: They need not be equal. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: Do not substitute silently. Report estimator. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 15: Validation

Step 15 isolates validation. Evaluate frozen optimizer settings on held-out data. The reproducible result is: Training speed and generalization are separated. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: One loss curve is insufficient. Repeat seeds. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Evidence notebook 16: Claim

Step 16 isolates claim. State model, Fisher, approximation and damping. The reproducible result is: Both coordinates give the same infinitesimal model direction in the example. This is evidence about information-geometric direction. Preserve parameter, observation and checkpoint identities beside every matrix. The safeguard is: Do not promise universal superiority. Bound conclusion. Change one input while holding the loss, coordinate system, damping, rank and sample fixed. Recalculate every derivative, eigenvalue, singular value, solve residual or condition number that must move. Explain whether the difference came from the model, geometry, approximation or numerical rule. Finish with the narrow conclusion and the additional validation still required.

Failure cases and responsible recovery

Euclidean shortcut

A raw gradient is not invariant to reparameterization. Rebuild a small counterexample, repeat under another defensible setting and state the extra evidence that would repair the conclusion.

Fisher confusion

Expected Fisher, observed Fisher and empirical gradient outer products are different objects. Rebuild a small counterexample, repeat under another defensible setting and state the extra evidence that would repair the conclusion.

Singular inverse

Direct inversion of a rank-deficient Fisher is unstable. Rebuild a small counterexample, repeat under another defensible setting and state the extra evidence that would repair the conclusion.

Damping omission

Regularization changes both stability and geometry. Rebuild a small counterexample, repeat under another defensible setting and state the extra evidence that would repair the conclusion.

Finite-step claim

Infinitesimal invariance does not make every finite discretization identical. Rebuild a small counterexample, repeat under another defensible setting and state the extra evidence that would repair the conclusion.

Approximation hiding

Block and diagonal methods can miss cross-parameter structure. Rebuild a small counterexample, repeat under another defensible setting and state the extra evidence that would repair the conclusion.

Curve cherry-pick

One fast training curve is not robust optimizer evidence. Rebuild a small counterexample, repeat under another defensible setting and state the extra evidence that would repair the conclusion.

Generalization promise

Conditioning gains do not guarantee better held-out performance. Rebuild a small counterexample, repeat under another defensible setting and state the extra evidence that would repair the conclusion.