Chapter map

1. 1. Rows
2. 2. Reference matrix
3. 3. Rotation
4. 4. Scale
5. 5. Cross matrix
6. 6. Cross norm
7. 7. X self norm
8. 8. Y self norm
9. 9. Linear CKA
10. 10. Contrasting Z
11. 11. Contrasting CKA
12. 12. Anisotropic transform
13. 13. Kernel CKA
14. 14. Layer sweep
15. 15. Stability
16. 16. Claim

Visit the How Mathematics Works hub for the wider series and its connections to education, reasoning, science, computing, design and careers.

---

1. Rows

A small example makes the mechanism visible. Use the same four observations in X and Y.

A concrete situation

Representations align by example.

What the mathematics does not license

CKA does not repair mismatched rows. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record IDs. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

2. Reference matrix

This is where modelling choices begin to affect evidence. Use X rows (1,0),(-1,0),(0,1),(0,-1).

A concrete situation

Both columns are centred.

What the mathematics does not license

Centre after any train-only preprocessing. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record X. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

3. Rotation

The central mathematical idea is concrete. Rotate X by 45 degrees.

A concrete situation

Feature coordinates change.

What the mathematics does not license

Use an orthogonal matrix. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record R. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

4. Scale

Here the geometry becomes operational. Multiply the rotated matrix by two.

A concrete situation

Y=2XR remains centred.

What the mathematics does not license

This is isotropic scaling. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record Y. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

5. Cross matrix

This step deserves a careful pause. Compute XᵀY.

A concrete situation

Entries are ±2.828427 in the example.

What the mathematics does not license

Keep full precision. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record matrix. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

6. Cross norm

The useful question is not only how to calculate, but what the calculation assumes. Square and sum its entries.

A concrete situation

||XᵀY||²_F=32.

What the mathematics does not license

The numerator is a squared norm. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record numerator. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

7. X self norm

A small example makes the mechanism visible. Compute ||XᵀX||_F.

A concrete situation

Value=2.828427.

What the mathematics does not license

Do not square it again in the denominator. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record value. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

8. Y self norm

This is where modelling choices begin to affect evidence. Compute ||YᵀY||_F.

A concrete situation

Value=11.313708.

What the mathematics does not license

Scale affects this term. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record value. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

9. Linear CKA

The central mathematical idea is concrete. Divide 32 by the norm product.

A concrete situation

CKA=1.

What the mathematics does not license

Roundoff may give 1+tiny epsilon. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Clamp only for display. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

10. Contrasting Z

Here the geometry becomes operational. Use z=[1,1,-1,-1] as one centred feature.

A concrete situation

Xᵀz=(0,0).

What the mathematics does not license

The observation order is unchanged. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record z. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

11. Contrasting CKA

This step deserves a careful pause. The numerator is zero.

A concrete situation

Linear CKA(X,z)=0.

What the mathematics does not license

Zero sample alignment is not universal independence. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record result. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

12. Anisotropic transform

The useful question is not only how to calculate, but what the calculation assumes. Stretch only one feature direction.

A concrete situation

CKA can fall below one.

What the mathematics does not license

CKA is not invariant to every invertible transform. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Explain purpose. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

13. Kernel CKA

A small example makes the mechanism visible. Replace linear Gram matrices with declared kernels.

A concrete situation

Nonlinear similarity can be compared.

What the mathematics does not license

Kernel parameters change the question. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Report settings. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

14. Layer sweep

This is where modelling choices begin to affect evidence. Compare prespecified layers.

A concrete situation

A similarity map can reveal correspondences.

What the mathematics does not license

Many comparisons need multiplicity awareness. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record matrix. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

15. Stability

The central mathematical idea is concrete. Repeat across seeds and held-out examples.

A concrete situation

CKA becomes a distribution.

What the mathematics does not license

One training run is not general evidence. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Summarise. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

16. Claim

Here the geometry becomes operational. State centring, rows, kernel and sample.

A concrete situation

Rotated, uniformly scaled Y has linear CKA one with X.

What the mathematics does not license

Do not claim identical computations. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Bound conclusion. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

A seven-day practice lab

1. Day 1: define the decision, observation unit, X variables and Y or reference condition.
2. Day 2: reproduce the smallest worked example and check every dimension, unit and sign.
3. Day 3: build a transparent baseline and write what it assumes.
4. Day 4: fit the main method inside a training-only pipeline; record every parameter.
5. Day 5: challenge one assumption with a deliberately difficult dataset or operating regime.
6. Day 6: compare validation metrics with the practical workload created by errors.
7. Day 7: present one supported conclusion, one unresolved uncertainty and one safe next experiment.

Questions for a student, parent or mentor

• Which part of the model is a mathematical definition, and which part is a choice?
• What small calculation would catch a sign, scale or dimension error?
• What evidence comes from training, validation and the untouched test?
• What benign condition could look like a fault or prediction failure?
• How would the answer change for another response, asset, school project or time period?
• What should a human review before acting on the score?

A method-specific audit before deployment

Centered Kernel Alignment, or CKA, compares representational geometries through centred Gram matrices. Linear CKA can be written as ||XᵀY||²_F divided by ||XᵀX||_F||YᵀY||_F when rows are aligned observations and feature columns are centred. It is especially useful when two neural-network layers use different coordinate bases.

Let X have centred rows (1,0),(-1,0),(0,1),(0,-1). Rotate X by 45 degrees and multiply every feature by two to obtain Y. The cross term ||XᵀY||²_F is 32. The self norms are ||XᵀX||_F=2.828427 and ||YᵀY||_F=11.313708.

Linear CKA is therefore 32/(2.828427×11.313708)=1. A contrasting centred one-feature representation z=[1,1,-1,-1] has Xᵀz=(0,0), so its linear CKA with X is zero. The example makes the invariances visible: isotropic scaling and orthogonal feature rotation leave linear CKA unchanged.

CKA is not invariant to every invertible linear transformation, and that limitation is intentional in the motivating work. High CKA does not prove equal units, identical computations, equal predictions or causal mechanism. Observation alignment, centring, sample size, layer width, kernels, preprocessing, training leakage and uncertainty all matter.

---

Frequently asked questions

What is centered kernel alignment?

A normalized similarity between centred Gram matrices; linear CKA has an equivalent feature-space formula.

Why centre?

Centring removes feature-space means so alignment concerns relational variation around them.

What leaves linear CKA unchanged?

Orthogonal feature transformations and isotropic scaling, with aligned rows and the stated formula.

Is CKA invariant to every invertible transform?

No, and the motivating work explains why complete invariance can be undesirable in high dimensions.

Does CKA one mean two networks are identical?

No. It indicates maximal alignment under the measure, not equal units, functions or mechanisms.

What must be reported?

Rows, layers, preprocessing, centring, kernel, parameters, estimator, sample split, seeds, uncertainty and multiple-comparison plan.

---

Primary source and fact-check note

Simon Kornblith, Mohammad Norouzi, Honglak Lee and Geoffrey Hinton, “Similarity of Neural Network Representations Revisited,” Proceedings of Machine Learning Research 97, 3519–3529 (2019). Definitions and core claims in this article were checked against this primary publication. Calculations shown here were independently recomputed. Examples are educational illustrations; they are not reported industrial performance results.

Useful next reading

• Representational Similarity Analysis
• RV coefficient
• Tucker congruence coefficient
• Procrustes analysis
• Bukit Timah Mathematics routes

Keep the options open. Mathematics education supports clearer questions, transferable problem-solving skills and informed choices across science, engineering, computing, AI, statistics, design and many other careers. It does not guarantee intelligence, income, admission or employment; those outcomes depend on far more than one subject or model.

Extended laboratory: dependence, alignment and relational evidence

Row audit

Confirm identical observation order and held-out status across every representation.

Centring audit

Apply one declared centring convention and verify centred feature or Gram sums.

Invariance audit

Test rotation, isotropic scaling and an anisotropic transformation with known outcomes.

Stability audit

Repeat across examples, seeds, layers and training splits rather than presenting one attractive heat map.

Mechanism workshop: calculate, challenge and transfer

Manual reconstruction

Rebuild the smallest example and label every row, matrix and normalization used for representation alignment.

One-row experiment

Change one observation while holding preprocessing and formula fixed. Recalculate all affected pairwise entries and the final statistic.

Geometry experiment

Change the distance, kernel or representation scale for a stated reason. Explain which question changed.

Permutation experiment

Shuffle only valid exchangeable units. Compare one illustrative shuffle with a complete or Monte Carlo null distribution.

Invariance experiment

Apply translation, rotation, isotropic scaling or relabelling where relevant. Predict the statistic before calculating it.

Stability experiment

Repeat across resamples, seeds, bandwidths, regularization values or solver starts and report a distribution.

Student transfer

Apply the method to handwriting, sensor readings, study patterns, image features or map distances using real identifiers.

Parent explanation

Explain why a straight-line correlation can miss a curve, why similarity kernels need a scale, or why two maps can match without shared coordinates.

Limit statement

Write one supported claim, one overclaim and one repaired claim that names data, estimator and uncertainty.

Next measurement

Choose a complementary predictive, local, causal or substantive check and explain the separate evidence it supplies.

Evidence notebook: reconstruct every mathematical step

Evidence notebook 1: Rows

Step 1 isolates rows. Use the same four observations in X and Y. The reproducible result is: Representations align by example. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: CKA does not repair mismatched rows. Record IDs. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 2: Reference matrix

Step 2 isolates reference matrix. Use X rows (1,0),(-1,0),(0,1),(0,-1). The reproducible result is: Both columns are centred. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: Centre after any train-only preprocessing. Record X. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 3: Rotation

Step 3 isolates rotation. Rotate X by 45 degrees. The reproducible result is: Feature coordinates change. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: Use an orthogonal matrix. Record R. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 4: Scale

Step 4 isolates scale. Multiply the rotated matrix by two. The reproducible result is: Y=2XR remains centred. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: This is isotropic scaling. Record Y. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 5: Cross matrix

Step 5 isolates cross matrix. Compute XᵀY. The reproducible result is: Entries are ±2.828427 in the example. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: Keep full precision. Record matrix. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 6: Cross norm

Step 6 isolates cross norm. Square and sum its entries. The reproducible result is: ||XᵀY||²_F=32. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: The numerator is a squared norm. Record numerator. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 7: X self norm

Step 7 isolates x self norm. Compute ||XᵀX||_F. The reproducible result is: Value=2.828427. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: Do not square it again in the denominator. Record value. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 8: Y self norm

Step 8 isolates y self norm. Compute ||YᵀY||_F. The reproducible result is: Value=11.313708. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: Scale affects this term. Record value. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 9: Linear CKA

Step 9 isolates linear cka. Divide 32 by the norm product. The reproducible result is: CKA=1. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: Roundoff may give 1+tiny epsilon. Clamp only for display. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 10: Contrasting Z

Step 10 isolates contrasting z. Use z=[1,1,-1,-1] as one centred feature. The reproducible result is: Xᵀz=(0,0). This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: The observation order is unchanged. Record z. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 11: Contrasting CKA

Step 11 isolates contrasting cka. The numerator is zero. The reproducible result is: Linear CKA(X,z)=0. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: Zero sample alignment is not universal independence. Record result. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 12: Anisotropic transform

Step 12 isolates anisotropic transform. Stretch only one feature direction. The reproducible result is: CKA can fall below one. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: CKA is not invariant to every invertible transform. Explain purpose. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 13: Kernel CKA

Step 13 isolates kernel cka. Replace linear Gram matrices with declared kernels. The reproducible result is: Nonlinear similarity can be compared. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: Kernel parameters change the question. Report settings. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 14: Layer sweep

Step 14 isolates layer sweep. Compare prespecified layers. The reproducible result is: A similarity map can reveal correspondences. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: Many comparisons need multiplicity awareness. Record matrix. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 15: Stability

Step 15 isolates stability. Repeat across seeds and held-out examples. The reproducible result is: CKA becomes a distribution. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: One training run is not general evidence. Summarise. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Evidence notebook 16: Claim

Step 16 isolates claim. State centring, rows, kernel and sample. The reproducible result is: Rotated, uniformly scaled Y has linear CKA one with X. This is evidence about centred representation Gram geometry and normalized alignment. Keep observation identities beside every row and write the exact formula before entering values. The safeguard is: Do not claim identical computations. Bound conclusion. Then vary one input while holding all other decisions fixed. Recalculate the matrix cells, trace, norm, coupling or statistic that must move. Explain whether the change came from data, centring, geometry, regularization or matching. A result that another reader cannot reconstruct is not ready for a scientific or practical claim. End with the narrow conclusion and name the extra evidence still needed.

Failure cases and responsible recovery

Row mismatch

A neat CKA value is meaningless when rows refer to different examples. Audit identifiers first. Rebuild a small counterexample, repeat under another defensible setting and state what evidence would change the conclusion.

Centring mismatch

Feature centring and Gram centring must follow one declared convention. Check zero sums. Rebuild a small counterexample, repeat under another defensible setting and state what evidence would change the conclusion.

Layer cherry-pick

Searching many layers and showing only the brightest cell exaggerates correspondence. Report the full comparison plan. Rebuild a small counterexample, repeat under another defensible setting and state what evidence would change the conclusion.

Seed blindness

Representation similarity can vary across training runs. Repeat seeds and data splits. Rebuild a small counterexample, repeat under another defensible setting and state what evidence would change the conclusion.

Invariance overclaim

Linear CKA ignores orthogonal rotation and isotropic scale, not arbitrary invertible transforms. Rebuild a small counterexample, repeat under another defensible setting and state what evidence would change the conclusion.

Function overclaim

High representation similarity does not guarantee identical predictions, calibration or robustness. Rebuild a small counterexample, repeat under another defensible setting and state what evidence would change the conclusion.

Sample leakage

Using evaluation examples to select models or layers contaminates later claims. Preserve held-out boundaries. Rebuild a small counterexample, repeat under another defensible setting and state what evidence would change the conclusion.

Responsible conclusion

Describe alignment of relational representation structure, not equality of neurons, models or mechanisms. Rebuild a small counterexample, repeat under another defensible setting and state what evidence would change the conclusion.