Chapter map

1. 1. Rows
2. 2. Centre
3. 3. SVD X
4. 4. Energy total
5. 5. Retained energy
6. 6. Retention ratio
7. 7. SVD Y
8. 8. Whitening
9. 9. Cross-covariance
10. 10. First pair
11. 11. Second pair
12. 12. Mean similarity
13. 13. Imperfect spectrum
14. 14. Threshold sensitivity
15. 15. Sample stability
16. 16. Claim

Visit the How Mathematics Works hub for the wider series and its connections to education, reasoning, science, computing, design and careers.

---

1. Rows

The central mathematical idea is concrete. Align the same examples in X and Y.

A concrete situation

Observation identity is shared.

What the mathematics does not license

CCA cannot repair mismatched rows. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record IDs. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

2. Centre

Here the geometry becomes operational. Subtract training-set feature means.

A concrete situation

Offsets are removed before SVD.

What the mathematics does not license

Do not centre test data with test means. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record means. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

3. SVD X

This step deserves a careful pause. Decompose X into singular vectors and values.

A concrete situation

Use singular values 3,1,.1 in the example.

What the mathematics does not license

State matrix orientation. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record spectrum. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

4. Energy total

The useful question is not only how to calculate, but what the calculation assumes. Sum squared singular values.

A concrete situation

Total=9+1+.01=10.01.

What the mathematics does not license

Energy convention must be explicit. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record denominator. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

5. Retained energy

A small example makes the mechanism visible. Keep the first two directions.

A concrete situation

Retained energy=10.

What the mathematics does not license

Threshold choice is analytical, not natural law. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record numerator. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

6. Retention ratio

This is where modelling choices begin to affect evidence. Divide 10 by 10.01.

A concrete situation

Ratio=.999001 or 99.9001%.

What the mathematics does not license

High energy retention does not guarantee task relevance. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record ratio. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

7. SVD Y

The central mathematical idea is concrete. Apply the same declared retention rule to Y.

A concrete situation

A comparable retained subspace results.

What the mathematics does not license

Ranks can differ. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record spectrum. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

8. Whitening

Here the geometry becomes operational. Scale retained coordinates for CCA.

A concrete situation

Within-view covariance becomes identity under the idealized full-rank case.

What the mathematics does not license

Regularize small eigenvalues. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record rule. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

9. Cross-covariance

This step deserves a careful pause. Form the whitened cross-view matrix.

A concrete situation

Its singular values are canonical correlations.

What the mathematics does not license

Use training observations only. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record matrix. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

10. First pair

The useful question is not only how to calculate, but what the calculation assumes. Maximise correlation for the first canonical variates.

A concrete situation

ρ1=1 in the constructed example.

What the mathematics does not license

Sign is arbitrary. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Record pair. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

11. Second pair

A small example makes the mechanism visible. Constrain orthogonality and find the next pair.

A concrete situation

ρ2=1.

What the mathematics does not license

Degenerate subspaces can rotate. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Record pair. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

12. Mean similarity

This is where modelling choices begin to affect evidence. Average retained canonical correlations.

A concrete situation

SVCCA=(1+1)/2=1.

What the mathematics does not license

Some summaries use other aggregation rules. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Name rule. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

13. Imperfect spectrum

The central mathematical idea is concrete. Use correlations .9 and .5.

A concrete situation

Simple mean=.7.

What the mathematics does not license

A mean can hide uneven directions. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Show both values. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

14. Threshold sensitivity

Here the geometry becomes operational. Keep the .1 direction and repeat.

A concrete situation

Similarity can change.

What the mathematics does not license

Report a curve, not one convenient threshold. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Stress-test. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

15. Sample stability

This step deserves a careful pause. Bootstrap examples or repeat seeds.

A concrete situation

SVCCA becomes a distribution.

What the mathematics does not license

Rows may be clustered or dependent. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Respect units. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

16. Claim

The useful question is not only how to calculate, but what the calculation assumes. State rows, centring, threshold, rank and regularization.

A concrete situation

Constructed retained subspaces are linearly equivalent.

What the mathematics does not license

Do not claim identical neurons or functions. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Bound conclusion. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

A seven-day practice lab

1. Day 1: define the decision, observation unit, X variables and Y or reference condition.
2. Day 2: reproduce the smallest worked example and check every dimension, unit and sign.
3. Day 3: build a transparent baseline and write what it assumes.
4. Day 4: fit the main method inside a training-only pipeline; record every parameter.
5. Day 5: challenge one assumption with a deliberately difficult dataset or operating regime.
6. Day 6: compare validation metrics with the practical workload created by errors.
7. Day 7: present one supported conclusion, one unresolved uncertainty and one safe next experiment.

Questions for a student, parent or mentor

• Which part of the model is a mathematical definition, and which part is a choice?
• What small calculation would catch a sign, scale or dimension error?
• What evidence comes from training, validation and the untouched test?
• What benign condition could look like a fault or prediction failure?
• How would the answer change for another response, asset, school project or time period?
• What should a human review before acting on the score?

A method-specific audit before deployment

Singular Vector Canonical Correlation Analysis, or SVCCA, compares two neural representations in two deliberate stages. Singular value decomposition first keeps directions that carry a declared share of representation energy. Canonical correlation analysis then finds paired linear combinations with maximum correlations across the two retained subspaces.

Consider a representation X with orthogonal latent directions whose singular values are 3, 1 and .1. Total squared singular-value energy is 9+1+.01=10.01. Keeping the first two directions retains 10/10.01=.999001, or about 99.9001 per cent, while removing the tiny third direction.

Construct Y as an invertible rotation and rescaling of those same two retained latent directions. After the truncation, canonical correlation recovers two perfectly paired directions with correlations one and one. Their mean gives SVCCA similarity one in this deliberately exact example. If two retained correlations were .9 and .5, the simple mean would be .7.

Perfect agreement in a constructed example does not make the method automatic. Rows must describe the same observations, centring and sample split must match, and SVD thresholds, rank, regularization and numerical precision affect results. SVCCA is insensitive to invertible linear changes inside the retained subspace, so a high value describes shared linear subspace structure rather than identical neurons, computations or predictions.

---

Frequently asked questions

What is SVCCA?

A two-stage comparison using singular-value reduction followed by canonical correlation analysis.

Why use SVD first?

It removes low-energy directions and stabilizes the subspace comparison under a declared threshold.

What is the SVCCA score?

Usually an aggregation of canonical correlations between the retained subspaces; the rule must be stated.

Is it invariant to neuron permutations?

Within the supported retained subspace, invertible linear reparameterizations can preserve canonical correlations.

Does one mean identical networks?

No. It means perfect retained linear-subspace correlation in the observed sample.

What must be reported?

Rows, centring, orientation, energy rule, retained ranks, regularization, aggregation, sample split, layers, seeds and uncertainty.

---

Primary source and fact-check note

Maithra Raghu, Justin Gilmer, Jason Yosinski and Jascha Sohl-Dickstein, “SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability,” Advances in Neural Information Processing Systems 30 (2017). Definitions and core claims in this article were checked against this primary publication. Calculations shown here were independently recomputed. Examples are educational illustrations; they are not reported industrial performance results.

Useful next reading

• Canonical Correlation Analysis
• Centered Kernel Alignment
• Procrustes analysis
• Representational Similarity Analysis
• Bukit Timah Mathematics routes

Keep the options open. Mathematics education supports clearer questions, transferable problem-solving skills and informed choices across science, engineering, computing, AI, statistics, design and many other careers. It does not guarantee intelligence, income, admission or employment; those outcomes depend on far more than one subject or model.

Extended laboratory: representation evidence and responsible comparison

Identity audit

Confirm the same observation rows, preprocessing boundaries and layer checkpoints in both representations.

SVD audit

Publish singular values, energy convention, threshold and retained rank for each view.

CCA audit

Record whitening, ridge regularization, rank tolerance and the full canonical-correlation spectrum.

Stability audit

Repeat thresholds, examples, checkpoints and seeds and report variation rather than one score.

Mechanism workshop: calculate, challenge and transfer

Manual reconstruction

Rebuild the smallest example for SVD retention and canonical subspaces; label observations, matrices, norms and every convention.

One-entry experiment

Change one similarity, activation, singular value or eigenvalue while freezing every other decision. Recalculate all affected terms.

Invariance experiment

Apply a feature permutation, orthogonal rotation, isotropic scaling or invertible subspace transform and predict the result before computing it.

Rank experiment

Remove or add a low-energy direction and explain whether the score changes because of signal, regularization or threshold policy.

Orientation experiment

Swap the two representations. Identify which measures remain unchanged and why PWCCA can move.

Validation experiment

Choose a method on training observations and test the frozen choice on held-out examples without reusing targets.

Student transfer

Compare handwriting features, image layers, language embeddings or sensor representations while preserving row identities.

Parent explanation

Describe how two maps of the same examples can use different coordinates yet preserve similar relationships.

Limit statement

Write one supported claim, one tempting overclaim and one repaired claim that names sample, method and uncertainty.

Next measurement

Choose a complementary predictive, causal, local or domain check and explain the separate evidence it supplies.

Evidence notebook: reconstruct every mathematical step

Evidence notebook 1: Rows

Step 1 isolates rows. Align the same examples in X and Y. The reproducible result is: Observation identity is shared. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: CCA cannot repair mismatched rows. Record IDs. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 2: Centre

Step 2 isolates centre. Subtract training-set feature means. The reproducible result is: Offsets are removed before SVD. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: Do not centre test data with test means. Record means. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 3: SVD X

Step 3 isolates svd x. Decompose X into singular vectors and values. The reproducible result is: Use singular values 3,1,.1 in the example. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: State matrix orientation. Record spectrum. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 4: Energy total

Step 4 isolates energy total. Sum squared singular values. The reproducible result is: Total=9+1+.01=10.01. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: Energy convention must be explicit. Record denominator. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 5: Retained energy

Step 5 isolates retained energy. Keep the first two directions. The reproducible result is: Retained energy=10. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: Threshold choice is analytical, not natural law. Record numerator. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 6: Retention ratio

Step 6 isolates retention ratio. Divide 10 by 10.01. The reproducible result is: Ratio=.999001 or 99.9001%. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: High energy retention does not guarantee task relevance. Record ratio. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 7: SVD Y

Step 7 isolates svd y. Apply the same declared retention rule to Y. The reproducible result is: A comparable retained subspace results. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: Ranks can differ. Record spectrum. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 8: Whitening

Step 8 isolates whitening. Scale retained coordinates for CCA. The reproducible result is: Within-view covariance becomes identity under the idealized full-rank case. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: Regularize small eigenvalues. Record rule. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 9: Cross-covariance

Step 9 isolates cross-covariance. Form the whitened cross-view matrix. The reproducible result is: Its singular values are canonical correlations. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: Use training observations only. Record matrix. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 10: First pair

Step 10 isolates first pair. Maximise correlation for the first canonical variates. The reproducible result is: ρ1=1 in the constructed example. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: Sign is arbitrary. Record pair. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 11: Second pair

Step 11 isolates second pair. Constrain orthogonality and find the next pair. The reproducible result is: ρ2=1. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: Degenerate subspaces can rotate. Record pair. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 12: Mean similarity

Step 12 isolates mean similarity. Average retained canonical correlations. The reproducible result is: SVCCA=(1+1)/2=1. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: Some summaries use other aggregation rules. Name rule. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 13: Imperfect spectrum

Step 13 isolates imperfect spectrum. Use correlations .9 and .5. The reproducible result is: Simple mean=.7. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: A mean can hide uneven directions. Show both values. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 14: Threshold sensitivity

Step 14 isolates threshold sensitivity. Keep the .1 direction and repeat. The reproducible result is: Similarity can change. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: Report a curve, not one convenient threshold. Stress-test. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 15: Sample stability

Step 15 isolates sample stability. Bootstrap examples or repeat seeds. The reproducible result is: SVCCA becomes a distribution. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: Rows may be clustered or dependent. Respect units. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Evidence notebook 16: Claim

Step 16 isolates claim. State rows, centring, threshold, rank and regularization. The reproducible result is: Constructed retained subspaces are linearly equivalent. This is evidence about retained canonical subspace correlation. Keep observation identifiers beside every matrix and write the exact formula before entering numbers. The safeguard is: Do not claim identical neurons or functions. Bound conclusion. Next change one input while holding centring, rank, normalization and sample fixed. Recalculate the cells, singular values, canonical correlations, weights or matrix roots that must move. Explain whether the change came from data geometry, numerical stabilization or an analytical choice. Finish with the narrow claim supported by this step and the additional evidence still needed.

Failure cases and responsible recovery

Row mismatch

Canonical correlation becomes meaningless when examples are not aligned. Rebuild a counterexample, repeat under another defensible setting and state what additional evidence would repair the claim.

Test centring

Using evaluation means leaks information across the boundary. Rebuild a counterexample, repeat under another defensible setting and state what additional evidence would repair the claim.

Threshold opacity

A hidden SVD energy rule prevents reproduction and can change the score. Rebuild a counterexample, repeat under another defensible setting and state what additional evidence would repair the claim.

Rank instability

Nearly singular covariance produces exaggerated canonical correlations. Rebuild a counterexample, repeat under another defensible setting and state what additional evidence would repair the claim.

Mean-only reporting

One average hides a weak tail of canonical directions. Rebuild a counterexample, repeat under another defensible setting and state what additional evidence would repair the claim.

Layer cherry-pick

Searching every layer and showing one maximum exaggerates correspondence. Rebuild a counterexample, repeat under another defensible setting and state what additional evidence would repair the claim.

Neuron identity claim

Subspace equivalence does not mean unit-by-unit equality. Rebuild a counterexample, repeat under another defensible setting and state what additional evidence would repair the claim.

Function claim

High SVCCA does not ensure equal outputs or errors. Rebuild a counterexample, repeat under another defensible setting and state what additional evidence would repair the claim.