Chapter map

1. 1. Why interpretable tokens help
2. 2. Expression vector
3. 3. Binary mask matrix
4. 4. Masked linear weights
5. 5. Pathway token calculation
6. 6. Token matrix geometry
7. 7. CLS token
8. 8. Queries, keys and values
9. 9. Scaled dot-product attention
10. 10. Softmax attention weights
11. 11. Value aggregation
12. 12. Multiple heads
13. 13. Residual and feed-forward layers
14. 14. Cell-type softmax
15. 15. Attention-based interpretation
16. 16. Governance and careers

Visit the How Mathematics Works hub for the wider series and its connections to education, reasoning, science, computing, design and careers.

---

1. Why interpretable tokens help

Here the geometry becomes operational. Gene-level neural features are difficult to name, while pathways provide a human-readable intermediate layer.

A concrete situation

A token represents interferon signalling.

What the mathematics does not license

Gene-set names can be broad or overlapping. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Version the knowledge source. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

2. Expression vector

This step deserves a careful pause. For one cell, e contains measured expression for n aligned genes.

A concrete situation

Ten thousand genes enter one vector.

What the mathematics does not license

Mismatched identifiers silently break masks. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Audit gene mapping and missingness. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

3. Binary mask matrix

The useful question is not only how to calculate, but what the calculation assumes. M has one when gene i belongs to pathway j and zero otherwise.

A concrete situation

Three genes connect to one pathway token.

What the mathematics does not license

Curated sets contain omissions and context mismatch. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Report set sizes and coverage. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

4. Masked linear weights

A small example makes the mechanism visible. Elementwise multiplication W times M forces forbidden gene-to-pathway weights to zero during learning.

A concrete situation

A gene outside pathway P cannot feed token P.

What the mathematics does not license

The mask encodes prior assumptions as architecture. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Perturb masks in validation. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

5. Pathway token calculation

This is where modelling choices begin to affect evidence. The masked weight matrix multiplies e, creating k token values; this embedding is repeated m times, with default m reported as 48.

A concrete situation

One pathway becomes a 48-number row.

What the mathematics does not license

Token width is a capacity choice. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Compare widths and seed stability. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

6. Token matrix geometry

The central mathematical idea is concrete. Stacking k pathway rows gives T in k by m dimensions, so rows are named biological entities and columns are learned features.

A concrete situation

Three pathways form a 3 by 48 matrix.

What the mathematics does not license

Overlapping pathways create correlated tokens. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Measure token redundancy. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

7. CLS token

Here the geometry becomes operational. A learnable 1 by m row is placed above T to gather information for whole-cell prediction.

A concrete situation

CLS becomes a cell summary after attention.

What the mathematics does not license

A summary vector can hide conflicting pathways. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Inspect pathway-level contributions. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

8. Queries, keys and values

This step deserves a careful pause. Linear projections transform every token into Q, K and V matrices used for contextual interaction.

A concrete situation

One pathway queries another.

What the mathematics does not license

Projection weights are learned from labels. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Archive weights and activations. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

9. Scaled dot-product attention

The useful question is not only how to calculate, but what the calculation assumes. Q times K-transpose measures compatibility; division by the square root of key dimension stabilises magnitude before softmax.

A concrete situation

Two aligned token vectors get a high score.

What the mathematics does not license

Large attention can reflect learned scale. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Verify scaling numerically. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

10. Softmax attention weights

A small example makes the mechanism visible. Exponentiation and normalisation make each attention row sum to one.

A concrete situation

Weights 0.6, 0.3 and 0.1 form a convex combination.

What the mathematics does not license

Relative weights do not measure causal effect size. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Compare with ablation. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

11. Value aggregation

This is where modelling choices begin to affect evidence. The attention matrix multiplies V, producing context-aware token outputs.

A concrete situation

A pathway output borrows information from a related pathway.

What the mathematics does not license

Correlations can propagate technical signal. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Test batch predictability. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

12. Multiple heads

The central mathematical idea is concrete. Separate Q, K and V projections run in parallel and are concatenated, allowing different token relationships.

A concrete situation

One head follows lineage; another follows activation.

What the mathematics does not license

Head interpretations are not guaranteed unique. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Compare head consistency across seeds. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

13. Residual and feed-forward layers

Here the geometry becomes operational. Skip connections preserve earlier representations while nonlinear dense layers expand model capacity.

A concrete situation

Output adds the input before normalisation.

What the mathematics does not license

Capacity can overfit a small reference. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Use donor-held-out early stopping. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

14. Cell-type softmax

This step deserves a careful pause. The final CLS representation is linearly scored across nc training labels and normalised to probabilities.

A concrete situation

A cell receives 0.80 type A.

What the mathematics does not license

An absent class still gets a known-label winner. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Calibrate and add review thresholds. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

15. Attention-based interpretation

The useful question is not only how to calculate, but what the calculation assumes. CLS-to-pathway weights form low-dimensional features that can compare cells, states and candidate regulators.

A concrete situation

One monocyte subtype emphasises a regulon.

What the mathematics does not license

Attention is one explanation lens. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Repeat with pathway ablation and expression evidence. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

16. Governance and careers

A small example makes the mechanism visible. Version gene sets, mask, tokens, heads, CLS, probabilities and explanations. This links matrices, attention and bioinformatics.

A concrete situation

Students can hand-calculate a tiny attention example.

What the mathematics does not license

Interpretability does not prove causality. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Write a knowledge-masked Transformer card. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

A seven-day practice lab

1. Day 1: define the decision, observation unit, X variables and Y or reference condition.
2. Day 2: reproduce the smallest worked example and check every dimension, unit and sign.
3. Day 3: build a transparent baseline and write what it assumes.
4. Day 4: fit the main method inside a training-only pipeline; record every parameter.
5. Day 5: challenge one assumption with a deliberately difficult dataset or operating regime.
6. Day 6: compare validation metrics with the practical workload created by errors.
7. Day 7: present one supported conclusion, one unresolved uncertainty and one safe next experiment.

Questions for a student, parent or mentor

• Which part of the model is a mathematical definition, and which part is a choice?
• What small calculation would catch a sign, scale or dimension error?
• What evidence comes from training, validation and the untouched test?
• What benign condition could look like a fault or prediction failure?
• How would the answer change for another response, asset, school project or time period?
• What should a human review before acting on the score?

A method-specific audit before deployment

A TOSICA audit starts with a reference expression matrix, cell labels and a declared gene-set collection. A binary mask permits each pathway token to receive expression only from genes assigned to that pathway; multi-head self-attention then learns how pathway tokens interact, while a class token gathers information for a softmax cell-type prediction.

Record gene identifiers, pathway database and version, mask entries, token width, attention heads, query-key-value projections, residual layers, class-token vector, logits, probabilities and attention scores. Because attention is learned inside a supervised predictor, a high pathway score is interpretable model evidence but does not by itself establish biological causation.

Validate across donors, platforms, tissues and alternative pathway collections; mask or reorder genes; repeat seeds; calibrate softmax probabilities; and test cell types absent from training. Interpretable AI earns trust through reproducible predictions and stable pathway explanations.

---

Frequently asked questions

What is TOSICA?

TOSICA, Transformer for One-Stop Interpretable Cell-type Annotation, is a supervised label-transfer model that represents a cell with pathway or regulon tokens and applies multi-head self-attention.

What is the knowledge mask?

It is a binary gene-by-gene-set matrix. A one permits an expression value to contribute to a pathway token; a zero blocks that connection.

What is a pathway token?

It is a learned weighted combination of expression values from genes belonging to one curated pathway or regulon, repeated across an embedding dimension.

Why add a class token?

A trainable CLS vector attends to pathway tokens and its final representation is passed to a fully connected layer and softmax to predict cell type.

What does attention calculate?

Queries and keys produce scaled dot products, softmax converts them into weights, and the weights average value vectors. Multiple heads learn several relationship patterns in parallel.

Are attention scores causal explanations?

No. CLS-to-pathway attention can show which gene sets the trained model emphasises, but it is model-dependent association and should be tested for stability and biological validity.

---

Primary source and fact-check note

Xiaoping Chen, Yuxuan Chen, Xiaofeng He et al., “Transformer for one stop interpretable cell type annotation,” Nature Communications 14, 223 (2023), DOI 10.1038/s41467-023-35923-4. Definitions and core claims in this article were checked against this primary publication. Calculations shown here were independently recomputed. Examples are educational illustrations; they are not reported industrial performance results.

Useful next reading

• OnClass ontology-aware unseen labels
• ACTINN neural cell typing
• scLearn learned cell-comparison geometry
• How Mathematics Works

Keep the options open. Mathematics education supports clearer questions, transferable problem-solving skills and informed choices across science, engineering, computing, AI, statistics, design and many other careers. It does not guarantee intelligence, income, admission or employment; those outcomes depend on far more than one subject or model.

Extended laboratory: topic-specific validation and responsible use

Gene-set provenance audit

Store MSigDB or regulon release, identifiers, set sizes, overlaps and genes missing from the expression matrix.

Mask-integrity test

Confirm every forbidden weight remains zero and every allowed edge maps the intended gene and pathway.

Attention arithmetic audit

For a small example, save Q, K, scaled logits, row-softmax weights and aggregated V outputs.

Explanation-stability test

Repeat seeds, donors and pathway collections. Compare top pathways and head-level attention, not only labels.

Ablation challenge

Remove high-attention pathways or their genes and measure the change in logits and held-out accuracy.

Calibration audit

Build reliability diagrams by cell type, donor and platform. Softmax sums to one without guaranteeing truth.

Absent-type test

Withhold one reference type and measure forced predictions, entropy and review-threshold performance.

Final claim

Name gene sets, mask, token width, heads, probabilities and explanation tests. Attention is model evidence, not causation.

Mechanism workshop: calculate, challenge and transfer

Workshop 1: Why interpretable tokens help

Start with the mechanism: Gene-level neural features are difficult to name, while pathways provide a human-readable intermediate layer. The worked case is concrete: A token represents interferon signalling. Expose the boundary: Gene-set names can be broad or overlapping. Complete this evidence step: Version the knowledge source. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 2: Expression vector

Start with the mechanism: For one cell, e contains measured expression for n aligned genes. The worked case is concrete: Ten thousand genes enter one vector. Expose the boundary: Mismatched identifiers silently break masks. Complete this evidence step: Audit gene mapping and missingness. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 3: Binary mask matrix

Start with the mechanism: M has one when gene i belongs to pathway j and zero otherwise. The worked case is concrete: Three genes connect to one pathway token. Expose the boundary: Curated sets contain omissions and context mismatch. Complete this evidence step: Report set sizes and coverage. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 4: Masked linear weights

Start with the mechanism: Elementwise multiplication W times M forces forbidden gene-to-pathway weights to zero during learning. The worked case is concrete: A gene outside pathway P cannot feed token P. Expose the boundary: The mask encodes prior assumptions as architecture. Complete this evidence step: Perturb masks in validation. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 5: Pathway token calculation

Start with the mechanism: The masked weight matrix multiplies e, creating k token values; this embedding is repeated m times, with default m reported as 48. The worked case is concrete: One pathway becomes a 48-number row. Expose the boundary: Token width is a capacity choice. Complete this evidence step: Compare widths and seed stability. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 6: Token matrix geometry

Start with the mechanism: Stacking k pathway rows gives T in k by m dimensions, so rows are named biological entities and columns are learned features. The worked case is concrete: Three pathways form a 3 by 48 matrix. Expose the boundary: Overlapping pathways create correlated tokens. Complete this evidence step: Measure token redundancy. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 7: CLS token

Start with the mechanism: A learnable 1 by m row is placed above T to gather information for whole-cell prediction. The worked case is concrete: CLS becomes a cell summary after attention. Expose the boundary: A summary vector can hide conflicting pathways. Complete this evidence step: Inspect pathway-level contributions. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 8: Queries, keys and values

Start with the mechanism: Linear projections transform every token into Q, K and V matrices used for contextual interaction. The worked case is concrete: One pathway queries another. Expose the boundary: Projection weights are learned from labels. Complete this evidence step: Archive weights and activations. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 9: Scaled dot-product attention

Start with the mechanism: Q times K-transpose measures compatibility; division by the square root of key dimension stabilises magnitude before softmax. The worked case is concrete: Two aligned token vectors get a high score. Expose the boundary: Large attention can reflect learned scale. Complete this evidence step: Verify scaling numerically. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 10: Softmax attention weights

Start with the mechanism: Exponentiation and normalisation make each attention row sum to one. The worked case is concrete: Weights 0.6, 0.3 and 0.1 form a convex combination. Expose the boundary: Relative weights do not measure causal effect size. Complete this evidence step: Compare with ablation. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 11: Value aggregation

Start with the mechanism: The attention matrix multiplies V, producing context-aware token outputs. The worked case is concrete: A pathway output borrows information from a related pathway. Expose the boundary: Correlations can propagate technical signal. Complete this evidence step: Test batch predictability. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 12: Multiple heads

Start with the mechanism: Separate Q, K and V projections run in parallel and are concatenated, allowing different token relationships. The worked case is concrete: One head follows lineage; another follows activation. Expose the boundary: Head interpretations are not guaranteed unique. Complete this evidence step: Compare head consistency across seeds. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 13: Residual and feed-forward layers

Start with the mechanism: Skip connections preserve earlier representations while nonlinear dense layers expand model capacity. The worked case is concrete: Output adds the input before normalisation. Expose the boundary: Capacity can overfit a small reference. Complete this evidence step: Use donor-held-out early stopping. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 14: Cell-type softmax

Start with the mechanism: The final CLS representation is linearly scored across nc training labels and normalised to probabilities. The worked case is concrete: A cell receives 0.80 type A. Expose the boundary: An absent class still gets a known-label winner. Complete this evidence step: Calibrate and add review thresholds. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 15: Attention-based interpretation

Start with the mechanism: CLS-to-pathway weights form low-dimensional features that can compare cells, states and candidate regulators. The worked case is concrete: One monocyte subtype emphasises a regulon. Expose the boundary: Attention is one explanation lens. Complete this evidence step: Repeat with pathway ablation and expression evidence. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.

Workshop 16: Governance and careers

Start with the mechanism: Version gene sets, mask, tokens, heads, CLS, probabilities and explanations. This links matrices, attention and bioinformatics. The worked case is concrete: Students can hand-calculate a tiny attention example. Expose the boundary: Interpretability does not prove causality. Complete this evidence step: Write a knowledge-masked Transformer card. Write inputs, the mathematical operation, expected output, uncertainty and one observation that would contradict the claim. Repeat under a nearby defensible setting and compare graph proximities, singular values, anchor weights, token masks, attention matrices, probabilities, embeddings, losses or calibration. Use untouched donors, studies or simulated truth because random cells can share hidden structure with training data. Keep measurement, model estimate, interpretation and decision on separate notebook lines so the conclusion remains auditable.