Chapter map

1. 1. Why clustering and correction interact
2. 2. Input representation and preprocessing
3. 3. The autoencoder
4. 4. Stacked pretraining
5. 5. Initial centroids
6. 6. Student-t soft assignments
7. 7. The auxiliary target distribution
8. 8. KL-divergence objective
9. 9. Iterative self-learning
10. 10. Batch attenuation without labels
11. 11. Soft assignments and transitional cells
12. 12. Cluster number and resolution
13. 13. Seed and stopping stability
14. 14. Donor-aware validation
15. 15. A student self-training project
16. 16. Governance and careers

Visit the How Mathematics Works hub for the wider series and its connections to education, reasoning, science, computing, design and careers.

---

1. Why clustering and correction interact

This is where modelling choices begin to affect evidence. If batch structure dominates the embedding, clusters follow experiments; if correction is too strong, real states collapse.

A concrete situation

Joint learning can focus on cluster-relevant structure.

What the mathematics does not license

No objective knows biology without assumptions. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Write the desired biological separation and nuisance structure. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

2. Input representation and preprocessing

The central mathematical idea is concrete. Counts are filtered and transformed before the network sees them. Scaling choices affect reconstruction and distances.

A concrete situation

Highly variable genes can focus the embedding.

What the mathematics does not license

Global preprocessing can leak test studies. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Repeat feature selection inside donor splits. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

3. The autoencoder

Here the geometry becomes operational. An encoder compresses expression into a bottleneck and a decoder reconstructs the input.

A concrete situation

The bottleneck learns a nonlinear representation.

What the mathematics does not license

Good reconstruction can preserve batch as well as biology. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Calculate dimensions through each layer. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

4. Stacked pretraining

This step deserves a careful pause. Layer-wise or stacked autoencoder pretraining initializes weights before clustering refinement.

A concrete situation

Pretraining avoids starting centroids in an arbitrary raw space.

What the mathematics does not license

Initialisation can still shape the final solution. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Save checkpoints and repeat seeds. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

5. Initial centroids

The useful question is not only how to calculate, but what the calculation assumes. A clustering method initializes centroids in the learned embedding before self-training.

A concrete situation

Each centroid represents a tentative cell group.

What the mathematics does not license

The requested cluster number can force splits or merges. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Compare several biologically defensible cluster counts. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

6. Student-t soft assignments

A small example makes the mechanism visible. Cell-to-centroid distances are converted into soft memberships using a heavy-tailed kernel.

A concrete situation

A boundary cell can share membership across clusters.

What the mathematics does not license

The values are model scores, not guaranteed biological probabilities. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Normalise a three-centroid score vector. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

7. The auxiliary target distribution

This is where modelling choices begin to affect evidence. DESC sharpens high-confidence memberships and adjusts for cluster frequency to build a target distribution P.

A concrete situation

Easy-to-cluster cells guide the next update.

What the mathematics does not license

Early mistakes can reinforce themselves. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Compute a toy target from Q and cluster frequencies. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

8. KL-divergence objective

The central mathematical idea is concrete. Training reduces KL divergence between target P and current assignments Q, updating the encoder and centroids.

A concrete situation

Cells move toward cluster-specific features.

What the mathematics does not license

A smaller objective does not certify correct biology. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Calculate KL for two small distributions. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

9. Iterative self-learning

Here the geometry becomes operational. Assignments create targets, targets update the network, and the cycle repeats until a stopping criterion.

A concrete situation

Confident cells teach the representation over rounds.

What the mathematics does not license

Self-training can amplify confirmation bias. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Plot assignment changes and objective by iteration. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

10. Batch attenuation without labels

This step deserves a careful pause. The method can ignore variation unrelated to clustering when technical differences are smaller than biological differences.

A concrete situation

Shared cell types across protocols may converge.

What the mathematics does not license

If batch is larger or confounded, the assumption fails. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Measure batch and biology separation before and after. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

11. Soft assignments and transitional cells

The useful question is not only how to calculate, but what the calculation assumes. Diffuse memberships can reflect uncertainty or continuous biological transitions.

A concrete situation

Progenitors may sit between mature centroids.

What the mathematics does not license

Low maximum assignment can also mean model failure. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Compare with markers and independent pseudotime. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

12. Cluster number and resolution

A small example makes the mechanism visible. The chosen number of centroids changes the target distribution and learned features.

A concrete situation

A broad immune group can split into subtypes at higher k.

What the mathematics does not license

More clusters are not automatically more true. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Use stability, markers and held-out evidence. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

13. Seed and stopping stability

This is where modelling choices begin to affect evidence. Network initialization, mini-batches and target updates create stochastic variation.

A concrete situation

Several seeds can recover the same major groups.

What the mathematics does not license

One clean run can be lucky. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Align clusters and report consensus and disagreement. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

14. Donor-aware validation

The central mathematical idea is concrete. Hold out donors or studies, then test cluster retrieval, batch attenuation and marker preservation.

A concrete situation

A new protocol tests external stability.

What the mathematics does not license

Random cells share donor structure. The habit of checking assumptions is one of the practical benefits of learning mathematics.

Try it, check it, transfer it

Separate cell count from biological replicate count. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

15. A student self-training project

Here the geometry becomes operational. Create two biological clusters with a smaller batch shift, then increase the shift until self-training fails.

A concrete situation

Known truth exposes the assumption boundary.

What the mathematics does not license

Real data have rare and continuous states. The same reasoning helps students move from an exercise to a real dataset.

Try it, check it, transfer it

Track Q, P, centroids and errors per round. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

16. Governance and careers

This step deserves a careful pause. Version preprocessing, network, centroids, cluster count, Q, P, KL traces, seeds and stopping. Deep clustering joins optimisation, geometry and biology.

A concrete situation

Students can begin with softmax-like weights and KL divergence.

What the mathematics does not license

A cluster proves no mechanism or diagnosis. That distinction matters because a correct formula can still answer the wrong operational question.

Try it, check it, transfer it

Write a stability and failure report. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

A seven-day practice lab

1. Day 1: define the decision, observation unit, X variables and Y or reference condition.
2. Day 2: reproduce the smallest worked example and check every dimension, unit and sign.
3. Day 3: build a transparent baseline and write what it assumes.
4. Day 4: fit the main method inside a training-only pipeline; record every parameter.
5. Day 5: challenge one assumption with a deliberately difficult dataset or operating regime.
6. Day 6: compare validation metrics with the practical workload created by errors.
7. Day 7: present one supported conclusion, one unresolved uncertainty and one safe next experiment.

Questions for a student, parent or mentor

• Which part of the model is a mathematical definition, and which part is a choice?
• What small calculation would catch a sign, scale or dimension error?
• What evidence comes from training, validation and the untouched test?
• What benign condition could look like a fault or prediction failure?
• How would the answer change for another response, asset, school project or time period?
• What should a human review before acting on the score?

A method-specific audit before deployment

A DESC audit starts with processed single-cell expression, an explicit clustering question and a design table of donors, batches and conditions. A stacked autoencoder initialises a nonlinear embedding, then iterative self-training refines the encoder and cluster centroids through soft assignments and an auxiliary target distribution.

Record preprocessing, network layers, pretraining, cluster count, initial centroids, Student-t assignment settings, target updates, KL objective, stopping rule and seeds. The method can attenuate batch structure without explicit batch labels when technical differences are smaller than biological differences, an assumption that must be tested rather than repeated as a guarantee.

Validate with donor or study holdouts, seed and cluster-number stability, marker preservation, known continuous trajectories, rare-state review and baselines. Soft assignment values describe the fitted clustering geometry; they are not automatically calibrated probabilities of biological identity.

---

Frequently asked questions

What is DESC?

DESC is an unsupervised deep embedding method that jointly refines a low-dimensional representation and soft cell clusters.

How does DESC begin?

A stacked autoencoder is pretrained, and its encoder initializes the iterative clustering network.

What are soft assignments?

Each cell receives a distribution of cluster memberships based on its embedding and centroid distances.

What is self-training?

High-confidence assignments are sharpened into an auxiliary target that guides further network updates.

Does DESC always remove batch effects?

No. The paper states a key condition: technical differences should be smaller than true biological differences.

How should DESC be validated?

Use donor holdouts, seed and cluster-number stability, marker preservation, batch and biology metrics, trajectory checks and external datasets.

---

Primary source and fact-check note

Xiangjie Li, Kui Wang, Yafei Lyu et al., “Deep learning enables accurate clustering with batch effect removal in single-cell RNA-seq analysis,” Nature Communications 11, 2338 (2020), DOI 10.1038/s41467-020-15851-3. Definitions and core claims in this article were checked against this primary publication. Calculations shown here were independently recomputed. Examples are educational illustrations; they are not reported industrial performance results.

Useful next reading

• scVI deep generative modelling
• Harmony soft clustering and correction
• Scanorama graph integration

Keep the options open. Mathematics education supports clearer questions, transferable problem-solving skills and informed choices across science, engineering, computing, AI, statistics, design and many other careers. It does not guarantee intelligence, income, admission or employment; those outcomes depend on far more than one subject or model.

Extended laboratory: validation, transfer and responsible use

Preprocessing ledger

Archive counts, filters, transformation, selected genes, scaling and donor splits. Repeat feature selection within training folds to avoid test-study leakage.

Architecture ledger

Record encoder and decoder widths, activations, bottleneck dimension, optimiser, learning rate, mini-batches and pretraining epochs.

Centroid ledger

Save initial and final centroids, cluster count and assignment changes. Align cluster identities across seeds before comparing them.

Q and P audit

For selected cells, store soft assignment Q, target P and the resulting gradient contribution. This exposes which confident cells drive self-training.

Confirmation-bias challenge

Perturb initial centroids or add ambiguous cells. If early mistakes become stable clusters, narrow the conclusion and improve validation.

Cluster-number sensitivity

Run a justified range of k values. Compare consensus groups, split and merge patterns, markers and held-out retrieval.

Seed stability

Repeat pretraining and clustering seeds. Report consensus matrices and unstable cells, not only the cleanest embedding.

Batch-assumption test

Increase technical shift in simulation until it exceeds biological separation. Record where batch becomes a cluster and where true states collapse.

Trajectory preservation

For continuous biology, compare local order, branches and marker progression before and after training. Soft assignments alone do not validate pseudotime.

Rare-state guard

Measure whether rare classes receive their own stable centroid, merge into a common cluster or fluctuate across runs.

External study

Apply locked preprocessing and architecture to an untouched study. Evaluate clusters and markers without reusing final labels for tuning.

Reader checkpoint

A trustworthy DESC claim names preprocessing, architecture, centroids, Q, P, KL objective, seeds, stopping, stability and assumption limits.

Deep dive: how to stress-test the mathematical claim

Deep dive 1

Track each cell’s Q and P vectors through iterations to reveal target sharpening.

Deep dive 2

Repeat initial centroids, network seeds and mini-batch order; align clusters before comparison.

Deep dive 3

Vary cluster number and bottleneck dimension under held-out donors.

Deep dive 4

Simulate technical shifts smaller than, equal to and larger than biological separation.

Deep dive 5

Compare cluster confidence with held-out accuracy to test calibration.

Deep dive 6

Inspect unstable and low-confidence cells for transitions, novelty and model failure.

Deep dive 7

Compare with PCA plus clustering and another deep baseline under identical splits.

Deep dive 8

Check whether stopping earlier preserves a rare state that later disappears.

Deep dive 9

Use original markers and donor-level tests for biological interpretation.

Deep dive 10

Separate optimisation convergence, clustering stability, batch attenuation and mechanism.

Worked evidence notebook

Autoencoder dimensions

Write input, hidden, bottleneck and reconstruction dimensions for one forward pass.

Soft assignment

Convert three centroid distances to Student-t scores and normalise them.

Target distribution

Square Q values, divide by cluster frequencies and renormalise to obtain P.

KL divergence

Calculate the sum of P log(P/Q) for a two-cluster toy cell.

Iteration ledger

Record Q, P, centroid, KL loss and assigned cluster at every target update.

Stability table

Align clusters across seeds and report consensus, splits and unstable cells.

Practice, misconception and transfer

A parent-friendly starting point

Use moving group centres on a scatter plot to show how confident points can guide uncertain ones.

A key misconception

Self-training does not turn an initial guess into truth; it can reinforce both good and bad early assignments.

When to pause

Pause when technical differences exceed biology, results change across seeds or the cluster number lacks justification.

A useful extension

Increase a simulated batch shift until the learned clusters follow batch.

Career transfer

Deep clustering appears in images, customers and sensors, but every domain must define meaningful groups.

Final learning move

Ask the learner to compute Q, P and one KL term before interpreting the embedding.

A final worked scenario

A final deep-clustering scenario

DESC is trained on several donor cohorts with a fixed cluster range and repeated seeds. Major cell groups recur, while one small group changes across seeds and receives low soft assignment. A simulation shows that the method fails once the technical shift exceeds biological separation, so the report states that boundary. Held-out markers and trajectories support the stable groups. The mathematics contributes a reproducible embedding, assignments and stability analysis; it does not convert a neural objective into biological truth.

Mathematical workshop: mechanisms, evidence and limits

Cluster alignment across runs is essential

Cluster labels are arbitrary identifiers: cluster 1 in one seed may correspond to cluster 4 in another. Match clusters using overlap, centroid similarity or an assignment algorithm before comparing them. Then build a consensus matrix showing how often each pair of cells co-clusters. Report stable cores and uncertain boundaries. Without alignment, apparent disagreement can be a label permutation, while genuine splits and merges can be missed. This simple bookkeeping turns multiple seeds into usable stability evidence.

Reconstruction pretraining and clustering goals differ

Autoencoder pretraining rewards reconstruction of the processed input, including both biology and technical variation. The later clustering objective rewards alignment with sharpened cluster assignments. A representation can therefore improve one objective while harming another. Save checkpoints before and during refinement, then compare reconstruction, batch structure, markers and held-out clustering. If later self-training erases a rare programme, an earlier checkpoint may be scientifically preferable even with a higher final clustering loss. Model selection must follow the biological task, not one optimisation number.

Stopping is a statistical decision

DESC iterations stop when assignments change little or another criterion is met. Stability of assignments can mean convergence to a useful solution or entrenchment of an early mistake. Plot assignment-change rate, KL loss, marker preservation, batch metrics and seed consensus by epoch. Choose stopping rules inside training data and lock them before the external study. Avoid selecting the epoch with the best test labels. The stopping rule is part of the method and belongs in the reproducibility record.

Rare cells may be weak teachers

Target sharpening favours easy-to-cluster cells, which are often abundant and near centroids. Rare or transitional cells can receive diffuse Q values and contribute less to updates, eventually being absorbed by a dominant group. Evaluate sensitivity by class size, oversample only with clear justification and compare with methods designed for rare-state retention. Preserve a list of unstable cells. Low confidence is not a nuisance to delete; it can signal transition, novelty, doublets or an insufficient model.

Why self-training can work

Large datasets often contain many easy-to-cluster cells near clear centres and fewer ambiguous cells near boundaries. DESC uses confident structure to refine the representation. The target distribution increases the influence of high-confidence assignments, and gradient updates move the encoder toward cluster-specific features. This can suppress variation unrelated to the dominant clusters. The same mechanism can also amplify an early mistake. Self-training is therefore a hypothesis about which cells are reliable teachers, not a mathematical guarantee that confidence equals correctness.

A soft-assignment calculation

Suppose a cell’s Student-t scores for three centroids are 4, 2 and 1. Normalising gives Q=(4/7,2/7,1/7). The largest assignment is about 0.571, which is not especially decisive. The auxiliary target squares assignments and adjusts by cluster frequencies before renormalisation, sharpening confident cases. If cluster frequencies are equal, squared values are proportional to 16, 4 and 1, giving P about (0.762,0.190,0.048). The KL objective then encourages Q to move toward this sharper target.

Why cluster frequency correction matters

Simply squaring Q would let large clusters dominate the target. Dividing by a cluster-frequency term compensates for how often each centroid is selected before row normalisation. This does not solve every imbalance: a rare biological state can still lack a stable centroid, and an artificial small cluster can be overemphasised. Report cluster sizes, target mass and sensitivity to k. Frequency adjustment is one design choice inside the objective, not proof that every cluster receives fair biological representation.

The key batch assumption

The authors describe successful attenuation when technical differences are smaller than true biological differences. If batch dominates, confident cells may first organise by batch, and self-training will sharpen that separation. Simulate two biological groups with a small shift, then raise the shift until clusters follow batches. The failure point depends on preprocessing, architecture and k. Reporting it turns a broad slogan into a measured operating range and helps decide whether a batch-aware alternative or a redesigned study is needed.

Soft probability is not calibration

DESC’s assignment values reflect distances in the learned embedding under the chosen kernel and centroids. A maximum of 0.9 is not automatically a 90% chance of correct biological identity. On held-out donors, bin maximum assignments and compare them with external labels or marker-supported outcomes. Examine low-confidence cells for transitions, doublets, novelty and model failure. A score becomes useful for decisions only after its behaviour is measured in the relevant domain.

Trajectories need independent checks

Diffuse memberships can be biologically meaningful for continuous differentiation, but they can also arise from overlapping clusters or poor fit. Compare embedding-based order with known marker progression, local neighbour preservation and independent trajectory methods. Hold out donors and examine whether branches recur. A smooth curve in two dimensions is not enough. When a branch changes across seeds or stopping times, report that instability rather than selecting the most appealing visualisation.

A student self-training workshop

Create two clusters with clear centres and a bridge of ambiguous points. Initialise centroids, calculate Q, sharpen to P and move points or a simple linear encoder to reduce KL divergence. Repeat after moving one batch by a growing offset. Track when the algorithm recovers biology and when it locks onto batch. This notebook teaches distances, probability normalisation, KL divergence, optimisation, confirmation bias and validation. It also shows why reproducibility requires more than one random seed.

A precise final statement

A careful result might say that across ten seeds and two held-out donors, four major groups recurred with stated consensus while one rare group remained unstable; batch metrics improved only within a measured shift range and marker progressions were preserved. It does not claim that the lowest KL objective discovered true cell types or that batch labels are unnecessary in every study. Reporting stability, assumption tests and unsupported cells makes deep clustering scientifically useful rather than merely impressive.

Method checklist and conclusion boundary

Reproducibility checklist

Freeze preprocessing, genes, architecture, pretraining, cluster count, initial centroids, Q, P, KL objective, optimiser, seeds and stopping. Reproduce one target update and cluster alignment.

Careful conclusion

A careful conclusion says that stable cluster-specific structure emerged across held-out donors and seeds under the method’s assumption. It does not claim that a cluster is a biological mechanism.

Final evidence checks

Final evidence checks 1

Version network, centroids and cluster count.

Final evidence checks 2

Archive Q, P and KL traces.

Final evidence checks 3

Repeat seeds and held-out donors.

Final evidence checks 4

Test the batch-versus-biology assumption.

Final evidence checks 5

Preserve original marker evidence.

Final evidence checks 6

Require independent reproduction of one target update.