Chapter map

1. 1. Why single-cell datasets need integration
2. 2. The upstream embedding
3. 3. Soft cluster membership
4. 4. The diversity objective
5. 5. Distances and cluster assignment
6. 6. Cluster-specific linear correction
7. 7. A one-dimensional correction example
8. 8. Alternating optimisation
9. 9. Diversity and ridge parameters
10. 10. Multiple covariates
11. 11. Mixing metrics and LISI
12. 12. Rare populations and unequal composition
13. 13. Donor-aware validation
14. 14. Comparing uncorrected and corrected worlds
15. 15. A student simulation
16. 16. Responsible use and careers

Visit the How Mathematics Works hub for the wider series and its connections to education, reasoning, science, computing, design and careers.

---

1. Why single-cell datasets need integration

This step deserves a careful pause. Single-cell studies can contain thousands of cells from different donors, technologies and laboratories. Technical differences can dominate an embedding and hide recurring cell states.

A concrete situation

The same T-cell type may separate by sequencing chemistry instead of biology.

What the mathematics does not license

Cells are observations, but donors remain important independent units. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Draw the hierarchy: study, donor, sample, cell and measurement. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

2. The upstream embedding

The useful question is not only how to calculate, but what the calculation assumes. Harmony works on a low-dimensional representation, commonly principal components after quality control, normalisation and feature selection. Those earlier choices define the geometry it receives.

A concrete situation

Removing mitochondrial or low-quality cells can change neighbourhoods before integration.

What the mathematics does not license

Harmony cannot recover information excluded upstream. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Record every preprocessing step and the dimensions passed to Harmony. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

3. Soft cluster membership

A small example makes the mechanism visible. Soft clustering assigns each cell a probability-like weight across candidate clusters. Boundary cells can contribute partly to several centroids.

A concrete situation

A transitional cell may be 0.6 in one cluster and 0.4 in another.

What the mathematics does not license

These weights describe the optimisation, not calibrated biological probabilities. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Calculate a weighted centroid from three cells and fractional memberships. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

4. The diversity objective

This is where modelling choices begin to affect evidence. Harmony modifies clustering so batches are represented within clusters more diversely, subject to the data and a penalty parameter.

A concrete situation

A cluster containing only one laboratory may incur a larger diversity cost than a mixed cluster.

What the mathematics does not license

Perfect mixing is not always desirable when batches contain real unique populations. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Compare observed and expected batch proportions inside a toy cluster. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

5. Distances and cluster assignment

The central mathematical idea is concrete. Cells are assigned using distances in the current embedding together with the diversity term. Scaling and component choice therefore affect every assignment.

A concrete situation

A high-variance principal component can dominate Euclidean distance.

What the mathematics does not license

A visually small UMAP gap is not the distance used by Harmony. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Inspect distances in the actual corrected dimensions. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

6. Cluster-specific linear correction

Here the geometry becomes operational. Within each soft cluster, Harmony estimates batch-related offsets and adjusts cell coordinates through weighted local linear models.

A concrete situation

Two batches can need different corrections in a T-cell region and a monocyte region.

What the mathematics does not license

A single global shift may be inadequate; a local shift can also overfit sparse regions. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Compute a weighted batch centroid difference for one cluster. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

7. A one-dimensional correction example

This step deserves a careful pause. Suppose a soft cluster has batch-A centroid 2 and batch-B centroid 5, with a pooled centroid 3.5. Local offsets point toward the pooled centre.

A concrete situation

A cell with mixed cluster membership receives a blend of cluster-specific corrections.

What the mathematics does not license

The real algorithm operates in several dimensions with regularisation and iteration. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Apply two weighted offsets to a cell and check the corrected coordinate. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

8. Alternating optimisation

The useful question is not only how to calculate, but what the calculation assumes. Harmony alternates clustering and correction. Corrected coordinates change clusters; new clusters change corrections, continuing until convergence criteria are met.

A concrete situation

Early iterations can remove large batch shifts and later iterations refine local structure.

What the mathematics does not license

Iteration can stabilise at a solution dependent on parameters or initialisation. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Plot objective or change measures across iterations and repeat seeds. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

9. Diversity and ridge parameters

A small example makes the mechanism visible. Parameters control the pressure for batch diversity, softness of clusters and regularisation of correction. They are modelling assumptions, not aesthetic controls.

A concrete situation

A stronger penalty may mix batches but blur a genuine condition-specific state.

What the mathematics does not license

Tuning on the final biological conclusion creates confirmation bias. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Use prespecified sensitivity ranges and an untouched validation study. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

10. Multiple covariates

This is where modelling choices begin to affect evidence. Harmony can account for several experimental or biological variables, but the role of each variable must be justified.

A concrete situation

Donor, platform and chemistry may all enter the correction design.

What the mathematics does not license

Correcting a variable that encodes the target can erase the target. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Build a causal diagram before deciding which covariates to integrate. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

11. Mixing metrics and LISI

The central mathematical idea is concrete. Local diversity metrics such as LISI summarise how well neighbourhoods mix by batch or separate by biology. A metric needs a reference expectation.

A concrete situation

High batch LISI can coexist with poor cell-type conservation.

What the mathematics does not license

One scalar cannot certify every region of a manifold. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Report metric distributions by cell type, donor and density. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

12. Rare populations and unequal composition

Here the geometry becomes operational. A rare state appearing in one batch may be technical, biological or simply unsampled elsewhere. Forcing it to mix can destroy a real finding.

A concrete situation

A disease-specific cell state may have no healthy counterpart.

What the mathematics does not license

Absence of overlap is not evidence that a correction should invent overlap. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Inspect rare-state markers and raw neighbours before alignment. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

13. Donor-aware validation

This step deserves a careful pause. Random cell splits leak donor and batch structure because thousands of related cells come from the same person. Hold out donors or studies.

A concrete situation

A model that mixes cells within known donors may fail on a new hospital.

What the mathematics does not license

Cell count does not equal independent sample count. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Report donors, samples and cells separately in every result. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

14. Comparing uncorrected and corrected worlds

The useful question is not only how to calculate, but what the calculation assumes. Retain the original PCA, corrected coordinates, raw counts and metadata. Compare both technical and biological structure.

A concrete situation

A corrected embedding may improve shared cell-type clustering while raw counts remain needed for differential expression.

What the mathematics does not license

Discarding the uncorrected view hides what changed. Evidence becomes stronger when the limitation is tested instead of hidden.

Try it, check it, transfer it

Create a before-after table of neighbourhoods, markers and batch association. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

15. A student simulation

A small example makes the mechanism visible. Simulate two cell types, two batches and one batch shift, then remove the shift with local centroids. Add a batch-specific third type.

A concrete situation

The third type shows why universal mixing can be harmful.

What the mathematics does not license

Simulations simplify gene-count noise and donor variation. Keeping this boundary clear makes the result easier to audit and transfer.

Try it, check it, transfer it

Vary overlap and penalty strength to map overcorrection. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

16. Responsible use and careers

This is where modelling choices begin to affect evidence. Version preprocessing, embeddings, parameters, seeds, corrected coordinates and validation. Single-cell integration joins linear algebra, optimisation, biology and software.

A concrete situation

Students can practise with centroids and small matrices before learning large datasets.

What the mathematics does not license

Harmony alone proves no cell identity, mechanism or career outcome. A trustworthy workflow records the choice before seeing the final test result.

Try it, check it, transfer it

Write a release note stating preserved biology, removed variation and unresolved regions. Then explain the result to someone who has not seen the formula. If the explanation needs a hidden assumption, add that assumption to the model record.

---

A seven-day practice lab

1. Day 1: define the decision, observation unit, X variables and Y or reference condition.
2. Day 2: reproduce the smallest worked example and check every dimension, unit and sign.
3. Day 3: build a transparent baseline and write what it assumes.
4. Day 4: fit the main method inside a training-only pipeline; record every parameter.
5. Day 5: challenge one assumption with a deliberately difficult dataset or operating regime.
6. Day 6: compare validation metrics with the practical workload created by errors.
7. Day 7: present one supported conclusion, one unresolved uncertainty and one safe next experiment.

Questions for a student, parent or mentor

• Which part of the model is a mathematical definition, and which part is a choice?
• What small calculation would catch a sign, scale or dimension error?
• What evidence comes from training, validation and the untouched test?
• What benign condition could look like a fault or prediction failure?
• How would the answer change for another response, asset, school project or time period?
• What should a human review before acting on the score?

A method-specific audit before deployment

A Harmony audit starts with a quality-controlled low-dimensional embedding, a batch or study variable and a clear statement of the biology that should remain. Harmony alternates soft clustering with cluster-specific batch correction so that clusters can contain diverse batches. It does not repair poor counts, wrong cell identities or a design in which biology is perfectly confounded with batch.

Save the upstream feature selection, scaling and principal components, the diversity and clustering parameters, random seeds, iteration traces and corrected coordinates. Evaluate both mixing and biological conservation. Strong batch mixing can be overcorrection when a batch truly contains a unique cell state, while perfect biological separation can simply preserve a technical artefact.

Use donor-aware validation and an untouched study when possible. Compare uncorrected PCA, Harmony and at least one simple baseline under the same cell-quality rules. Report cell-type recovery, neighbourhood preservation, batch mixing, rare-population behaviour and sensitivity. A polished UMAP is an illustration, not sufficient evidence that integration succeeded.

Keep interpretation tied to the exact corrected space. Harmony can make a shared representation more useful for clustering and visualisation, while differential-expression questions may still require raw counts, donor-level replication and explicit covariate models. Write down which downstream analysis consumes which data object. This governance step prevents an embedding correction from being treated as though it rewrote every measurement into a batch-free value.

---

Frequently asked questions

What data does Harmony usually receive?

It usually receives a low-dimensional embedding, often principal-component scores derived from quality-controlled and normalised single-cell measurements.

What is soft clustering?

Each cell receives fractional membership across clusters rather than one permanent hard label, allowing uncertainty near boundaries.

Why use a diversity penalty?

It encourages clusters to include cells from multiple batches when the data support such mixing, reducing batch-dominated clusters.

Does Harmony change raw counts?

The standard workflow corrects the embedding. Raw counts remain separate for appropriate downstream expression modelling.

Can Harmony overcorrect?

Yes. Aggressive integration can merge genuinely different states or obscure batch-specific biology, especially under confounding or missing overlap.

How should it be validated?

Assess donor-aware batch mixing, neighbourhood and cell-state preservation, rare populations, downstream stability and performance on an untouched study.

---

Primary source and fact-check note

Ilya Korsunsky, Nghia Millard, Jean Fan et al., “Fast, sensitive and accurate integration of single-cell data with Harmony,” Nature Methods 16, 1289–1296 (2019), DOI 10.1038/s41592-019-0619-0. Definitions and core claims in this article were checked against this primary publication. Calculations shown here were independently recomputed. Examples are educational illustrations; they are not reported industrial performance results.

Useful next reading

• ComBat and empirical-Bayes batch adjustment
• CORAL and covariance alignment
• Surrogate Variable Analysis

Keep the options open. Mathematics education supports clearer questions, transferable problem-solving skills and informed choices across science, engineering, computing, AI, statistics, design and many other careers. It does not guarantee intelligence, income, admission or employment; those outcomes depend on far more than one subject or model.

Extended laboratory: validation, transfer and responsible use

Cell hierarchy

Record studies, donors, samples and cells separately. Use donor-aware uncertainty and splits. Treating every cell as independent makes confidence intervals too narrow and can let donor-specific patterns appear universally reproducible.

Upstream lock

Freeze quality filters, normalisation, feature selection, scaling and principal components before comparing integration settings. If upstream steps change, the geometry and therefore Harmony solution change. Archive both original and corrected embeddings.

Parameter map

Evaluate a prespecified grid for cluster softness, diversity pressure, cluster number and correction regularisation. Report a frontier between batch mixing and biological preservation instead of selecting the prettiest UMAP.

Confounding test

Cross-tab batch with donor, condition, tissue and cell state. If disease occurs only in one batch, correction cannot know which difference is technical. Narrow the claim or redesign the study instead of forcing stronger mixing.

Rare-state guard

List rare populations, their donor support and their nearest neighbours before and after correction. A state found in one batch should not be automatically erased. Require marker and replication evidence before calling it artefact or discovery.

Metric panel

Use batch-mixing, cell-type conservation, neighbourhood retention, silhouette or graph-connectivity measures as appropriate. Show distributions by cell state. One global score can hide poor correction in a sparse or clinically important region.

Seed and convergence

Repeat seeds and store objective traces. Similar scores can still swap a few boundary cells, affecting downstream clusters. Quantify stability of neighbourhoods and labels rather than reporting only that the optimiser converged.

Downstream separation

Do not use corrected embeddings as if they were raw counts for every analysis. Define which downstream tasks use corrected coordinates, original expression or models with batch covariates. Each task has a separate inferential contract.

New-study test

Apply the locked preprocessing and integration strategy to a new study without using final labels to tune. Report cells rejected for poor coverage. Integration that works only after viewing new labels is adaptation, not untouched validation.

Overcorrection scenario

Simulate a cell state unique to one batch and a technical shift shared across states. Observe when stronger diversity penalties merge the unique state. This test teaches that mixing and truth can conflict.

Governed release

Archive raw references, corrected coordinates, metadata, parameters and evaluation notebooks. Define who can approve a changed integration. Downstream conclusions should state the exact embedding version used.

Reader checkpoint

A trustworthy Harmony claim names the upstream space, batch variables, parameters, convergence, donor counts, mixing evidence, biological preservation and rare-state limits. A UMAP alone is not a validation report.

Deep dive: how to stress-test the mathematical claim

Deep dive 1

Run an uncorrected baseline first. If known biology is already organised and batch association is weak, integration may add risk without benefit.

Deep dive 2

Inspect cells with the largest coordinate changes. Their quality, donor, state and marker patterns often reveal whether correction is fixing batch or moving unsupported cells.

Deep dive 3

Compare neighbourhoods before and after correction rather than relying on a two-dimensional visualisation. UMAP can exaggerate or hide differences.

Deep dive 4

Tune diversity and regularisation with study-held-out evidence. Final cell labels should not select the parameters that later appear to recover those same labels.

Deep dive 5

Use a batch-specific unique-state simulation. A method that always maximises mixing will erase it; a good analysis reports the trade-off.

Deep dive 6

Check whether corrected clusters depend on the number of upstream principal components. Large sensitivity can reveal that technical signal remains in omitted or included dimensions.

Deep dive 7

Report donor-level marker summaries. Cell-level significance after integration can remain confounded when donor counts are small.

Deep dive 8

Treat a new sequencing chemistry as a new domain. Reuse of old parameters requires a coverage check and fresh validation, not an assumption of universality.

Deep dive 9

Preserve the original coordinates so investigators can trace every changed conclusion. Integration should be reversible and auditable.

Deep dive 10

Distinguish representation quality, clustering quality and differential-expression validity. One successful task does not certify the others.

Worked evidence notebook

Soft-membership arithmetic

With weights 0.7 and 0.3 across two clusters, a cell can receive 70 percent of the first cluster correction and 30 percent of the second. Compute the blended vector coordinate by coordinate.

Diversity table

For each cluster, compare batch counts with expected counts under the global batch proportions. Large deviations guide investigation but do not prove artefact.

Centroid correction

Given cluster-specific batch centroids, subtract the batch offset relative to a pooled or reference centroid using the fitted weighted model. Check the sign on a two-dimensional example.

Iteration trace

Store objective change, cluster membership change and coordinate change by iteration. Convergence should be a documented numeric condition.

Biology panel

Track cell-state markers, neighbourhood retention, trajectory order and donor replication beside batch-mixing metrics.

Release note

State which analyses use Harmony coordinates and which return to raw counts. This prevents the corrected embedding from silently becoming a universal data replacement.

A final worked scenario

A final integration scenario

Suppose five donors are measured with two single-cell technologies. Uncorrected principal components separate technologies strongly. Harmony is fitted on four donors with fixed upstream preprocessing and tested on the fifth. Batch mixing improves for common T-cell and monocyte states while marker gradients and neighbourhoods remain stable. A rare state appears only in one disease donor and moves substantially under aggressive diversity settings, so the team selects a milder setting and labels that region uncertain. Differential expression returns to donor-level models using raw counts rather than treating corrected coordinates as measurements. The final report includes mixing metrics, biological preservation, rare-state limits and every parameter. The method succeeds without claiming universal correction: it provides a more useful shared embedding for supported populations and a clear map of where the evidence stops.

Method checklist and conclusion boundary

A compact reproducibility checklist

Freeze cell-quality rules, feature selection, scaling and principal components. Save batch variables, parameters, seeds, objective traces and original and corrected coordinates. Use donor-aware holdouts, inspect the largest cell movements, and report batch mixing beside markers, neighbourhoods, trajectories and rare states. Define which downstream tasks use corrected coordinates and which use raw counts.

What a careful conclusion sounds like

A careful conclusion says that Harmony reduced specified batch structure in supported cell populations while preserving named biological properties under donor-aware validation. It identifies uncertain regions and parameter sensitivity. It does not say that all corrected neighbours are biologically identical, that raw measurements became batch-free, or that a mixed UMAP proves every downstream inference.

Practice, misconception and transfer

Read the corrected space as a map

A corrected embedding is a coordinate map designed to make supported biological neighbourhoods easier to compare across batches. It is not a replacement genome, a final cell atlas or proof that two cells are equivalent. Readers should ask which distances changed, which states supplied cross-batch support and which regions remained dependent on one donor or technology.

Keep the two denominators visible

A million cells can give detailed local geometry, while five donors still provide only five independent biological units for many population claims. Report both numbers. Cell-level precision helps describe an embedding; donor-level replication supports conclusions about people or conditions. Confusing these denominators can make tiny technical effects look overwhelmingly certain.

Why beautiful mixing can mislead

A colourful plot with batches evenly mixed is attractive because it looks resolved. Yet a method could achieve that picture by removing real condition structure. The safeguard is a paired evidence panel: batch association should fall, while markers, known states, trajectories and held-out-donor behaviour remain. The best result is not the most mixed plot; it is the most defensible balance.

What to do with disagreement

If one parameter setting preserves a rare state and another mixes batches better, record the disagreement. Inspect raw expression, donor support and experimental controls, then narrow the claim. Uncertainty is not a failure of the method. It identifies the region where additional samples or orthogonal assays would add the most information.

A classroom calculation

Give four points in two dimensions, two batches and fractional cluster weights. Let learners calculate weighted centroids, batch offsets and one corrected coordinate. Then change one weight and observe how the correction moves. This small exercise connects ratios, coordinates and matrices to a live problem in biology while making parameter sensitivity visible.

A responsible deployment boundary

A Harmony model built for exploratory integration should not silently become a clinical classifier. If corrected coordinates feed a decision system, define inputs, version the reference, measure out-of-distribution cells, validate donor-level outcomes and provide abstention rules. Each new use creates a new claim and needs its own evidence.

Final evidence checks

Final evidence checks 1

Keep donor counts prominent beside cell counts.

Final evidence checks 2

Preserve uncorrected principal components and raw measurements for audit.

Final evidence checks 3

Report mixing and biological conservation together.

Final evidence checks 4

Flag rare states with weak cross-batch support rather than forcing alignment.

Final evidence checks 5

Version every upstream and Harmony parameter that changes coordinates.

Final evidence checks 6

Require independent review of one soft assignment and one corrected coordinate.