eduKateSG · Genetics · The Gold Standard of Education
Replicate a rare-variant set across cohorts without hiding important cohort dependence
Choose the explanation you need, or follow all twelve chapters in order.
Full chapter index · Independent practice · Genetics subject guide
Rare-variant set tests gain power by pooling carriers inside genes or masks, but a discovery is not automatically portable merely because its combined p-value is small. Cohorts can differ in ancestry mix, recruitment, sequencing, phenotype coding, covariates and carrier composition. A strong replication plan therefore preserves the same scientific question while allowing each biobank to expose whether the signal depends on one dataset. This guide builds that plan around a fictional cross-biobank example, leave-one-biobank-out checks and explicit effect-heterogeneity reporting.
The standard is not four copies of the same p-value. It is a reproducible chain from a frozen mask and harmonised phenotype to cohort-level diagnostics, a compatible combined analysis and a narrow conclusion. A reader should be able to see which variants and carriers drive the set, where estimates disagree, and whether removing one biobank changes direction, magnitude or calibration.
Continue from The Gold Standard of Genetics within The Gold Standard of Education: learning that transfers into life and the How X Works learning library.
Full chapter index
See the mechanism · Chapters 1–2
Build the worked model · Chapters 3–5
Test what changes · Chapters 6–8
Check evidence and limits · Chapters 9–10
Practise and continue · Chapters 11–12
Replication starts with an estimand, not a label. State the gene or region, phenotype, qualifying-variant mask, allele-frequency rule, analysis population, genetic model and effect measure. If discovery used a burden test for a binary outcome, a follow-up variance-component test answers a related but not identical question. Both may be useful, but they should not be called interchangeable replication.
Freeze the discovery rule before opening the replication result. Transcript, genome build, annotation version, frequency threshold and weight function all belong in that rule. A changed rule can be a sensitivity analysis, not a silent replacement.
Evidence check: for Define what is being replicated, preserve the observation before the interpretation. Record estimand, phenotype and cohort population. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Map phenotype codes to a common operational definition and record exclusions, observation windows and ascertainment. Align reference build, alleles, variant normalisation and consequence annotation. Verify that covariates have the same meaning and that relatedness, ancestry structure and case-control imbalance are handled by a method calibrated for each cohort.
Harmonisation does not mean making different cohorts look identical. It means making differences visible and deciding which are compatible with the target estimand. A hospital biobank and a population cohort may both contribute, yet their prevalence and selection mechanisms still require separate diagnostics.
Evidence check: for Harmonise the cohorts before combining them, preserve the observation before the interpretation. Record mask, build and annotation manifest. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
CHAPTER 3 OF 12 · Build the worked model
3. Freeze masks and cohort-specific rarity rules
A variant set might include predicted loss-of-function variants alone, or that set plus damaging missense variants. Preserve a manifest listing every included variant, transcript consequence, annotation source and weight. Decide whether rarity is assessed within each biobank, within ancestry strata, against an external reference, or by a declared combination.
A globally rare variant can be more common in one population. A site that silently applies a different threshold is not replicating the same mask. Report site-specific allele counts even when a common membership rule is used.
Evidence check: for Freeze masks and cohort-specific rarity rules, preserve the observation before the interpretation. Record within-cohort rarity and carrier counts. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Imagine Biobanks A, B and C analyse the same gene and narrow mask. Their estimated log-odds effects are 0.42, 0.36 and 0.05 with standard errors 0.14, 0.18 and 0.22. A and B point in the same direction; C is imprecise and close to zero. This is not proof that C has no effect. Its interval is wide because it has few qualifying carriers.
A fixed-effect summary may be positive, but the cohort table remains essential. Record cases, controls, total carriers, case carriers, mask variants and calibration statistics. The example is fictional: its purpose is to show how apparent disagreement can reflect uncertainty rather than a true opposite effect.
| Cohort | Effect | Standard error | Interpretation |
|---|---|---|---|
| A | 0.42 | 0.14 | Positive, precise |
| B | 0.36 | 0.18 | Positive, compatible |
| C | 0.05 | 0.22 | Imprecise, near zero |
| Rule | Keep all | Show intervals | Do not count votes |
Evidence check: for Worked example: three biobanks, preserve the observation before the interpretation. Record every estimate, interval and calibration check. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Gene-level meta-analysis can combine cohort score statistics and covariance information without moving individual records. This preserves local modelling of relatedness and covariates while allowing a common set test. The covariance matrices must use the same ordered variant manifest and allele orientation.
A combined burden statistic assumes effects are sufficiently aligned for the burden contrast. Variance-component methods tolerate mixed directions. An omnibus result may protect against choosing the architecture after inspecting outcomes, but multiplicity across masks, phenotypes and test families still needs control.
Evidence check: for Combine compatible score information, preserve the observation before the interpretation. Record compatible score and covariance summaries. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
CHAPTER 6 OF 12 · Test what changes
6. Treat replication as estimation, not a second threshold
A replication cohort contributes an estimate, interval and direction, not merely significant or non-significant status. Compare compatible effect scales and examine whether the discovery estimate shrinks. Winner’s curse, different ascertainment and sparse carriers can all change magnitude.
If a replication sample is underpowered, a wide compatible interval is inconclusive rather than negative. Conversely, a precise estimate near zero challenges a broad claim even if the pooled analysis remains significant.
Evidence check: for Treat replication as estimation, not a second threshold, preserve the observation before the interpretation. Record replication as estimation rather than vote counting. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Recompute the combined result after omitting A, then B, then C. Record the new estimate, uncertainty, set-test statistic and mask carrier count. A robust multi-biobank claim should not collapse only because the largest cohort is removed, although power will naturally fall.
Leave-one-out analysis is a dependence diagnostic, not a license to delete an inconvenient cohort. If one omission reverses direction or eliminates calibration problems, investigate phenotype drift, technical batches, ancestry composition and particular carrier clusters.
Evidence check: for Run leave-one-biobank-out validation, preserve the observation before the interpretation. Record leave-one-biobank-out results. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Report cohort estimates and a heterogeneity statistic when the model supports it, but interpret sparse settings cautiously. With only a few cohorts, heterogeneity tests have low power and random-effects variance is unstable. Forest plots, intervals and carrier composition often reveal more than a single heterogeneity p-value.
Distinguish magnitude heterogeneity from directional conflict. A smaller effect in one cohort may be compatible with the same biology under different case definitions. Opposite directions demand closer checks of alleles, coding and mask composition before mechanistic interpretation.
Evidence check: for Measure heterogeneity with context, preserve the observation before the interpretation. Record effect direction and heterogeneity. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
CHAPTER 9 OF 12 · Check evidence and limits
9. Stress-test mask and carrier dependence
Repeat prespecified analyses with a narrower mask, an alternative frequency boundary, removal of one highly influential variant and leave-one-carrier-cluster-out checks where privacy permits. Compare annotation versions without treating the most favourable result as primary.
A set association driven by one variant may still be real, but it is no longer evidence that the entire functional class acts coherently. State that narrower conclusion and direct laboratory follow-up toward the influential variant.
Preserve both the variant-level contribution table and the cohort-level table. They answer different questions. The first asks which alleles supply evidence inside a set; the second asks which sampling and technical environments supply evidence across studies. A stable pooled result can still rely on one variant present in several cohorts, while a broad mask can look stable only because different variants dominate in different places. Those patterns imply different follow-up experiments.
Predefine what counts as materially stable. A useful rule can specify an acceptable range for effect change, direction concordance and the maximum share of the combined score contributed by one cohort. This is not a universal threshold. It is a declared interpretation aid that prevents the standard from being invented after the result is known.
| Check | Changed | Held fixed | Question |
|---|---|---|---|
| Leave one biobank out | Cohort set | Mask and phenotype | Single-cohort driver? |
| Narrower mask | Variant membership | Cohorts | Annotation dependence? |
| Influential variant | One variant | Null models | Variant-driven set? |
| Phenotype mapping | Case definition | Genotypes | Endpoint dependence? |
Evidence check: for Stress-test mask and carrier dependence, preserve the observation before the interpretation. Record variant and mask dependence. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Phenotype mismatch can create incompatible endpoints. Sequencing depth and calling pipelines can make absence look biological. Related participants, site-specific batch effects and population structure can distort sparse tests. Selective reporting can make a failed replication disappear.
Good statistical calibration does not prove causality, penetrance or clinical utility. A replicated gene-level association still requires variant-level, functional and clinical validation before use for individual decisions.
Privacy restrictions can also distort what is visible. Sites may suppress very small carrier cells or round counts. Agree in advance which aggregate diagnostics can be shared and how suppressed cells will be represented. Never infer an exact count from a suppressed interval. A federated audit can verify that all sites ran the same manifest even when individual-level records remain local.
Publication bias matters at the cohort level as well. Keep a registry of invited cohorts, exclusions, failed runs and unavailable phenotypes. If a site cannot contribute because calibration fails, report that fact and diagnose it; do not quietly redefine the consortium to include only successful results.
A final reproducibility bundle should include complete analysis code, software and annotation versions, phenotype mapping, mask checksums, site-level quality-control summaries and the exact meta-analysis command. Where data cannot be shared, publish synthetic test data that exercise the same pipeline. Re-running the bundle should reproduce the reported tables and expose every manual step.
Evidence check: for Know the main failure modes, preserve the observation before the interpretation. Record association, causality and clinical limits. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
CHAPTER 11 OF 12 · Practise and continue
11. Transfer check: change the dominant cohort
Return to the example. Suppose Biobank A grows fourfold while its effect remains 0.42. Predict how the fixed-effect summary changes, then compare that result with an equal-weight descriptive average. Next, hold sample size fixed but change C to -0.40 with a standard error of 0.10. Explain why a pooled p-value alone would be inadequate.
Finally, remove the single most frequent qualifying variant from A. If its estimate falls to 0.08, revise the claim from mask-wide replication to variant-driven evidence requiring targeted confirmation.
Evidence check: for Transfer check: change the dominant cohort, preserve the observation before the interpretation. Record the opposing-effect and dominant-cohort contexts. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.
Does leave-one-biobank-out prove portability? No. It tests dependence on the observed cohorts, not unseen settings. Must every cohort be significant? No; compatible estimates and uncertainty matter. Can a meta-analysis repair incompatible phenotypes? No. Harmonisation is a scientific prerequisite, not a statistical afterthought.
Primary reading includes the 2024 cross-biobank, cross-ancestry rare-coding analysis in Nature Genetics via PubMed, the 2025 MetaSTAAR framework in Nature Genetics, and the foundational gene-level meta-analysis description at arXiv. Continue with eduKate’s verified guides to rare-variant meta-analysis and cross-site harmonisation and cross-ancestry aggregation and calibration.
Evidence check: for FAQs, routes and authoritative sources, preserve the observation before the interpretation. Record the narrowest cross-cohort claim. Then state the inference, one credible alternative, and the additional observation that would distinguish them. This compact audit keeps a fluent explanation tied to evidence that another reader can inspect.
Changed-context transfer: repeat this chapter’s decision after changing one consequential condition while holding the others fixed. Calculate or predict what should change, name the threshold or mechanism, and compare with the original case. If the conclusion survives, state the range tested; if it reverses, explain why. Transfer is demonstrated by the changed case, not asserted from the first example.
Practical decision record: write one sentence for what is known, one for what remains uncertain, and one for the next safe action. Include the evidence owner, the latest useful decision time, and the condition that would stop or reverse the action. This turns a technical result into a reviewable choice without pretending uncertainty has disappeared.