EDUKATE SINGAPORE · GOLD STANDARD PRACTICAL CASE STUDIES · CASE 36
A correct-looking chart is not reproducible without its actual method.
Student researchers obtain eighteen flagged observations in one version of a study and seven when the notebook is rerun. This fictional Gold Standard case shows how data identity, software versions, parameters and responsible correction preserve scientific meaning.
The environmental observations, people, numbers and research findings are invented for education. The exercise is not a statement about actual water quality, a public-health assessment or external certification of a study.
Chapter index
- 1. The school research team cannot regenerate its own chart
- 2. A spreadsheet name is not a reproducible experiment
- 3. What reproducibility means in this exercise
- 4. Find the version break rather than guessing the cause
- 5. Decide what the presentation may now say
- 6. The difference between a repair and a cosmetic rerun
- 7. An evidence pack small enough for student researchers
- 8. A check by someone who did not write the notebook
- 9. From student research to public claims
- 10. Practice: what does eighteen versus seven actually prove?
1. The school research team cannot regenerate its own chart
A fictional student environmental research group studies trends in a set of collected water-quality observations from a local learning project. Its goal is to practise careful scientific reasoning, record how measurements become a chart and explain what conclusions the data might or might not support. The group presents a graph showing eighteen observations meeting a predefined screening flag under its analysis rules.
Three weeks later, another student opens the archived input file and reruns the project on a different laptop. Their software reports only seven flagged observations. The new result comes from apparently the same data and the same notebook title. The original analyst is confident the chart was correct. The second student is confident the new output is what the notebook now produces.
Neither confidence resolves the contradiction. The issue may involve code changes, input differences, software dependencies, missing-value treatment or parameters, and the group has not preserved enough context to know which. This Gold Standard case is about making the path from evidence to conclusion reproducible, not about deciding actual environmental safety from a fictional chart.
2. A spreadsheet name is not a reproducible experiment
The team finds three files named `final_data.csv`, two analysis notebooks with nearly identical titles and a chart image copied into the final report. The notebook automatically updated software packages when run on the new computer. A student recalls changing the missing-data handling rule after the presentation, but the version history does not show exactly when the change was saved.
A screenshot of eighteen flagged records proves that the earlier output existed. It does not fully identify the exact input bytes, code, parameters or computational environment needed to reproduce it. The new result of seven similarly proves only what the second environment produced, subject to proper checking. A file’s familiar name is not a stable identifier for its content.
The first diagnostic move is to preserve both present states rather than repeatedly editing files to make the chart appear consistent. The reviewer copies the available inputs, scripts, notebook exports, configuration information and output artefacts into a controlled investigation record. Each file is dated, attributed and distinguished from later reconstructions.
The group also separates original observations from derived results. The field collection record says what was measured or reported; the analysis script determines how observations were filtered and flagged. A discrepancy in the second does not automatically prove an error in the first, although it may prompt a review of collection methods too.
3. What reproducibility means in this exercise
The US National Academies of Sciences, Engineering, and Medicine’s Reproducibility and Replicability in Science distinguishes computational reproducibility from the separate question of whether a new independent study obtains consistent results. It recommends sharing or adequately describing underlying data, methods, parameters and computational environment as appropriate.
For this school project, computational reproducibility asks whether another authorised person can obtain the documented analysis result from the same relevant inputs and method. Replicability would involve appropriately collecting new observations in a comparable study to see whether the result is consistent. The two concepts should not be merged in a publicity sentence.
Even a fully reproducible graph can be based on a biased sample, an unsuitable measure or a flawed question. Reproduction establishes that an analysis path can be repeated; it does not certify truth, good study design or scientific importance. The Gold Standard must keep evidence quality and analytical repeatability linked without pretending they are identical.
The student’s first task is therefore narrower than ‘prove which laptop was right’. It is ‘establish exactly which data and method produced each output, and decide what the existing report may honestly claim while that comparison is unresolved’.
4. Find the version break rather than guessing the cause
The group creates a comparison table with columns for input file identifier, recording date, script version, parameter values, relevant software versions, run date and output summary. The old chart has incomplete entries. The new run can be documented more completely, but the missing information in the old run cannot be filled with unverified guesses.
Suppose investigators discover that the newer notebook removes rows with missing values before a screening calculation, while a saved earlier cell suggests the previous notebook included a defined substitute value. This is a plausible cause of the eighteen-versus-seven difference, but it is not confirmed until the group can reproduce each behaviour from suitable preserved inputs and code states.
A second possibility is that `final_data.csv` was overwritten with a cleaned file that excluded duplicate or invalid observations. A third is that a library update altered parsing or grouping behaviour. The investigation should test candidate explanations separately, rather than change several variables together and celebrate when one chart looks familiar.
GS-R08 requires traceable observations. GS-R12 requires controlled change. When an analyst adjusts a cleaning rule, that change must become visible in the interpretation of outputs rather than silently rewriting the historical meaning of an earlier result.
5. Decide what the presentation may now say
The project supervisor asks the students to qualify the published statement that eighteen observations met the flag criterion. The current archive does not yet provide a complete reproducible method for that specific count. The team can preserve the original output as a historical artefact and disclose that its analysis is under reconstruction.
The supervisor should not replace eighteen with seven in every document simply because the latest computer produced seven. Nor should the group use the older screenshot as proof that eighteen is the authoritative answer. The decision needs a review of the actual data and approved method, with an explanation of differences in inputs and procedures.
If the chart was used only in a classroom exercise, the correction can be proportionate. If it informed a real decision, different competent and potentially urgent procedures might apply. This fictional case does not establish real water safety or public-health limits and must not be used to issue an environmental advisory.
A useful interim statement is: ‘We found that the published count does not presently reproduce from the available archived analysis. We are preserving both versions and checking the input data, transformation rules and software environment. The figure should not be used as a verified operational conclusion until the analysis is resolved.’
6. The difference between a repair and a cosmetic rerun
One student proposes editing the new script until it produces eighteen again. That could manufacture apparent consistency without finding why the earlier result occurred. Another suggests deleting the older chart, leaving only the new result. That would remove relevant history and weaken accountability. Both approaches treat appearance as more important than understanding.
The supervisor instead asks for a controlled reconstruction. The team first fixes the input file or defines a clearly versioned input set; records a transparent analysis specification; runs it in a documented environment; then checks that another authorised person can repeat the result. Differences between historical and reconstructed methods are described, not erased.
Where source data include personal, sensitive or otherwise restricted information, access controls and appropriate permissions must still apply. Reproducibility does not give everyone unlimited access to protected records. Depending on the project, a synthetic example, controlled environment or sufficient metadata may support legitimate review without exposing confidential data.
GS-R10 asks whether the problem is corrected and recurrence prevented. A rerun that happens to match once is weak repair if the team still cannot explain which code and data created it.
7. An evidence pack small enough for student researchers
The group designs a compact research-run record. It contains a project question, collection protocol, input data identifier, transformation steps, analysis code version, parameters, relevant software dependencies, timestamp, person responsible and the output to which those details apply. Changes carry a short explanation of why they were made and which results they affect.
For a small project this can be a structured folder with read-only releases and a simple changelog. For a more advanced project, version-control software and environment configuration files can improve consistency. The tool is not the Gold Standard; the ability to reconstruct and explain an analysis is. A beautiful repository that no student can actually run may still fail the purpose.
The team keeps raw observations distinct from processed tables, flags and presentation graphics. It also documents definitions such as missing values, repeated observations and the unit or category used in a calculation. A chart without a stable denominator can mislead even when the software executes perfectly.
The student who joins midterm should be able to locate the current released method, understand the changes and identify which published chart belongs to which run. That is a test of knowledge continuity as much as technical precision.
8. A check by someone who did not write the notebook
After the reconstruction, a classmate who was not involved in the initial coding receives the approved input and analysis instructions. The reviewer attempts to regenerate the summary count and a representative chart in a documented environment. The reviewer also tries to identify which assumptions most affect the result, especially handling of incomplete observations.
A successful run supports a claim of computational reproducibility for the released files and specified environment. The classmate should record any manual steps or undocumented knowledge needed. If the result matches only after a private explanation by the original author, the documentation still needs improvement.
The review should include a deliberate minor change to an input or parameter in a sandbox copy to confirm that the workflow records the resulting change rather than overwriting the earlier release. A team that cannot retain past versions will struggle to compare later analytical decisions.
Repeatability under one environment does not establish robustness across all software or future updates. The release record can state the tested conditions and define when a significant change should trigger re-verification.
9. From student research to public claims
The group considers what it wants to tell a wider audience. A cautious educational summary might describe the question investigated, how observations were collected, what analysis method was checked, what is still uncertain and why the result should not be used as a real-world safety judgement. That is more valuable than an impressive but unexplained number.
If the reconstructed figure differs from the historical chart, the team should explain what changed and which conclusion is now supported. It must not claim the original researchers were dishonest without evidence. Files change accidentally; software behaves differently; people make ordinary mistakes. A Gold Standard research culture finds and corrects those problems without turning a discrepancy into either a witch hunt or a cover-up.
New analyses may improve a project, but they should not retroactively erase the limitations of earlier outputs. A clear dated correction demonstrates that the evidence and the claim are still connected. That is how scientific learning becomes trustworthy across changes in people, tools and time.
10. Practice: what does eighteen versus seven actually prove?
Exercise. The earlier presentation reported eighteen flagged observations. A later computer produces seven from files with apparently identical names. The original environment and versions were not recorded. Write the supported finding, immediate publication restriction, reconstruction steps and one acceptance test. Avoid declaring either count definitively correct.
Reasoned finding: ‘The historical result is not currently reproducible from the available archived computational materials. The difference may arise from input, code, parameter or environment changes, and the cause has not been established.’ The immediate publication action is to qualify the affected result and preserve both output histories.
The repair is to identify stable data versions, inspect transformation and filtering rules, record the relevant code and environment, and create an attributable analysis release. A credible acceptance test gives the release package to someone else, who generates the specified result without relying on undocumented author memory and records exactly what worked.
The Gold Standard lesson is that a result can look exact while its path of production is uncertain. Dependable research keeps data, method, version, environment and conclusion connected so another person can check not only the number, but also what that number means.
Further reading and reproducibility reference
- National Academies · Reproducibility and Replicability in Science
- Gold Standard of Biotechnology · Reproducible Evidence
- MOE V3.0 · Research Methods and Ethics Literacy
- Gold Standard · Evidence, Claims and Verification
- Gold Standard · Information and Record Integrity
- Case 29 · Calibrated Sensor in the Wrong Room
- Case 32 · A Gold Standard Claim Outlived Its Evidence
- Gold Standard Casebook and Decision Practice
- Gold Standard · Master Hub
The National Academies report is cited for its definitions and general computational research guidance. The school case is an original fictional educational exercise; scientific and environmental decisions require appropriately designed, competent studies.