HOW SINGAPORE EDUCATION WORKS · University research data, traceable cleaning and reproducibility
Find your next learning step
Preserve the source, define one cleaning rule, log what changed and why, check affected records and create a new version that another researcher can reproduce.
Full chapter index · Practice and checking · Sources and next reading · How Singapore Education Works
Data cleaning can improve consistency while also changing evidence. Replacing values, recoding categories, excluding rows or standardising dates should therefore leave a trace: the input version, rule, affected records, reason, reviewer and check result.
This is a general research workflow, not a replacement for university, discipline, ethics, funder, security or data-protection requirements. Preserve approved source data, use authorised environments, avoid exposing sensitive values in logs and follow the project’s review and retention rules.
The choices at a glance
| What you notice | Likely issue | Useful next move |
|---|---|---|
| A cleaned file has fewer rows but no explanation | Exclusions are not traceable | Log the rule, count and stable record identifiers |
| A value was corrected by hand | The transformation cannot be replayed | Turn the edit into a documented rule or exception table |
| The script runs but the result changed | Input, code or environment version may differ | Record hashes or versions and compare summary checks |
Full chapter index
See the structure
Work and diagnose
Repair and practise
Recall, transfer and questions
A dataset does not become neutral because it is tidy. A rule that converts “N/A” to missing, merges category labels or removes an outlier can alter counts, models and interpretation. The decision needs the same care as the final chart.
The UK Data Service advises that corrections and exclusions should be recorded so changes can be traced and understood. NUS research data management guidance similarly places integrity, validation and retrievability inside responsible data stewardship.
Keep source, working and released layers distinct. A cleaning output should not silently replace the approved source file.
| Field | Example | Purpose |
|---|---|---|
| Change ID | CL-017 | Stable reference |
| Input version | survey_raw_v03 | Names the source state |
| Rule | Trim surrounding spaces in district_code | States the transformation |
| Affected records | 42 rows; IDs in exception file | Shows scope without exposing values |
| Reason | Codes failed documented lookup | Explains necessity |
| Method | clean.R lines 84–91; commit 7ac2… | Enables replay |
| Check | Distinct codes 18→12; all in dictionary | Tests the result |
| Reviewer and date | AK; 2026-10-24 | Records responsibility |
For sensitive data, reference protected exception files or counts instead of copying confidential values into a broadly accessible log.
One log row may represent one rule applied to many records. Record individual exceptions only when they need separate justification.
Assumptions: 1,000 records should contain ISO dates. Nine values use day/month/year text, two are impossible dates and three are blank. The source file is read-only. A documented rule parses unambiguous text dates; impossible values become missing with an error code; blanks remain missing.
The output contains 995 valid dates and 5 missing values: the three original blanks plus the two newly flagged invalid values. The log records counts before and after, the parser version, affected IDs in a protected exception file and a range check.
Checked result under these assumptions: 986 dates were already valid, 9 text dates were standardised, 2 impossible dates were not guessed and 3 blanks remained missing. The reconciliation is 986 + 9 + 2 + 3 = 1,000 records; valid = 995 and missing = 5.
The arithmetic check matters because even a sensible cleaning rule can hide a row-count error if the before-and-after totals are not reconciled.
CHAPTER 4 OF 8
4. Diagnose overwritten sources, hidden exceptions and drifting rules
If only the final file exists, provenance is broken. Restore the approved source from controlled storage and rebuild the output if policy permits. Do not fabricate a retrospective log that claims certainty you do not have.
If a cleaning rule has dozens of manual exceptions, ask whether the rule is wrong, the source contains distinct valid cases or the data dictionary is incomplete. Exceptions are evidence, not clutter to hide.
If reruns differ, compare input version, code commit, configuration, software environment and ordering assumptions before blaming randomness.
- Preserve the approved input as read-only or version controlled.
- Specify the rule, scope, reason and expected effect before running it.
- Transform data through a script or controlled exception table.
- Reconcile row counts, missingness, ranges, categories and key summaries.
- Review the log and output, then release a new named version.
Automated checks should fail loudly when assumptions break. A script that completes is not the same as a transformation that is scientifically appropriate.
Never use a change log as permission to retain personal data beyond the approved purpose or access period.
Guided
Review a prepared before-and-after pair, identify the rule and recompute the affected count.
Shared
Write one cleaning rule, exchange logs with a partner and reproduce each other’s output from the same non-sensitive input.
Independent
Produce a versioned cleaned dataset, code, exception handling record, summary reconciliation and reviewer-ready change log under the project’s governance.
After one week, reconstruct the five verbs: preserve, specify, transform, reconcile, review. Then inspect an old log and ask whether a new team member could identify the exact input and rerun the change.
Global repositories and disciplines use different tools, but the evidence questions recur: What changed? Why? Which records? Under which rule? With what check? That shared logic supports transparent collaboration without pretending every field uses one standard workflow.
Changed-context transfer: apply the same log design to qualitative transcript redactions. Record scope and reason without exposing the removed confidential text.
Should every formatting change be logged?
Use project rules and materiality. Changes that affect interpretation, linkage, inclusion or reproducibility need clear records; repetitive cosmetic changes may be grouped.
May I correct the raw file directly?
Usually preserve an approved source and write a new version. Follow the institution’s exact data-management rules.
Is a script enough documentation?
No. Code shows how, but a log should also capture why, scope, checks and review.
What if a correction cannot be reproduced?
Flag it, preserve evidence and seek review. Do not present an unexplained manual edit as a verified rule.
Sources and next reading
- NUS — Research Data Management
- UK Data Service — Data Quality and Integrity
- UK Data Service — Managing and Sharing Data: Best Practice for Researchers