HOW SINGAPORE EDUCATION WORKS · University data-cleaning decision logs
Find your next learning step
The practical answer is to keep raw data unchanged and add one dated log entry for every cleaning rule: issue found, decision made, reason, affected records or variables, action or code, check performed and output version. The log explains the transformation; it does not replace the script, cleaned file or research protocol.
Full chapter index · Practice and checking · Sources and next reading · How Singapore Education Works
This is a research-practice guide, not a substitute for a university's ethics, data-management, security or repository requirements. If personal or sensitive data are involved, use the approved storage and disclosure controls before sharing any example or log.
The choices at a glance
| What you notice | Possible issue | Useful next move |
|---|---|---|
| Cell changed but no reason recorded | The cleaned result cannot be audited | Log issue, rule, action and affected records |
| Raw file is overwritten | Original evidence and transformation are mixed | Keep raw data read-only and create a versioned output |
| Missing becomes zero | Absence and measured zero are confused | Preserve the source meaning and document any recode |
Full chapter index
Protect raw data and define the log
Work a cleaning decision
Check, repair and practise
Reuse, hand over and questions
CHAPTER 1 OF 8 · Protect raw data and define the log
1. Cleaning is a claim about evidence
Changing “S'pore” to “Singapore” may look harmless, but it is still a decision that two strings represent the same category. Removing a duplicate asserts that two rows refer to one record. Replacing a blank with zero asserts that absence means a measured value of zero — an assertion that is often false.
A decision log makes these claims inspectable. It lets the researcher, supervisor or future collaborator see what changed and why. The aim is not to produce a heroic diary of every mouse click. It is to preserve decisions that can affect interpretation, counts or reproducibility.
Keep three layers distinct: the untouched raw input, the transformation process, and the cleaned output. A log connects them.
CHAPTER 2 OF 8 · Protect raw data and define the log
2. Use a minimum complete entry
A useful entry has an identifier and date; the dataset and version; the issue; the decision rule; the rationale; the variables or record IDs affected; the action or code reference; the validation check; the person responsible; and the resulting output version.
Avoid copying sensitive values into the log when an internal record identifier will do. If a script performs the change, record the script name, version or commit and the relevant step. If the change is manual, state how it was selected and independently checked.
Write the rule before writing the success statement. “Fixed dates” is too vague. “Converted valid ISO-like text in visit_date to YYYY-MM-DD; quarantined three ambiguous day/month values for review” can be tested.
CHAPTER 3 OF 8 · Work a cleaning decision
3. Worked case: duplicate survey submissions
Assumptions: a fictional survey export has 1,204 rows; participant_code should be unique; two rows share code P-071; timestamps differ by four minutes; all substantive answers are identical; the study protocol says an accidental resubmission may be removed after review; no personal identifiers are placed in the teaching example.
A defensible log entry states: issue — possible duplicate P-071; rule — where participant_code and all response fields match, retain the earliest complete submission and quarantine the later row; reason — protocol treats exact accidental resubmissions as one response; action — cleaning script step dedupe_exact_v1; affected rows — source rows 311 and 312; validation — row count falls from 1,204 to 1,203, unique participant_code count remains 1,203, and a duplicate scan returns zero exact pairs; output — survey_clean_v03.csv.
Checking answer: the arithmetic reconciles. One row is removed, so 1,204 − 1 = 1,203. The unique participant count does not fall because the two rows shared one participant code. The quarantined row remains available for audit under the approved access controls.
CHAPTER 4 OF 8 · Work a cleaning decision
4. Missing, impossible and unusual are different
A blank can mean not asked, declined, not applicable, system failure or genuinely missing. Zero can be a real measurement. Do not collapse these meanings for convenience.
An impossible value violates a defined rule: for example, a study visit dated before enrolment when the protocol requires the reverse. An unusual value may still be true. A very long commute should not be deleted merely because it is rare.
Record the source rule and the action. Possible actions include retaining with a flag, querying the source, correcting from an authorised record, recoding to a documented missing category or excluding from a named analysis. “Looked wrong” is not a reproducible rule.
If the cleaned file cannot be recreated, check whether the script version and input version were recorded. If counts changed without explanation, reconcile rows before and after each major step.
If a category was merged, ask whether the mapping table is saved. If dates were standardised, check ambiguous day/month cases separately. If a manual correction was made, identify who approved it and what evidence supported it.
If the log is enormous but still unhelpful, group repeated deterministic changes under one documented rule and keep machine-level details in code or structured output. The human-readable log should point to the executable evidence, not imitate it poorly.
CHAPTER 6 OF 8 · Check, repair and practise
6. Guided practice and independent logging
For guided practice, use a fictional column course_mode with values “Online”, “online ”, “On-line” and blank. Define the project-approved categories before changing anything. A sample decision might standardise case and trailing spaces, map “On-line” to “Online” only if documentation confirms equivalence, and retain blank as missing rather than inventing a category.
For an independent task, give a 20-row fictional table containing one exact duplicate, one impossible date order, one unusual but plausible age and two blanks with different source reasons. Ask the learner to propose a log entry for each decision. The answer must state what remains unresolved.
Checking answer: a strong response removes or quarantines the exact duplicate under a stated rule, flags the impossible date for source review, retains the plausible unusual age unless evidence says otherwise, and preserves distinct missing meanings.
CHAPTER 7 OF 8 · Reuse, hand over and questions
7. Recall, handover and changed-context transfer
One week later, hide the cleaned file and ask the researcher to reconstruct the decisions from the log. Then give the log and approved files to a teammate who was absent. Can that person identify the input, run or inspect the transformation, reproduce the counts and explain unresolved cases?
Transfer the method from a survey to interview metadata, sensor readings or image annotations. The details change, but the questions remain: what was observed, what rule was applied, why was it defensible, what changed, and how was the result checked?
Global connection: research data may move between Singapore teams, overseas collaborators, journals and repositories. Clear documentation reduces dependence on private memory and helps another user understand how a release version was created. It does not remove ethical or legal limits on reuse.
Is a script enough?
No. Code shows operations but may not explain the research reason, source evidence, unresolved ambiguity or institutional approval. Link code and log.
Should raw data ever be edited?
Keep an immutable or access-controlled original according to the project plan. Work on copies or versioned derivatives. Follow institutional requirements for corrections, security and retention.
Must every changed cell appear in the log?
Not necessarily. One rule can cover many deterministic changes if affected counts or identifiers, code and checks make the scope inspectable. Sensitive values should not be exposed unnecessarily.
Does a clean dataset mean a correct dataset?
No. Cleaning can improve consistency and usability, but it cannot guarantee truthful measurement or valid inference. Preserve limitations and unresolved issues.
Sources and next reading
UK Data Service: Preparing data for sharing and reuse — official guidance distinguishing cleaning for sharing from analytical cleaning and stating that changes should be documented.
UK Data Service: Documenting and organising research data — institutional guidance on documentation, data validation, cleaning, manipulation, versioning and ReadMe files.
UNESCO project on scientific data literacy and reproducible research — a global institutional connection to reproducible research capability; it is not a cleaning protocol for a specific university project.
University research data: count people, visits and records correctly — a companion guide to defining the unit before counting.
University questionnaire codebooks — a companion guide to keeping categories and evidence inspectable.
Learning at other stages
Continue with a different learning task in this series. Each guide has its own worked cases, practice and limits.