HOW SINGAPORE EDUCATION WORKS · University research, data quality and reproducible checks
Find your next learning step
Translate each important data definition into an executable or manually repeatable test. Separate structural failures from plausible unusual values, preserve raw data, and release analysis only after the validation report is reviewed.
Full chapter index · Practice and checking · Sources and next reading · How Singapore Education Works
A data dictionary says what a field is meant to contain. A validation rule asks whether the actual records obey that definition. Documentation without checks can leave silent errors inside an apparently tidy dataset.
This is a general research workflow, not a substitute for an institution’s ethics, security, disciplinary or supervisory requirements. Validation should minimise exposure of personal data, preserve authorised raw files and document every transformation.
The choices at a glance
| What you notice | Likely issue | Useful next move |
|---|---|---|
| The script runs but counts change unexpectedly | Keys or missing-value rules are wrong | Validate identifiers, row counts and permitted nulls first |
| Every outlier is deleted | Plausibility and validity are being confused | Quarantine unusual values and review context |
| Checks exist only in one researcher’s notebook | The workflow is not reproducible | Store versioned rules, results and decisions with the project |
Full chapter index
Work and diagnose
Repair and practise
Recall, transfer and questions
A schema describes structure: field names, types, formats, allowed values, uniqueness and relationships. A validation suite applies those expectations to a particular data release.
Validity is narrower than truth. A date can follow YYYY-MM-DD and still be entered for the wrong participant. Automated checks locate contradictions and risks; they do not replace domain knowledge or provenance review.
Write rules while designing collection, not only after analysis begins. Early rules can prevent defects instead of merely cataloguing them.
| Rule class | Question | Example |
|---|---|---|
| Structure | Are required fields present? | participant_id exists |
| Type and format | Can the value be interpreted as defined? | visit_date matches ISO date |
| Domain | Is the value among permitted codes? | arm is A, B or NA |
| Uniqueness | Should this key repeat? | participant_id + visit_no is unique |
| Relationship | Do fields agree? | end_time is not before start_time |
| Referential integrity | Does a foreign key have a parent? | every visit participant exists in enrolment |
Give every rule a stable identifier, severity, owner and rationale. A message such as V014: visit number missing is more actionable than “data invalid”.
Distinguish an error that blocks release from a warning that requires review. Severity should be decided before seeing whether a favourite result is affected.
Assumptions: the enrolment table has 120 unique participant IDs. The visit table should contain one row per participant and visit number, visit numbers 1–4, dates on or after enrolment, and scores from 0 to 40 or missing with an approved reason.
| Check | Observed result | Decision |
|---|---|---|
| Composite key unique | 2 duplicated participant_id + visit_no pairs | Block release and reconcile duplicates |
| Participant exists in enrolment | 1 unmatched ID | Block and inspect import/provenance |
| Visit number in 1–4 | All pass | Pass |
| Score domain | One value is 44 | Quarantine; verify source rather than force to 40 |
| Missing reason present | 3 missing scores, 2 reasons | Block one incomplete record |
Checked answer: this table is not ready for analysis. At least the duplicates, unmatched identifier and undocumented missing score must be resolved; the score of 44 needs source review because it may be a data error or an outdated rule.
Do not silently coerce 44 to 40 or drop the row. Preserve the received value, the decision and any corrected release.
When many records fail together, inspect the rule version, locale and import process before labelling all records wrong. A day-month conversion or changed code list can produce systematic false failures.
When one record fails, trace it through authorised source material without placing identifiers in logs. Use row keys or protected references appropriate to the project.
When two rules conflict, return to the authoritative protocol and data dictionary. Record which definition governs the release and why.
- Keep the authorised raw deposit immutable.
- Import into a controlled staging layer.
- Run versioned validation rules and save a summary report.
- Quarantine failures without deleting them.
- Apply approved corrections in code or a logged change table.
- Rerun the full suite and release a versioned validated dataset.
Record software versions, rule-set version, input checksum, execution time and row counts. These details make later reproduction and audit possible.
Reports should disclose enough to repair data without leaking sensitive values. Aggregate counts and protected row references are usually safer than copying complete records into error messages.
Guided
Translate three dictionary entries into one structure rule, one domain rule and one relationship rule.
Shared
A researcher writes rules while a collaborator supplies deliberately valid and invalid test rows.
Independent
The researcher runs the suite on a clean copy, reviews every failure and submits the report with the analysis-ready release.
At the next data release, rebuild the critical-rule checklist before opening the old script. Then compare it with the maintained suite. This tests whether the team understands the controls rather than merely reruns them.
Changed-context transfer: apply the same rule classes to survey exports, laboratory instrument files or qualitative coding tables. The domains differ, but keys, formats, permitted values and provenance still need explicit treatment.
International repositories and collaborators may use different metadata standards or tools. Exchange the schema, rule definitions and validation report in documented, portable formats instead of assuming one local software environment.
Is validation the same as data cleaning?
No. Validation detects departures from defined expectations. Cleaning is the controlled process of resolving or documenting them.
Should every warning block analysis?
No. Predefine severity. A plausible extreme value may require review but remain valid.
Can spreadsheet validation be enough?
It may help small projects, but rules and outcomes must remain repeatable, reviewable and versioned. Manual steps should be documented explicitly.
What belongs in the repository deposit?
Follow consent, ethics, repository and institutional requirements. Where permitted, deposit documentation, code and non-disclosive validation summaries alongside appropriate data products.
Sources and next reading
- NUS Libraries — Research Data Management: Metadata
- NUS Libraries — Research Data Management
- UK Data Service — Prepare your data for deposit