The gold standard of Data Quality is information that is fit for a stated use, with limitations that its users can see and manage. Six dimensions—completeness, uniqueness, consistency, timeliness, validity and accuracy—help locate defects, but the best test is whether a real downstream task can be completed on the right evidence.
This root guide walks from purpose and record definitions to common failure patterns, then works through school registers, grading scales, laboratory units and community surveys. It provides an eight-question audit and links to deeper eduKateSG guides about data governance, systems, research and learning.
All examples are illustrative rather than descriptions of actual student records or institutions. The UK Government Data Quality Framework is a helpful public reference, not a requirement for every organisation or a substitute for applicable privacy duties.
Find the part you need
Begin with the data decision that matters, then follow the dimension, failure mode or worked case closest to your problem.
Chapter index
Define the decision
Build truthful records
Interrogate gaps
Work through cases
CHAPTER 1 OF 23 · Define the decision
1. The purpose determines which data are good enough
A dataset is high quality when it is sufficiently trustworthy and suitable for the purpose for which somebody intends to use it. “Complete”, “clean” and “large” can describe desirable properties, but none alone makes data fit for a decision. A school attendance record that works for daily classroom registration may need more context before being used to explain why pupils missed particular lessons.
The gold standard begins with the downstream action. Who uses these data, what conclusion or operation depends on them, and what harm might follow from an error? A room-booking system needs the right location and time before an event; a research dataset needs unambiguous variables and sampling information before a calculation.
Distinguish data quality from the overall quality of a scientific argument. A well-formed dataset can be analysed using an inappropriate method; poor-quality inputs can produce precise-looking outputs. This root connects Data, Data Governance, Quality and Statistics without treating those topics as interchangeable.
CHAPTER 2 OF 23 · Define the decision
2. Identify the critical records, fields and decision boundaries
Start with the few information elements that can change the result. For a tuition timetable, those might be the learner, correct subject level, allocated tutor, valid session time and room. An optional preference note should not be given the same operational weight as the identity of the student booked for the lesson.
Write a miniature data contract that identifies the source, intended use, valid range, owner, change process and acceptance check for each critical field. Ask the receiving tutor to use a representative record without prompting. Unspoken definitions often become visible only when someone must act.
A practical check is to try a deliberately incomplete but realistic row. Can the downstream user identify what is missing, and is there a safe way to resolve it? The Interoperability guide shows why transferring a field does not automatically preserve its meaning.
The UK Government Data Quality Framework describes six commonly used dimensions: completeness, uniqueness, consistency, timeliness, validity and accuracy. They are different questions, not six names for the same favourable score. An item can pass one and fail another.
Completeness asks whether expected records and values are present; uniqueness checks unintended duplication; consistency asks whether related values contradict each other; timeliness concerns the period or update delay; validity checks stated rules; accuracy compares recorded values with reality.
Choose thresholds against the relevant use and consequence. A contact number can match the required number of digits yet belong to the wrong person. A unique student ID can still contain an incorrect birthday. Avoid averaging all six percentages into an attractive number that hides the one defect affecting the actual decision.
Imagine 120 expected parent emergency-contact records. The system contains usable entries for 96 of them. Field completeness is 96 ÷ 120 = 80%. This is an honest descriptive statistic; it does not imply that four in five phone numbers would reach the correct parent.
The next test examines accuracy against suitable evidence, such as a contact-confirmation process. If 12 of those 96 numbers are outdated, the verified-current share of the expected set would be 84 ÷ 120 = 70% under this fictional example. Document both the denominator and the verification method.
Never repair a blank phone number by inventing a plausible value. Record the gap and provide an authorised contact-update route. When critical records are missing, an operational fallback may be needed while the quality issue is corrected at its source.
CHAPTER 5 OF 23 · Build truthful records
5. Find duplicates without merging different people
Uniqueness concerns the entity being represented. Repeated names are not necessarily duplicated people; a learner may legitimately appear in several attendance records and several students may share the same name. Define the unit of record before deciding what qualifies as a duplicate.
A hypothetical membership list contains “A. Tan” and “Alicia Tan” with the same date of birth. Those similarities justify further review but not an automatic merge. A mistaken merge can transfer notes, attendance or permissions between different people. Conversely, the same student registered under two spelling variants can distort counts.
Choose stable identifiers where appropriate, document match rules and preserve unresolved cases. A person able to explain why records were joined is more valuable than an algorithm that silently produces a tidy list. See Database Administration and Information Systems.
CHAPTER 6 OF 23 · Build truthful records
6. Define consistency across datasets and time
Consistency means values that describe the same fact do not contradict each other without an explained reason. If a room is marked available in one booking system and reserved in another, both interfaces may be internally valid while the school’s schedule remains unusable.
Before deciding which value is wrong, ask whether they refer to the same moment and same underlying state. A student’s class changed after the morning register was exported; an older snapshot can be historically correct but operationally stale. A snapshot needs an as-of time and an authorised update path.
A reconciliation process identifies the authoritative source for the current decision and preserves reasons for changes. Coordination and Systems Integration show why the dispute often sits between systems, not inside either database.
CHAPTER 7 OF 23 · Build truthful records
7. Verify the meaning of units, codes and dates
A number without its unit or reference period can be valid syntax and bad information. A height of 165 might mean centimetres or inches; a financial figure without a currency is open to several interpretations. A date of 03/04/2026 can be read differently in systems using different day-month conventions.
Design an explicit representation such as an unambiguous ISO date where appropriate, and carry units and conversion rules with data rather than guessing from the surrounding context. When the data are imported, test that the interpreted value agrees with the source’s meaning.
A quality check should use counterexamples: a date whose day and month are both 12 or less; a measurement that would be plausible in either of two units. This catches errors that a routine easy example does not reveal. Measurement supplies the wider logic.
CHAPTER 8 OF 23 · Build truthful records
8. Recognise timeliness as a fit-for-use judgement
Old data are not automatically poor data. A historical school register may be precisely what a researcher needs. The same register could be unsuitable for locating a pupil this afternoon. Timeliness must be described in terms of collection date, update lag and the decision’s permitted staleness.
Suppose a school requires emergency contacts to be updated in its system within two working days after a verified change. A notice received on Monday and recorded on Thursday breaches that rule even if Thursday’s value is accurate. The quality issue lies in the delayed process.
Give time-sensitive records a source timestamp and an update status. Do not replace historically correct observations whenever the present changes; preserve both the observation and its effective period so future analyses remain interpretable.
CHAPTER 9 OF 23 · Build truthful records
9. Distinguish format validity from accuracy
A field may satisfy every data-type and range rule while remaining factually wrong. A birth date that is syntactically valid can describe another student. A score between 0 and 100 can still be entered for the wrong assessment. Validation rules catch an important class of mistakes, not every mistake.
Build layered tests: syntax and range checks at entry, cross-field rules where relationships are defined, source verification for high-consequence values and downstream acceptance checks for the real task. Avoid overly restrictive rules that reject legitimate outliers as impossible.
The Verification root explains how to turn requirements into checks. A useful rule should tell the operator what failed, why it matters and what correction or escalation is permitted; a generic red error message is rarely enough.
CHAPTER 10 OF 23 · Build truthful records
10. Preserve provenance, changes and source limitations
Provenance answers where the information originated, how it was transformed and which version is being read. A dataset copied through three spreadsheets can acquire a corrected category, a new rounding convention or missing columns without anyone documenting the change.
Keep an original or authoritative source where appropriate, record transformations and use a repeatable preparation procedure. If a correction is made, note what evidence justified it and which outputs must be recomputed. A traceable correction is stronger than an unexplained “cleaned” file.
This is also an educational skill. When pupils gather measurements, ask them to preserve observations before calculating averages. Their teacher should be able to trace a plotted point back to a dated record rather than accept a result because the graph looks polished.
Missing values have different meanings: not collected, refused, unknown, inapplicable, lost in transit or suppressed for protection. Replacing all blanks with zero changes the question. A missing family-income value is not equivalent to a reported income of zero.
Use explicit codes or categories where they help, but do not create an elaborate taxonomy that workers cannot maintain. Record which fields can be absent for a legitimate reason. Before imputing a value for analysis, explain the method and the assumptions it needs.
A classroom survey where absent pupils are more likely to be struggling may systematically exclude a relevant group. The average of those who replied is not necessarily representative of every pupil. Statistics and Research explain why missingness can affect the meaning of an estimate.
CHAPTER 12 OF 23 · Interrogate gaps
12. Ask whether the population actually matches the question
A correct dataset can still omit the people or events the decision is intended to represent. If a parent survey is distributed only through a smartphone app, households that cannot access it may be underrepresented. No amount of column cleaning can directly repair that coverage limitation.
Define the target population and the observed population separately. Record selection procedures, exclusions and response patterns. If subgroup comparisons are planned, verify that categories mean the same thing in each collection wave and that disclosure does not compromise privacy.
Data quality therefore crosses into evidence quality. Documenting the source and scope is essential even when every row is correctly typed. The Evidence guide helps distinguish a truthful dataset from a broader claim the dataset cannot support.
Two registries may each be accurate for their original purpose and incompatible when combined. One system records a course “completion” at the last session; another records completion only after the final assessment has been passed. Joining them under one completed flag can create an unsupported comparison.
Before integrating, align definitions, units, reference periods, identifiers and source responsibilities. Record which values were transformed and which remain non-comparable. When a mapping would require an invented assumption, surface the mismatch rather than silently forcing equivalence.
The Education Data Interoperability and Learner Transfer specialist article is a useful example: a student’s progression must retain its meaning when the record changes systems. Interoperability develops this interface problem more generally.
CHAPTER 14 OF 23 · Interrogate gaps
14. Use quality rules at the source and check at every handoff
End-of-pipeline cleaning can hide the origin of recurring errors. If an assessment form invites staff to enter percentages in a field intended for raw marks, fixing the export every week consumes time without addressing the confusing interface.
Improve the form, instructions or data model where the mistake arises, then monitor whether the correction holds. Add receiving checks for failures that can arise during transport or processing. A check at one point does not guarantee that later transformations preserve the same quality.
The UK Data Quality Framework guidance describes root-cause work and action plans. Local teams can adapt that principle: log the defect, locate the creating step, assess its impact and make a source-level change that can be tested with actual work.
CHAPTER 15 OF 23 · Interrogate gaps
15. Report quality in a way that changes a decision
A dashboard that says “97% quality” hides the definition, sample and unmeasured problems. Different users need different dimensions. An emergency-contact manager may prioritise accuracy and timeliness; a longitudinal researcher may need stable identifiers, consistent definitions and transparent missingness over many years.
A useful quality statement names the dataset version, assessment date, rules tested, denominator, results, known exceptions and appropriate uses. It should also state unsuitable uses. A number without those boundaries can create more confidence than the evidence warrants.
For example: “Of 120 expected contacts, 96 were present; 84 had been verified current; the remainder require follow-up.” This is more actionable than “data quality below target”. Pair the statement with owners and the next review condition; see Measurement and Performance Review.
CHAPTER 16 OF 23 · Work through cases
16. Worked case: two class registers disagree about attendance
A fictional Secondary 1 school has an attendance sheet and an online class portal. One lists a pupil absent on a Tuesday; another marks them present. A coordinator changes the portal value to match the sheet and considers the problem fixed.
The records actually describe different events: the pupil was present at assembly but left before the afternoon lesson. The correct solution may require separate attendance definitions, clear timestamps and responsible handoffs, not choosing one record as always superior.
To test the repair, use a day on which a pupil is present for some activities and absent for others. Can authorised staff describe the true sequence without forcing it into one misleading label? The case illustrates how a consistency problem can be a data-design problem.
CHAPTER 17 OF 23 · Work through cases
17. Worked case: course marks move between grading schemes
A fictional programme moves pupils from one system to another. The source has a 20-point rubric score and the destination expects a percentage. A clerk multiplies every value by five, creating neat percentages. The destination then treats the result as directly comparable to a different examination.
The arithmetic mapping is correct for rescaling a 20-point maximum under the stated assumption; it does not prove the assessments measure the same content or difficulty. A transformed mark cannot be treated as an equivalent achievement claim without a defensible basis.
Document the original scale, assessment purpose and any valid conversion; preserve non-comparability when that is the truthful outcome. The Clementi Mathematics reference emphasises the same core habit: inspect what a symbol or operation represents before manipulating it.
CHAPTER 18 OF 23 · Work through cases
18. Worked case: a laboratory table changes a unit
A fictional environmental club records dissolved-substance concentrations in milligrams per litre. A student exports the values and relabels the column as grams per litre without converting the numbers. The graph remains smooth and the report looks professionally typeset.
The repair is to compare the original instrument output, unit metadata and calculations. If a reading is 250 mg/L, the corresponding value is 0.250 g/L, not 250 g/L. The factor of 1,000 illustrates why a syntactically plausible graph can carry a large scientific error.
Do not edit the final chart alone. Correct the source transformation, regenerate derived results and check whether every downstream decision changes. Preserve an audit note so future students can understand the correction.
CHAPTER 19 OF 23 · Work through cases
19. Worked case: a community survey misses some residents
A fictional town surveys households about library opening hours via an online link shared with current subscribers. The collected answers are accurate records of those respondents, but the sample excludes some people who do not use the digital service or already visit the library.
The team should distinguish respondent preference from community-wide demand. It might add assisted response channels or explicitly limit the inference. Adding more digital replies from the same population does not by itself address the gap.
Measure coverage and response patterns separately from file validity. The gold standard is not to make the charts look more complete; it is to understand whose voice the data support and whose perspective still needs to be sought.
CHAPTER 20 OF 23 · Choose a next action
20. Diagnose a dataset through a small purposeful audit
Pick one dataset that supports a real decision. State the receiving task and the most harmful plausible error. Identify critical fields, their sources, the intended reference period and which quality dimensions matter to that use.
Check eight things: expected records, unintended duplicates, contradictory values, update delay, rule validity, factual accuracy, source provenance and population coverage. Mark each supported, disputed or unknown; do not invent confidence where no test was performed.
Investigate the earliest consequential defect. If a phone number is missing because the form never asked for it, repair collection. If an existing value is delayed between systems, repair the handoff. If a correctly entered score is wrongly interpreted, repair the definition.
CHAPTER 21 OF 23 · Choose a next action
21. The Data Quality specialist reading library
Use these guides to follow the specific quality failure. Data quality is an operating discipline between data creation, interpretation, governance and downstream user action; it is not a substitute for the larger fields surrounding it.
For broad framing, read Data, Data Governance and Quality. To understand how a record moves, continue with Interoperability or Systems Integration. For claims made from the records, begin with Statistics and Evidence.
Definitions, records and governance
- Data
- Data Governance
- Information Systems
- Information Science
- Quality
- Documentation
- Database Administration
Transfer, evidence and verification
- Interoperability
- Systems Integration
- Coordination
- Process Mapping
- Workflow Design
- Verification
- Measurement
- Statistics
- Evidence
- Investigation
Protection and responsible use
- Information Security
- Personal Data Retention and Disposal
- Digital Preservation
- Management
- Strategy
- Decision Making
- Measurement and Performance Review
- Education Data Interoperability
- Education: Learning That Transfers into Life
- Secondary 1 Mathematics Clementi
Primary public reference guidance
UK Government Data Quality Framework · Practical guidance on action plans and root-cause analysis · Data Quality Action Plan implementation guide. These sources supply established dimensions and assessment approaches; all worked cases here are original educational illustrations.
CHAPTER 22 OF 23 · Choose a next action
22. Frequently asked questions about data quality
Are valid data accurate? Not necessarily. A number may meet its format and range rules yet describe the wrong entity, value or time.
Can a dataset be complete but unsuitable? Yes. It may be complete for a narrow user group while excluding a population needed for the decision, or it may accurately describe the wrong period.
Should all blanks become zero? No. A blank may indicate unknown, inapplicable, withheld or not observed; zero is a substantive value in many contexts.
Does removing duplicates improve every dataset? No. Repeated events or legitimately separate people can look alike. Uniqueness rules depend on the defined unit of record.
Who owns an error between systems? Agree on a source owner, receiver acceptance and escalation route. Never leave the affected user responsible for resolving a disagreement between organisations that have the authority to fix it.
How do students practise? Build a small, labelled table, preserve raw entries, test one rule for each meaningful quality dimension and ask a partner to use the results independently.
Choose a small but consequential record, such as a session booking, assessment rubric or scientific observation. Write its intended use in a single sentence. Define three field-level checks and one whole-task check that a receiver could perform.
Measure the baseline, document a defect and repair it as close as possible to its origin. Keep an example that should pass and a realistic example that should fail. After making the change, ask a second person to complete the original task, not just confirm that the spreadsheet opens.
Learning gains come from finding the precise unstable relationship and checking that it has been understood. The Clementi Secondary 1 Mathematics tutorial demonstrates that first-principles method in teaching. Data quality applies it to the information that other people must rely upon.