One-sentence answer: Data works when observations, events or states are recorded in a form that preserves identity, time, units, context, provenance, missingness and permissions well enough for later users to retrieve, transform and interpret them without silently inventing facts.
Data is often described as “the raw material” of modern systems. That metaphor is useful only up to a point. Data is not raw reality. It is a representation created by a measurement process, a sensor, a transaction, a human entry, a code system, an instrument, a model, a log or another recording mechanism.
A datum becomes trustworthy only when we can still answer: what does this value refer to, where did it come from, when was it true, what does missing mean, and what happened to it on the way here?
PHASE 4++++ · DEFINITIVE DATA MAP
How to Read This Data Guide
Data is not raw reality. It is recorded representation: a selected state, event or observation encoded into a form that can travel, be stored, transformed and reused. A definitive data system therefore preserves not only values, but also identity, time, units, missingness, provenance, transformations, permissions and the route back to the world that produced the record.
The invariant data chain
World state or event → observation or capture → entity identity → attribute and value → unit or code → event and observation time → context and metadata → storage → transformation and lineage → quality and missingness checks → access and permission → use in analysis, model or decision → correction propagated downstream.
Every transformation can add utility and also add assumptions. Cleaning, joining, aggregation, imputation, normalisation, anonymisation and feature construction should therefore remain recoverable rather than disappearing into a final table that looks self-explanatory.
What this article owns
This article owns the general mechanism by which recorded states remain interpretable, traceable and fit for later use as they move through systems.
Observation owns the encounter. Measurement owns formal comparison and uncertainty. Metadata owns structured description. Databases own storage constraints and transactions. Evidence owns the claim-support relationship. Information owns receiver interpretation. Interoperability owns cross-system meaning transfer. Decision-making owns action.
The lineage and permission test
A record can be technically valid and still be unusable because it refers to the wrong entity, is stale, was transformed under an unknown rule, carries ambiguous missingness, or is being reused outside the permissions attached to its collection.
The practical question is: Can this value still be traced to what it represented, how it changed, when it was true, and whether this receiver is permitted to use it for this purpose?
Four distinctions that prevent data overreach
- Data is not reality. A record represents selected features of the world through a capture process and can omit decisive context.
- Missing is not zero. Unknown, unavailable, withheld, not applicable and measured zero are different states with different downstream meaning.
- Database integrity is not factual truth. A system can preserve exactly what was written while the written value itself is wrong or stale.
- Derived data is not fresh observation. A model feature, normalised score, aggregate or imputed value inherits the assumptions and losses of its transformation path.
Clickable deep routes
- How Observation Works and How Measurement Works — how records begin and how values become comparable.
- How Metadata Works — identity, description, rights, versions and structured context.
- How Databases Work — constraints, transactions and reliable operational state.
- How Interoperability Works — how meaning survives transfer across system boundaries.
- How Evidence Works — when a dataset should change confidence in a claim.
- How Information Works — when recorded differences become useful updates for a receiver.
- How X Works Master Hub — return to the wider mechanism library.
A definitive data system lets us move from world to record and back again: who or what the value referred to, when it was valid, what transformed it, what was lost, who may use it, and which downstream outputs must change when the source is corrected.
Quick Read: the causal chain
WORLD STATE / EVENT → OBSERVATION OR CAPTURE → IDENTIFIER → VALUE → UNIT / CODE → TIME → CONTEXT → METADATA → STORAGE → TRANSFORMATION → LINEAGE → QUALITY CHECK → ACCESS / PERMISSION → USE → RECEIVER RESULT → CORRECTION
1. Data is a representation, not the world itself
A temperature reading is not the room. An exam score is not the student. A GPS coordinate is not the journey. A diagnostic code is not the person. A transaction record is not the whole economic relationship.
Data represents selected aspects of reality through a recording process. That selection creates both power and limitation. A dataset may preserve exactly what a task needs while omitting many other truths.
This is why Observation and Measurement sit upstream. Data cannot inherit more truth than the process that generated it.
2. A useful datum needs identity
A value without an entity is often meaningless. “72” could be a score, age, temperature, identifier, quantity or percentage.
A well-formed record usually needs to identify:
- what entity or event the value belongs to;
- which attribute is being recorded;
- what units or code system apply;
- when the value was valid or observed;
- where it was observed if location matters;
- which method or source generated it.
Identity errors are especially dangerous because they can produce records that look internally valid while referring to the wrong person, object, sample or event.
3. Time belongs inside the data model
Many values change. A train status, account balance, machine state, blood pressure, classroom attendance record or inventory level can be accurate at one moment and wrong later.
Useful data systems distinguish at least some of these times:
- when the real-world event happened;
- when it was observed;
- when it entered the system;
- when it was processed;
- when it became effective;
- when it expired or was superseded.
A stale record can be historically correct and operationally dangerous at the same time.
4. Missing is not the same as zero
One of the most common data errors is collapsing different states into the same value.
| State | Meaning |
|---|---|
| 0 | A measured or declared value of zero. |
| Unknown | The value exists but is not known. |
| Not measured | No observation was attempted. |
| Not applicable | The field does not apply to this case. |
| Withheld | The value exists but is intentionally not disclosed. |
| Missing due to failure | The expected capture process failed. |
Treating unknown as zero silently invents a fact. Treating withheld as missing can leak policy meaning. Treating “not applicable” as false can create invalid classifications.
5. Metadata preserves context
Metadata describes the dataset, field, record, method or object so later users know how to interpret it. NIST’s Research Data Framework treats metadata and provenance as central to effective use, reuse and preservation, and describes provenance as the historical, attributed record of where data came from, how it was generated or acquired and what alterations occurred.
Metadata may include:
- field definitions;
- units;
- method or instrument;
- creator or source;
- timestamps;
- version;
- licence or access restrictions;
- schema;
- quality flags;
- provenance links.
How Metadata Works owns that descriptive mechanism in more detail.
6. Provenance tells us how the data became what it is now
Data is frequently transformed: cleaned, merged, normalised, aggregated, filtered, imputed, classified, anonymised or converted between schemas.
Lineage connects the current value back to its sources and transformations:
SOURCE RECORD → TRANSFORMATION → INTERMEDIATE DATASET → JOIN / FILTER → DERIVED VALUE → OUTPUT
Without lineage, a derived number may be impossible to correct when an upstream record changes. With lineage, the system can answer: which outputs are affected by this correction?
7. Data quality is fitness for purpose
There is no universal single score called “data quality.” A dataset can be excellent for one purpose and inadequate for another.
| Quality dimension | Question |
|---|---|
| Accuracy | How closely does the record correspond to the target? |
| Completeness | Are required values/cases present? |
| Consistency | Do related records obey compatible rules? |
| Timeliness | Is the record current enough for this job? |
| Validity | Does the value conform to the declared format/domain? |
| Uniqueness | Are duplicates being mistaken for distinct events/entities? |
| Representativeness | Does the dataset cover the population or conditions needed? |
| Traceability | Can the result be linked back to source and transformations? |
NIST RDaF 2.0 explicitly frames quality in relation to planned use and includes accuracy, completeness, consistency and timeliness among example quality measures. That is a stronger model than treating “more data” as automatically “better data.”
8. Large volume cannot repair biased capture
Millions of records can reproduce the same systematic error millions of times. If a sensor misses a class of events, if a survey systematically excludes a group, if labels were generated by a biased process or if data comes only from successful users, volume does not create representativeness.
A useful hostile test is to ask: Who or what could not enter this dataset?
9. Derived data inherits assumptions
A raw sensor reading, cleaned value, aggregate statistic, model feature and classification label are different data objects. Derived data may be extremely useful, but each transformation introduces assumptions.
For example:
- an average hides distribution;
- an imputed value was not directly observed;
- a category boundary was chosen;
- a normalised score depends on a reference population;
- a model feature may combine several fields;
- a risk score is a model output, not an independent observation.
Data lineage should preserve these transformations so downstream systems do not mistake derived representation for fresh observation.
10. Data, information, evidence and knowledge are different jobs
| Data | Recorded representation of observations, events or states. |
|---|---|
| Information | Data interpreted in a context that makes a distinction meaningful. |
| Evidence | Information used to support, weaken or constrain a claim. |
| Knowledge | Usable understanding built from evidence, models, experience and correction. |
A dataset becomes evidence only relative to a claim. The same record can support one question, be irrelevant to another and be dangerously misleading for a third.
11. Databases store data; they do not define the truth of the data
Databases provide structures, constraints, transactions and queries for maintaining data state. A database can preserve a wrong value perfectly. Transactional consistency is not semantic truth.
The distinction matters:
- storage integrity: did the system preserve what was written?
- data validity: does the value satisfy schema/rule constraints?
- semantic correctness: does the value mean what users think it means?
- world correctness: does it correspond well enough to the real target?
12. Interoperability requires data meaning to survive the boundary
A CSV, JSON or database row can be transferred successfully while losing meaning. Units, identifiers, code systems, missingness and context must travel with the value—or be supplied by a shared contract.
This is the handoff to How Interoperability Works: cross-system exchange succeeds only when the receiver can correctly use what arrived.
13. Access and permission are properties of use, not afterthoughts
Data can be technically available and still not be legitimately usable for every purpose. Privacy, confidentiality, contractual restrictions, consent, licences, security classification and institutional authority constrain who may access or transform it.
NIST RDaF includes restrictions, usage agreements, levels of protection, security and privacy as lifecycle concerns. This is essential because data governance does not begin only after collection; collection itself creates obligations.
Can access ≠ may access ≠ may reuse for a different purpose.
14. Worked example: an assessment dataset
Imagine a dataset containing student ID, subject and score.
Before it can support a meaningful learning decision, we may also need:
- assessment identifier;
- date;
- maximum marks;
- topic coverage;
- timed/untimed condition;
- question-level responses;
- marking scheme version;
- whether assistance was used;
- missing-question meaning;
- provenance and correction history.
A score of 68 is data. “The learner is weak” is an inference. “The earliest weak link is sign control in algebra under time pressure” requires a much richer evidence chain.
15. Worked example: a sensor stream
A factory temperature sensor produces one reading per second. To use that stream for maintenance, the system needs more than values:
- sensor identity;
- physical location;
- measurement unit;
- calibration history;
- clock synchronisation;
- quality flags;
- dropout handling;
- maintenance/replacement history.
If the sensor was replaced and the identifier silently reused, a false trend may appear even though every stored number is syntactically valid.
16. Data across domains
| Domain | Typical data | Critical boundary |
|---|---|---|
| Science | Measurements, observations, experimental outputs | Method, uncertainty, provenance and reproducibility. |
| Education | Responses, scores, attendance, work samples | Data is evidence about learning, not identity. |
| Medicine | Clinical observations, tests, records | Clinical semantics, privacy and specialist interpretation. |
| Finance | Transactions, prices, balances, exposures | Time, identity, jurisdiction and reconciliation. |
| Transport | Positions, schedules, status, demand | Freshness and route semantics. |
| AI | Training, evaluation, tool and interaction data | Sampling, labels, leakage, rights and context shift. |
| Government | Administrative and statistical records | Definitions, population coverage, authority and purpose. |
17. Common data failures
| Failure | What goes wrong | Repair |
|---|---|---|
| Unknown→zero | Missingness becomes a fabricated value. | Preserve missing-state semantics. |
| Identity collision | Records attach to wrong entity. | Use governed identifiers and reconciliation. |
| Staleness | Old state is used as current state. | Carry observation/effective time and expiry. |
| Unit loss | Magnitude becomes ambiguous or wrong. | Carry explicit units and conversions. |
| Lineage break | Derived output cannot be traced or corrected. | Record transformations and versions. |
| Sampling bias | Dataset excludes important cases. | Inspect capture process and population coverage. |
| Label error | Target/category is wrong or inconsistent. | Audit definitions and adjudication. |
| Volume illusion | More records are mistaken for better evidence. | Evaluate fitness for the intended use. |
| Permission collapse | Availability is mistaken for legitimate reuse. | Preserve purpose, licence and authority constraints. |
18. Hostile test: reconstruct one output from the sources
- What real entity/event does this record represent?
- Where and when was it generated?
- What method or instrument produced it?
- What do units, codes and missing values mean?
- Which transformations were applied?
- Can the current value be traced to source records?
- Which corrections occurred?
- Which cases are absent from the dataset?
- Is it current enough for this decision?
- Is this use authorised?
- Can an independent checker recompute one downstream result?
19. Where Data fits in the wider How Things Work map
Data connects Observation, Measurement, Metadata, Databases, Interoperability, Evidence, Information and AI.
Its distinct public job is: How do recorded states remain interpretable, traceable and fit for later use as they move through systems?
20. What this article does not claim
- Data is not identical to reality.
- A large dataset is not automatically representative or high quality.
- A database preserving a record does not prove the record is true.
- A derived value is not automatically an independent observation.
- Missing does not mean zero, false or absent unless defined that way.
- Available data is not automatically authorised for every use.
- Data does not become evidence until it is connected to a specific claim and reasoning task.
21. Observable mastery test
You understand data when you can take an unfamiliar record and identify its entity, attribute, units/codes, time, source, collection method, missingness semantics, provenance, transformations, quality limits, permissions and intended use—and trace a downstream conclusion back to the source records that justify it.
Authoritative source corridor
- NIST Research Data Framework (RDaF) — current public lifecycle framework for research data.
- NIST SP 1500-18r2: RDaF Version 2.0 — provenance, quality, metadata, FAIR principles, security/privacy and lifecycle practices.
- NIST: RDaF 2.0 release — Version 2.0 scope and lifecycle architecture.
- How Metadata Works and How Databases Work — neighbouring public owners.
Governing idea: Data becomes durable knowledge infrastructure only when the path from world to record to transformation to receiver remains visible enough to correct.