Did you know? A bank-like payment can be encrypted correctly, authenticated and replicated to backups yet still be wrong because one service treats cents as dollars. Protecting information means safeguarding interpretation and authorised action, not only bytes.
The Gold Standard of Information Integrity Engineering follows facts from original source and identity through validation, transactions, provenance, correction and recovery. Its method distinguishes what each safeguard can prove and what remains unverified.
This forty-chapter guide includes fictional examples of unit scaling, duplicate event delivery, ledger reconciliation, migrations and exception rates. Each numerical case is for learning rather than certification of a real system.
Chapter index
Sources, identity and safe transactions · Chapters 1–20
- Information integrity is a promise about meaning and legitimate changes
- Integrity, accuracy, authenticity and availability differ
- Begin with critical business invariants
- Give identity a stable and documented namespace
- Validation should check syntax and legitimate meaning
- Units, currencies and scales are part of the record
- Worked example: cents accidentally treated as dollars
- Atomic transactions prevent partial operations
- Concurrent users can silently overwrite work
- Idempotency prevents repeated deliveries becoming repeated effects
- Worked example: duplicate event deliveries change reporting counts
- Sequence and ordering can change legitimate state
- Multiple clocks create false timelines
- Provenance connects every important derived result
- Define authority for each type of fact
- Byte integrity is not factual truth
- Signatures connect credentials to particular content
- Audit logs need actual transaction context
- Correction history should coexist with lawful amendment
- Version interfaces and reference tables through change
Reconciliation, recovery, software and audit · Chapters 21–40
- Reconciliation is more than matching an attractive total
- Worked example: balanced totals can conceal wrong assignments
- Protect referential relationships during migration
- Worked example: a near-perfect transfer can still be untrustworthy
- Backups may preserve an error rather than correct it
- Semantic validation belongs at receiving interfaces
- Control retry and replay across system restarts
- Source corrections must reach dependent reports
- Measure coverage of checks, not only passing outcomes
- Worked example: exception counts need the denominator
- A failed check is a starting point for investigation
- Secure development protects system and information integrity
- AI-generated outputs should remain labelled as derived
- Keep privacy and retention rules inside the integrity design
- Supplier integrations carry a shared evidence boundary
- Monitoring must show when its own sensors stop working
- Changes should renew affected evidence, not erase history
- Independent review should reproduce one complete decision
- Frequently asked questions about integrity engineering
- Continue through the connected Gold Standard library
Quick information: what each safeguard actually shows
| Test | Claim supported |
|---|---|
| Hash | Bytes match a trusted earlier reference |
| Signature | A credential signed specific content |
| Validation | Declared structural and domain rules passed |
| Reconciliation | Independent business expectations agree |
| Provenance | Source and transformations are traceable |
1. Information integrity is a promise about meaning and legitimate changes
Information integrity engineering designs verifiable paths through which real events become records and records become decisions. It protects against incorrect identity, unauthorised alteration, semantic mismatches, missing relationships and errors introduced during transfer or processing. Its goal is not merely keeping a stored file unchanged.
Did you know that an encrypted payment can arrive without a single corrupted bit and still be wrong? A system deserves confidence when a competent reviewer can reconstruct the sources, checks, versions, authorisations and limitations of its result.
2. Integrity, accuracy, authenticity and availability differ
Integrity concerns protection against improper modification; accuracy concerns correspondence with the real-world fact; authenticity concerns genuine origin; and availability concerns timely authorised access. These properties interact but cannot be reduced to one dashboard percentage.
An invoice signed by its apparent originator may still contain a mistake. A database can be continuously available while replicating that mistake to every backup. Design tests that clearly identify which property has and has not been demonstrated.
3. Begin with critical business invariants
An invariant is a legitimate condition a system should preserve, such as each payment referring to the correct approved invoice, a clinical specimen remaining linked to its subject, or an inventory movement reconciling with authorised stock changes.
Domain owners and engineers must define those conditions together. A programmer who guesses rules from a column name can faithfully implement the wrong business process. The strongest integrity check begins with correctly understood purpose and responsibilities.
4. Give identity a stable and documented namespace
Names, local row numbers, filenames and email addresses can change or be duplicated across systems. Records need stable identifiers with an understood scope, including what happens when an entity is merged, corrected or no longer active.
Never recycle a permanent identifier for a different customer merely to save storage. In a migration, check that links to cases, measurements and transactions still refer to the same real-world entities after transfer.
5. Validation should check syntax and legitimate meaning
Input validation can enforce permitted values, required fields, referential links and appropriate domain rules. It should reject obvious errors while allowing documented exceptions such as an unknown value that really is not available.
NIST SP 800-53 Revision 5 includes System and Information Integrity controls. Those controls must be implemented according to the application’s risks: a correctly formatted telephone number is not proof it belongs to the named person.
6. Units, currencies and scales are part of the record
An integer might represent dollars, cents, millimetres, metres or a timestamp in seconds. Numeric types alone cannot prevent the receiver from applying a different unit or scale to the same bits.
Use explicit currency, unit, precision, time basis and versioned interface rules. A result with many displayed decimal places can be wrong or unsupported by the original measurement. Test conversions with known examples.
7. Worked example: cents accidentally treated as dollars
Suppose a fictional payment carries an integer amount of 129900 in cents. Correct interpretation is 129900 ÷ 100 = S$1,299.00. If a receiving service incorrectly assumes it was denominated in whole dollars, it displays S$129,900.00.
The difference is one hundredfold, although the message may have been transmitted exactly. This is a semantic integrity defect. End-to-end tests and explicit minor-unit conventions can detect the mismatch before it enters financial decisions.
8. Atomic transactions prevent partial operations
Some actions need several dependent records to change together or not at all. A bank-like transfer, for example, might be invalid if one balance changes while the intended counterpart never records the transaction.
Use database transactions and appropriate recovery behaviour according to real domain rules. Atomicity does not independently prove a transfer was authorised or honest. The whole business state must satisfy its stated requirements.
9. Concurrent users can silently overwrite work
Two legitimate users may read the same state, make separate changes and save them in an unfortunate order. Without appropriate concurrency control, the second update may erase the first while both interfaces indicate success.
Test actual parallel workloads, version checks and retry paths. A single-user demo is weak evidence for a system that processes thousands of events simultaneously. A conflict should have an accountable resolution rather than a hidden last-write-wins rule.
10. Idempotency prevents repeated deliveries becoming repeated effects
Networks retry and people resubmit requests when confirmation is delayed. An idempotent business operation recognises repeated processing of the same identified request so that its intended effect is not applied again.
Use request identities with a justified scope and lifespan. Two real purchases for the same amount are not duplicates merely because their amounts match. Preserve evidence of delivery attempts while enforcing the correct business outcome.
11. Worked example: duplicate event deliveries change reporting counts
A fictional service receives 1,250 event rows. Seventy are documented retries of already identified events, and ten are outside the agreed report window; assume those categories do not overlap. In-scope unique events therefore number 1,250 − 70 − 10 = 1,170.
That figure describes events, not distinct customers or approved transactions. Its validity depends on how unique identity and timing were defined. The original rows should remain traceable for any independent audit.
12. Sequence and ordering can change legitimate state
An approval, cancellation and reinstatement may each be valid events with different effective times. A delayed older event can restore an obsolete state if the receiving system applies records in arrival order without checking their business sequence.
Record the source version, event identity and relevant ordering rules. A later ingestion timestamp does not mean the underlying event occurred later. Replays and late messages need designed handling.
13. Multiple clocks create false timelines
Different services may store local clock times, UTC timestamps, device observations or upload times. Clock drift and delayed collection can create an incorrect sequence even when all stored records are byte-for-byte genuine.
Keep original time representation, source and uncertainty. A timeline should separate event, recording, ingestion and later correction. For consequential use, uncertain time offsets are part of the evidence, not an inconvenience to hide.
14. Provenance connects every important derived result
A public dashboard may depend on several source tables, a cleaning step, a join, a correction and a report generator. If any stage changes, the output’s meaning or value can change without the original source changing.
The W3C PROV-O recommendation defines entities, activities and agents for representing lineage. Keep source versions and transformation methods so others can reproduce a consequential result, not merely view the same finished chart.
15. Define authority for each type of fact
A customer master may be authoritative for contact details while an accounts system controls approved payment status. A cached report should not gain unilateral authority to rewrite either source.
Assign ownership at the field or entity level and provide an explicit reconciliation process when sources disagree. An organisation-wide slogan about one source of truth is less useful than a verified map of actual authority and correction.
16. Byte integrity is not factual truth
A cryptographic hash comparison can detect changes to a defined byte sequence relative to a trusted reference. It cannot determine whether the original measurement was correct, whether a statement was honest or whether a particular change was authorised.
Store expected digests securely with source identity, algorithm and capture date. A mismatch may result from an approved transformation or real tampering; each possibility needs evidence-based investigation.
17. Signatures connect credentials to particular content
A digital signature can help bind defined bytes to a cryptographic credential and show whether the signed object changed under the verification model. Its meaning depends on who controlled the key and what the signer was authorised to approve.
A signed false invoice remains false. A technically valid signature does not establish that all figures were independently reviewed. Identity assurance and business authorisation are separate checks.
18. Audit logs need actual transaction context
An audit trail should describe actor or service identity, relevant time, affected object, change, result and software version. A generic success entry may not show whether the right recipient received the right amount.
Protect logs while minimising unrelated private information. Shared administrator accounts weaken attribution, and a system that logs only successful operations can hide the failed attempts necessary to interpret an incident.
19. Correction history should coexist with lawful amendment
Records contain mistakes, and authorised owners may need to correct them. An append-only history can show what was previously believed while a current view presents the corrected statement with suitable qualification.
Do not use immutability to justify displaying known false facts. The correction process should document authority, reason, source evidence and downstream systems that need an updated view.
20. Version interfaces and reference tables through change
Schemas, category codes, exchange-rate tables and authorised conversion rules can change over time. A value valid under one version may be misinterpreted by a consumer running another version.
Preserve effective dates and compatible interfaces. Test old and new records together, and compare end-to-end business outcomes after deployment. A green network response does not demonstrate that source and destination agree on meaning.
21. Reconciliation is more than matching an attractive total
Reconciliation checks whether independently derived expectations and recorded events agree under the business’s rules. A bank statement, approved invoice list and internal ledger can help challenge one another when they do not all originate from the same compromised source.
Comparing two dashboards built from a single corrupted export provides little independent assurance. Preserve discrepancies with identifiers and investigate them instead of silently dropping unmatched rows to force the totals to balance.
22. Worked example: balanced totals can conceal wrong assignments
Suppose a fictional organisation expects S$62,000 in approved invoices, with S$60,500 posted payments and S$1,500 legitimately pending. The arithmetic reconciles: 60,500 + 1,500 = 62,000. That is useful but narrower than demonstrating that every invoice reached the correct supplier.
Two payments credited to the wrong counterparties can cancel in the total. A proper check also compares invoice identity, recipient, status and currency. The calculation is educational, not an accounting assurance opinion.
23. Protect referential relationships during migration
A data transfer can copy all customer records but lose the relationship between those people and their orders. It might preserve an amount yet attach it to the wrong document after a mistaken join.
Test key uniqueness, linked records, code meanings and essential user workflows as well as file counts. A data-copy success response is not evidence that the reconstructed service preserves every important relationship.
24. Worked example: a near-perfect transfer can still be untrustworthy
A fictional migration expects 50,000 records and imports 49,940, with 60 exceptions. Forty-five exceptions are intentionally quarantined with verified identity; fifteen have no confirmed destination or disposition. Imported coverage is 49,940 ÷ 50,000 = 99.88%.
That percentage may sound excellent while fifteen consequential records remain unexplained. Identify affected people or transactions through authorised methods and reconcile the exceptions before declaring the specific business process ready.
25. Backups may preserve an error rather than correct it
Regular backups help recover data after loss, but can capture corruption or an incorrect state. A clean file restoration does not establish that customers, permissions and financial relationships are accurate after the return to service.
Test meaningful tasks and compare recovered states with independent source evidence. The organisation should understand its recovery point and recovery time objectives without pretending a successful backup icon is a guarantee of usable information.
26. Semantic validation belongs at receiving interfaces
Source and destination systems can exchange a valid message but apply different field meanings, such as a price including tax versus one excluding it. Each side may pass its local schema tests while delivering a wrong commercial result.
Use contract tests with representative approved examples and explicit units, effective dates and permitted states. Verify the final user’s outcome, not only that the data moved without a network error.
27. Control retry and replay across system restarts
After an outage, an event processor may replay a queue or rebuild derived state from prior events. Replaying should lead to the intended result without duplicating approved financial effects or overlooking a past cancellation.
Keep suitable request identities, sequence rules and tested restoration procedures. The correct behaviour depends on the business domain; a universal ‘process once’ slogan cannot replace an actual idempotency and reconciliation design.
28. Source corrections must reach dependent reports
A legally authorised correction to an original record may need to propagate to reports, search indexes, caches and analytical copies. If a downstream product continues to present the old value as current, users may make inconsistent decisions.
Document dependencies and versioning. Preserve the earlier state where legitimate audit needs require it, but label it as superseded. An integrity programme includes controlled recovery from known mistakes.
29. Measure coverage of checks, not only passing outcomes
A quality check that examines only one department may show 100% success while an unmonitored branch is missing. The population, affected data and detection capability belong in any summary.
Report exceptions by severity and user consequence. A small count can represent low risk or merely a broken detector, so coverage evidence is important alongside raw rates.
30. Worked example: exception counts need the denominator
A fictional service records 90 validation exceptions in 30,000 transactions, an observed rate of 0.30%. In a later period it records 60 exceptions in 10,000 transactions, an observed rate of 0.60%.
The count fell while the rate doubled. Before claiming deterioration, check whether transaction populations, monitored fields and detection rules were consistent. These are fictional numbers, not measurements of a real organisation.
31. A failed check is a starting point for investigation
A mismatch might come from authorised correction, format conversion, delayed updates, failed ingestion, malicious alteration or a mistaken reference table. The relevant response depends on which meaning and controls are involved.
Preserve original evidence and record the explanation. Do not automatically punish a user or destroy the source when a hash, total or format validator fails. A reliable programme uses judgement and an authorised correction route.
32. Secure development protects system and information integrity
Software build processes, dependencies and configuration can introduce vulnerabilities or defects. NIST SP 800-218, Secure Software Development Framework 1.1, recommends organised practices for reducing vulnerability risks during development.
The NIST project publication list still identifies proposed version 1.2 as a draft. A security scanner or framework checklist alone cannot prove the system implements correct business arithmetic or authorisations.
33. AI-generated outputs should remain labelled as derived
A classifier or generative system can create summaries, inferred categories or extracted names. These may be useful when their uncertainty and source relationship are understood, but can also invent values that sound credible.
Distinguish original observation from model output and establish human review rules for consequential uses. A generated answer should not silently replace the verified source record or become the definitive record of a person’s identity.
34. Keep privacy and retention rules inside the integrity design
Reproducibility may require retaining important historical decisions, but collecting every private field indefinitely creates new legal and security risks. Integrity evidence should be designed to serve legitimate purpose and lawful access.
Work with relevant privacy and records specialists and use proportionate retention. A system can demonstrate that an action occurred without copying all sensitive payloads into an unrestricted logging service.
35. Supplier integrations carry a shared evidence boundary
External processors may transform, store or transmit critical records. A supplier’s uptime or security certificate does not automatically certify the customer’s business rules, field meaning or data reconciliation.
Specify contractual and technical ownership of interface versions, exception reporting, corrections, access and exit. Test a full legitimate transaction across the provider boundary rather than relying on the most attractive supplier assurance summary.
36. Monitoring must show when its own sensors stop working
An integrity monitor can fail, lose access or be misconfigured. An apparent decline in bad records may mean the process improved or that the quality checks silently stopped observing part of the workflow.
Track expected event volume, rule coverage and monitor health as well as defects. Anomaly results are clues to investigate, not proof of wrongdoing or causal blame.
37. Changes should renew affected evidence, not erase history
A schema revision, model deployment, new tax table or supplier change can alter expected output. Previous successful tests may not cover the new configuration.
Use controlled review and known-value regression tests with versioned source material. Preserve the older assumptions so that historic decisions can still be reconstructed without pretending their context was identical to today’s.
38. Independent review should reproduce one complete decision
Select a consequential reported number and reconstruct source identity, original units, time, validation, permissions, transformations, corrections and reconciliations. A second competent person should be able to follow the trail with the authorised records available.
Challenge one awkward case, such as a duplicate retry or unknown migration identifier. A gold-standard review explains that exception clearly rather than hiding it inside a large success percentage.
39. Frequently asked questions about integrity engineering
Does a matching hash prove the content is true? No. Can a secure system contain wrong facts? Yes. Does a digital signature prove a payment is legitimate? No. Can totals reconcile when individual payees are wrong? Yes. Can retry delivery create duplicate business effects? Yes if controls are inadequate.
Does a successful backup automatically restore a correct business service? No. Are AI summaries original evidence? No. Is a high imported-record percentage proof all important relationships survived? No. Integrity is a layered claim.
40. Continue through the connected Gold Standard library
Study Information Quality, Information Provenance, Information Assurance, Information Forensics, Information Lifecycle Engineering and Interoperability Engineering.
Primary references: NIST SP 800-53, NIST secure software development and W3C PROV-O. All numerical exercises are fictional educational examples; none proves any real service meets an operational or legal standard.