THE GOLD STANDARD OF INFORMATION · SPECIALIST STUDY
Tell the truth about when an event happened—and when the system learned it
Streaming systems observe the world out of order. A payment can occur before a network outage yet arrive after later transactions. A sensor can buffer readings, a school register can be corrected, and a mobile device can reconnect hours later. Event-time semantics preserve when the event belongs; processing time records when computation saw it.
Watermarks and lateness policies turn that distinction into operational decisions. They estimate progress, decide when a window may emit, and govern whether late records correct, reopen or miss a result. The gold standard makes those policies visible, measures their error and preserves every correction trail.
Start with the closest route. This guide belongs to the Information subject guide, the Gold Standard of Education, and the How X Works learning library.
Choose a reading route
Understand time · chapters 1–3
Window the stream · chapters 4–5
Handle lateness · chapters 6–8
Test correctness · chapters 9–10
Transfer and govern · chapters 11–12
The whole problem at a glance
| Object | Meaning | Audit question |
|---|---|---|
| Event time | When the represented event occurred | Who assigned the timestamp and in which clock? |
| Processing time | When a stage handled the record | Can retries or backlog change the result? |
| Watermark | Estimate of event-time completeness | What evidence supports its progress? |
| Allowed lateness | How long corrections remain accepted | What is discarded, side-routed or restated? |
A watermark is an estimate, not proof that nothing older will arrive. Every policy trades latency, completeness, cost and stability.
CHAPTER 1 OF 12 · Understand time
1. Separate event time from processing time
Event time belongs to the event: the moment a purchase, observation or attendance action occurred. Processing time belongs to the computer stage that handled it. They may coincide in a quiet system and diverge dramatically under buffering, retry or outage.
Processing-time windows answer what arrived during this computation interval. Event-time windows answer what happened during the represented interval. The first is operationally simple; the second supports historically meaningful totals when arrival order is unstable.
Record timestamp provenance. Device time, server receipt time, a human-entered date and a derived business date are different claims. A field called timestamp without its clock, timezone and authority invites silent error.
Keep both times when useful. Their difference is observable delay and can reveal congestion, offline devices or correction workflows. Do not overwrite event time merely to make records appear ordered.
CHAPTER 2 OF 12 · Understand time
2. Define clocks, zones and timestamp quality
Use an unambiguous instant for machine comparison and retain the local civil context needed by readers. Daylight-saving changes, leap seconds, clock skew and timezone conversion can place records in the wrong reporting period.
Assign a timestamp-quality class: source clock synchronised, server-stamped, user-entered, estimated or recovered. A precise-looking value may still be weak evidence. Quality should influence reconciliation and confidence.
For multi-device processes, define which event marks completion. Order placed, authorised, packed and delivered are all legitimate times for different measures. Name the business event instead of using one generic created-at field.
Test boundaries: midnight, month-end, daylight transitions and clock correction. A pipeline that works on ordinary hours can still fail at the moments reports and obligations change.
CHAPTER 3 OF 12 · Understand time
3. Model ordering without pretending the stream is sorted
Distributed sources provide partial order. Sequence numbers may order events within one device or account but not across the whole system. Network arrival order is not a reliable history.
Use stable event identifiers and idempotent processing so retries do not create extra facts. Where an update supersedes an earlier record, carry version or revision semantics rather than trusting last arrival.
Define whether the domain needs total order, per-key order or only commutative aggregation. Stronger ordering costs latency and coordination. Do not purchase it when the calculation is order-independent.
When conflicting versions arrive, preserve provenance and apply a declared resolution rule. Last-write-wins is only meaningful when the chosen time and clock deserve authority.
CHAPTER 4 OF 12 · Window the stream
4. Choose windows that match the question
Fixed windows divide time into equal intervals; sliding windows overlap; sessions group activity separated by gaps. The correct choice follows the decision: daily totals, rolling load or user sessions answer different questions.
Window boundaries are policy. A day in Singapore local time is not the same as a UTC day. A seven-day sliding result can count one event in several outputs by design. Explain this to downstream readers.
Keying precedes many windowed calculations. If identity changes or duplicates split across keys, correct timing cannot repair the total. Validate event identity alongside window logic.
Avoid interpreting a partial window as final. Label early results and expose the completeness basis. A fast dashboard should tell readers whether it is provisional.
CHAPTER 5 OF 12 · Window the stream
5. Understand watermarks as progress estimates
A watermark estimates a lower bound on future event timestamps at a point in the pipeline. When it passes a window end, the system may treat that window as ready for an on-time result.
The estimate depends on source behaviour, partitions and idle detection. One stalled partition may hold back progress; an over-aggressive policy may advance while older records remain possible.
Watermarks can differ across stages as joins, buffers and slow transforms alter progress. Observe them per source and operator instead of assuming one global truth.
Publish watermark lag and late-arrival distributions. An unexplained watermark is hidden policy; an observed watermark can be tuned against actual completeness and delay.
CHAPTER 6 OF 12 · Handle lateness
6. Choose early, on-time and late firings
Early firings provide provisional results before the watermark reaches the window end. On-time firing reflects the watermark crossing. Late firings incorporate records arriving after that point.
Decide whether outputs accumulate, retract or replace. A consumer must know whether a new value is a delta, a full restatement or a correction. Otherwise repeated panes can be double-counted.
Early results improve responsiveness but increase revision. Use them where action benefits from speed and can tolerate change. Do not present them with the visual authority of final audited totals.
Name each pane with window, timing class and revision. Stable identifiers let databases and dashboards update the intended result instead of appending confusion.
CHAPTER 7 OF 12 · Handle lateness
7. Set allowed lateness from evidence
Allowed lateness defines how long state remains open for delayed records. It should follow measured arrival distributions, reporting obligations, correction value and storage cost—not a convenient round number alone.
Examine percentiles and tails by source. A mobile source may be late for days while a server event is late for seconds. One global threshold can waste state or discard meaningful corrections.
State what happens after the threshold: drop, dead-letter, audit queue, manual restatement or next-period adjustment. Silent disposal makes the published total look more certain than the pipeline was.
Review the threshold when source behaviour changes. New integrations, outages and policy shifts can move the delay distribution without changing code.
CHAPTER 8 OF 12 · Handle lateness
8. Work a delayed-attendance example
A school publishes a morning attendance count at 9:00. Most teachers submit by 8:15, one classroom device syncs at 9:20, and a medical absence is authorised at noon. Event time places attendance in the morning session; processing time records later knowledge.
An early pane at 8:30 supports staffing, the on-time pane at 9:00 is labelled provisional, and late firings revise the count through a stated cutoff. Each revision keeps the previous value and reason.
The medical authorisation changes classification, not presence. The model must distinguish late arrival of an event from late correction of meaning. Revision identifiers and provenance prevent the two from collapsing.
Safeguarding decisions should not rely on a raw provisional total alone. The changed context requires named exceptions, human review and a route for corrections after the technical lateness window closes.
CHAPTER 9 OF 12 · Test correctness
9. Make aggregations correction-safe
Sum and count can be corrected by retraction or replacement; distinct counts, percentiles and external effects are harder. Choose an output contract before choosing a trigger.
Idempotent sinks use stable keys and versions. Exactly-once claims should be scoped: source read, state update and external side effect may have different guarantees. Test failure between each boundary.
Keep the raw immutable event where lawful, plus derived revisions. Recomputing from source is the strongest correction path when retained data and code versions are available.
For non-retractable actions such as messages or payments, separate provisional computation from commitment. A late record may update analysis without reversing an already executed real-world act.
CHAPTER 10 OF 12 · Test correctness
10. Test with disorder, delay and replay
Generate events in event-time order, shuffle arrival order, duplicate some, delay some beyond the watermark and inject clock errors. Assert both intermediate panes and final results.
Replay the same event set under different worker timing. Event-time results should remain stable where the contract promises determinism. Processing-time-dependent results should be labelled accordingly.
Test idle partitions, source restarts, backfills and schema revisions. Watermark logic often fails at operational edges rather than ordinary traffic.
Compare streaming output with a bounded batch recomputation after lateness closes. Differences require explanation: loss, duplicate, divergent logic or an intentional processing-time rule.
CHAPTER 11 OF 12 · Transfer and govern
11. Transfer the method to environmental sensors
A river sensor records readings locally during a network outage and uploads them later. Event time restores the hydrograph at the measured moments; processing time shows when monitoring staff learned of them.
An alert cannot travel backward in time. Late data may correct the scientific record without claiming the alert system could have acted earlier. Store both the reconstructed condition and the knowledge timeline.
Sensor clock drift and calibration changes add uncertainty beyond arrival delay. Quality flags, device identity and calibration version belong beside timestamps.
The transfer check shows the limit: event-time correctness repairs historical analysis, not missed real-time intervention. Governance must review both data and operational response.
CHAPTER 12 OF 12 · Transfer and govern
12. Publish the temporal contract
Document timestamp source, timezone, window type, boundary, watermark method, triggers, accumulation mode, allowed lateness, after-cutoff handling and sink update semantics.
Expose provisional versus final status in APIs and interfaces. Consumers should not infer finality from a single number. Include revision time and reason.
Monitor watermark lag, late fraction, discarded fraction, revision magnitude, state cost and replay discrepancy. Averages alone can hide one source that is systematically late.
The final audit asks whether a reader can reconstruct what happened, when the system knew it and how later evidence changed the published result.
Practice and evidence check
- Label five timestamps as event time, processing time or derived business time.
- Choose a window and explain its boundary to a non-technical reader.
- Plot arrival delay by source before choosing allowed lateness.
- Specify whether a late pane replaces, retracts or accumulates.
- Compare a stream result with batch replay after the window closes.
A temporal design is complete only when provisional results, revisions and discarded records are visible and testable.
Event-time pipeline audit
What clock created the event timestamp?
Record device, server, human or derived authority, timezone, synchronisation and quality. A millisecond value is not automatically trustworthy.
Can one slow partition hold the watermark?
Inspect progress per partition and idle-source policy. Advancing too slowly delays every result; advancing too quickly manufactures lateness.
What becomes of post-cutoff data?
Name the dead-letter, correction, restatement or disposal path and its owner. Silent drops are not a governance policy.
Can consumers apply revisions exactly once?
Use stable window keys, pane versions and explicit replacement or delta semantics. Test retries at the sink boundary.
Are early results visibly provisional?
Interfaces, exports and downstream tables should carry timing status and revision time, not rely on explanatory documentation elsewhere.
Can the stream be reconstructed?
Retain lawful raw events, code version and configuration long enough to reproduce the result and investigate discrepancies.
Does lateness differ by source?
Measure tails separately. One threshold may systematically exclude offline, rural or manual sources.
What action cannot be reversed?
Separate analytic correction from irreversible notifications, payments or safety actions. Late truth does not undo earlier operational limits.
Frequently asked questions
Is a watermark the latest event time seen?
No. It is an estimate of progress and expected completeness, not a maximum timestamp or proof.
Is late data wrong data?
No. It may be valid evidence that arrived after the system’s timing estimate.
Should all late data reopen results?
No. The value of correction must be balanced with state, cost, stability and obligations.
Can event time recreate a missed alert?
It can repair history, not the action opportunity that already passed.
What makes a result final?
A declared policy, not time alone: watermark, lateness horizon, correction path and consumer contract together define finality.
Continue through the ecosystem
The Gold Standard of Information — the subject root for meaning, provenance and verification
Schema Evolution, Backward Compatibility and Semantic Drift — evolve temporal records without changing their meaning
Checksums, Record Identity and Detecting Silent Change — verify record identity through correction and replay
Sources and further reading
- Apache Beam Programming Guide — windowing, watermarks, triggers and late data
- Apache Beam model basics — event time and processing time
- Apache Beam glossary — watermark and late-data definitions
- Apache Beam Managed I/O — allowed lateness and dead-letter handling
Use sources to inspect definitions and limits, not as decoration. Apply methods in ways that fit the actual decision, data and people affected.
Operational scenarios and failure checks
Monthly billing correction
A utility estimates a bill before all meter events arrive. The first pane is explicitly provisional; an on-time pane uses the watermark; late readings create a numbered restatement. The customer sees the old amount, corrected amount, reading provenance and adjustment date. The design separates computation from payment collection so a late reading does not silently create an unexplained debit.
Mobile offline learning records
A learning app stores completed exercises while offline. Event time is the learner’s completion time, server receipt may be hours later, and device clock quality is recorded. Weekly progress accepts late events for a defined horizon, but badges already awarded are not repeatedly revoked. Reconciliation handles duplicate uploads by stable exercise-attempt ID rather than arrival order.
Fraud-monitoring latency
A transaction event arrives promptly but a chargeback label appears weeks later. The event-time stream can score the transaction at purchase and later update model-evaluation data when the outcome is known. The label is not treated as if available at decision time. This prevents future information leakage from making the historical detector look unrealistically accurate.
Backfill after a source outage
A source resumes and sends three days of records. The pipeline marks the batch as backfill, assigns original event times, and routes it through late-data logic. Dashboards restate affected windows rather than adding the records to today. Operators compare totals with source control counts and publish the revision, because the shape of history has changed.
Join between fast and slow streams
Orders arrive quickly while shipment confirmations lag. A join watermark can be held by the slower side, delaying output, or emit unmatched provisional records with later correction. The chosen rule depends on whether users need immediate exception visibility or complete fulfilment metrics. Timeout, late match and duplicate handling must share stable order identity.
Idle partition problem
One partition has no traffic. If the system treats silence as delay, its watermark can hold every window open; if it declares idleness too soon, a returning old event becomes late. Configure and observe idle detection per source. Test restart behaviour so operational quiet is not confused with event-time completeness.
Clock-skewed sensor
A sensor clock drifts twelve minutes ahead. Records can appear to come from the future, advance windows incorrectly or miss joins. Timestamp quality monitoring compares device and receipt times, quarantines implausible skew and preserves the original value. Correction does not erase evidence of the faulty clock, because later analysis may need to identify affected periods.
Data deletion and open state
Long allowed lateness keeps state and perhaps personal records longer. Retention should be designed with privacy obligations, not assumed from window configuration. Where raw data must expire, store the minimum aggregate state and make post-expiry correction limits explicit. Completeness is not an unlimited licence to retain source records.
Downstream cache behaviour
A pipeline emits a corrected pane, but a cache keyed only by window date keeps the earlier response. End-to-end correctness requires revision-aware invalidation, not just correct stream computation. Test APIs, warehouses, extracts and notifications after correction. A late record that never reaches the reader has not repaired the information system.
Finality communication
Different consumers need different finality. Operations may act on a five-minute provisional view; finance may wait three days; audit may accept formal restatement after month close. Provide one temporal contract with named service levels rather than a universal final flag. Each consumer should know which revisions it must accept and for how long.
Extended practice laboratory: reconstructing an event-time result
A useful streaming exercise starts with a result that is plausible but unstable. Imagine a council counting bicycle entries into a park in ten-minute windows. Gate A sends events promptly. Gate B stores readings during a mobile-network outage and uploads them later. The dashboard refreshes at 09:20 and reports 118 entries for the 09:00–09:10 window. At 09:31, thirty-two Gate B events arrive with event timestamps inside that old window. The practical question is not simply whether to add them. It is which result was shown, what the system knew when it showed it, and how later evidence changes the claim.
Step 1: name every clock
Write three times beside each record: event time, ingestion time and processing time. Event time is when the bicycle crossed the gate. Ingestion time is when the platform accepted the record. Processing time is when the particular job evaluated it. These times can coincide, but they answer different questions. A report about use of the park belongs to event time. A report about platform workload may belong to ingestion or processing time. If a query groups by the easiest timestamp rather than the timestamp implied by the question, it can be fast, reproducible and wrong.
For the delayed Gate B batch, event times range from 09:03 to 09:09, ingestion time is 09:31, and processing time is shortly after. Grouping by ingestion time moves genuine 09:00–09:10 activity into the 09:30–09:40 window. That does not merely delay the evidence; it changes the history being described.
Step 2: turn the watermark into an explicit claim
Suppose the job’s watermark reaches 09:10 at processing time 09:20. A careful statement is: “The system estimates that most events with event times before 09:10 have arrived.” It is not: “No earlier event can arrive.” The watermark is a progress estimate used to decide when to emit a result. Its meaning depends on the source, partitioning, estimator and observed delay pattern. When one partition stalls, a combined watermark may also stall; when an estimator is aggressive, it may advance before a delayed source catches up.
Record the estimator version with the result. If the estimator is later tuned from a five-minute to a fifteen-minute delay allowance, the same raw events can produce different first emissions. Without the policy version, an auditor may see two totals and be unable to tell whether the data changed, the code changed or the lateness policy changed.
Step 3: calculate the result states
| State | Observed total | What may be said | What remains open |
|---|---|---|---|
| Initial emission at 09:20 | 118 | Provisional count under the current watermark policy | Late Gate B records may revise the window |
| Correction at 09:31 | 150 | Revised count after thirty-two late events | Further arrivals may still occur before finalisation |
| Finalised after the lateness horizon | 150, if unchanged | Closed operational result under the stated policy | Backfills or source repairs require a new version |
The table separates a value from its status. “118” and “150” are not rival facts unless they claim to be the same kind of result. One is an early estimate; one is a later correction. A responsible interface displays the window, revision number, last update, completeness state and policy. Replacing the number silently destroys the audit trail. Keeping every version without identifying the current one creates a different confusion.
Step 4: choose a correction contract
There are several legitimate contracts. An append-only system can emit a correction record that refers to the earlier result. An upsert system can replace the current aggregate under a stable window key while keeping a change log. A retraction-capable system can negate the old value and add the new one. The correct choice depends on downstream behaviour. If a consumer adds every message, sending “150” after “118” produces 268. If a consumer replaces by key, sending only a delta of 32 produces 32. Producer and consumer must share the same semantics.
Give the aggregate a durable identity such as gate-group, window start, window end and metric definition. Add a revision or sequence number. Make retries idempotent so the same correction cannot be applied twice. Exactly-once transport language does not remove the need for an application-level identity; a corrected business result is a new statement even when the transport processed each message once.
Step 5: test the difficult arrivals
- Send an event one second before the window boundary and another exactly on it. Confirm the interval convention, such as inclusive start and exclusive end.
- Delay one source partition while the others continue. Observe whether the watermark, output latency and backlog alarms behave as documented.
- Replay an identical event. Confirm whether record identity suppresses a duplicate without suppressing a legitimate repeated observation.
- Backfill a corrected event whose payload changes but whose identity remains stable. Confirm whether the aggregate retracts the old contribution before adding the new one.
- Advance beyond the allowed-lateness horizon, then deliver an older event. Confirm whether it is dropped, quarantined, routed to a correction workflow or used to create a new historical version.
A test passes only when both the number and the explanation are correct. It is not enough for the final total to equal 150 if the dashboard briefly reports 118 as complete, the audit log loses the correction, or a replay applies the late batch twice.
Changed-context transfer: emergency attendance data
Move the reasoning to a hospital attendance feed. Ambulance arrivals are transmitted immediately, while a partner clinic uploads a file every hour. The decision is now staffing rather than park planning. The same concepts transfer, but the cost of delay changes. A provisional low count might understate near-term demand; waiting for completeness might make the signal operationally useless. The team can publish an early estimate with a visible uncertainty band, source-coverage indicator and scheduled correction. The watermark is not a safety guarantee. It is one input to a decision that must also consider missing-source alarms and the cost of acting late.
The transfer check is precise: identify the business clock, list delayed sources, state what an early result permits, define the correction path and choose a lateness horizon that reflects consequences rather than convenience. If those elements cannot be named, the design has not transferred—only the vocabulary has.
Audit prompt
Take one production aggregate and reconstruct it from raw events. Preserve the query version, event-time field, timezone, window rule, watermark policy, allowed lateness, trigger, accumulation mode, deduplication key and source coverage. Compare the first emitted result with the latest result. Explain every difference without appealing to “the stream was late.” The aim is a chain of evidence: which records arrived, which state changed, which result was superseded and which decision saw each version.