Did you know? A network response can contain a complete JSON object, half of the next object, and the beginning of a third object in a single read. An AI agent that treats each network chunk as one record may silently omit data or act on incomplete information. Newline-delimited JSON, often called NDJSON, gives a useful record boundary, but only if the receiving system buffers bytes, validates complete lines, applies strict size limits and separates receipt from irreversible processing.
The safe answer is to parse a stream as a sequence of UTF-8 JSON texts terminated by line delimiters, not as arbitrary transport chunks. Buffer an unfinished line, decode and validate each completed record, assign stable provenance and deduplication identity, acknowledge only durable progress, and reconcile after disconnects. A newline indicates the end of a serialized record; it does not prove a business action has been committed. A client must check the producer’s documented replay or cursor semantics before claiming exact-once effects.
This article concerns current software agents and data pipelines. Artificial superintelligence remains speculative; the series name does not assert its existence. The original examples below are fictional engineering cases with explicitly checked quantities, not claims that a live service was operated or tested.
Contents
- The first principle: transport chunks are not records
- Establish the producer and consumer contract before reading
- UTF-8 and partial characters need careful buffering
- A small state machine is safer than one clever split call
- Worked case: seven records split across five network reads
- Offsets, sequence numbers and idempotency are separate tools
- Line-size, decompression and computational limits
- Untrusted JSON is data, not an instruction channel
- When events arrive out of order or change meaning
- Recovery after connection loss must start from durable truth
- Common failure patterns and precise fixes
- Transfer problem: NDJSON records become student feedback messages
- What this format guarantees—and what it does not
- Frequently asked questions
- Learning routes and primary documentation
The first principle: transport chunks are not records
A byte stream is a sequence of bytes whose transport may expose arbitrary portions at a time. A reader can receive five bytes now, twenty later and one final byte on disconnect. Those chunk boundaries come from buffering, scheduling, proxies and runtime behaviour. They do not correspond reliably to JSON object boundaries, document boundaries or transaction boundaries. A program that calls JSON.parse on each received chunk is making an assumption the transport did not promise.
NDJSON defines its own simpler boundary: each individual JSON text is followed by a line feed, optionally with a preceding carriage return. Each record is JSON according to RFC 8259, and the NDJSON specification requires UTF-8. This design lets a consumer process complete records incrementally without holding the entire dataset in memory. The delimiter is part of the framing convention, not a feature of ordinary top-level JSON arrays.
An unescaped newline cannot appear inside a JSON string under the JSON grammar. A string containing a conceptual line break serializes the break as an escape rather than as an actual line delimiter. That is why splitting a correctly produced NDJSON stream at complete newlines can work. It is still unsafe to split arbitrary network chunks directly, because a line can span many chunks and multibyte UTF-8 characters can be split across reads.
A meaningful mental model has three nested layers: bytes arrive from transport; framing assembles complete lines; JSON parsing turns a valid line into a value. Business validation and side effects happen afterward. Every layer has its own error classes. A byte decoding error is not the same as invalid JSON; a valid object without a required ID is not a successful business record; and a correctly applied record is not necessarily acknowledged to the sender.
Compare this with server-sent events, which has event framing and reconnection fields of its own. An SSE event’s data lines and optional event IDs should not be parsed as plain NDJSON framing. The reader must identify the actual media type and protocol rather than treating every streaming API as the same line-oriented language.
Establish the producer and consumer contract before reading
An agent should know what the producer claims to send: object records, arbitrary JSON values, or only a predefined schema; whether blank lines are permitted; whether an ordinary line feed or carriage-return-plus-line-feed is accepted; the character encoding; the maximum record size; and whether the stream is finite or indefinite. The NDJSON specification allows parsers to ignore empty lines if that behaviour is documented and configurable. A production pipeline should choose one explicit policy and test it.
Some applications call their format JSON Lines rather than NDJSON. The families are closely related, but real producers may differ on whether the final record must be newline-terminated, whether blank lines are accepted and whether records can be top-level scalar values. A conservative importer should record the expected dialect and not infer correctness from a file extension alone. The authoritative producer documentation wins over a casual generalisation.
Media type and provenance also matter. The NDJSON project suggests application/x-ndjson, but receiving an HTTP response with that type does not make its contents trusted. Inspect the declared origin, authenticated caller or server, content limits and intended use. A response may be proxied, transformed, compressed or cut short. The agent should verify content encoding before applying line framing to decoded body bytes.
Schema is a separate layer from syntax. A line such as a JSON string or the number 42 is syntactically legitimate JSON, even if the business workflow expects objects with id, timestamp and status. Implement a validation rule for the intended record shape, required fields, allowed numeric ranges and version identifiers. A successfully parsed JSON value can still be the wrong semantic kind of record.
Document what an invalid record means. In a human-reviewed import, quarantine a bad record with its safe source context and let the operator repair it. In a stream controlling external actions, stop before executing later records if ordering or cumulative invariants might be affected. A system that silently discards malformed lines can produce misleading totals and an inconsistent downstream state.
UTF-8 and partial characters need careful buffering
Consider the character é represented in UTF-8 by multiple bytes. One transport read can stop after the first byte; the next read delivers the remainder. Calling a strict UTF-8 decoder on each chunk independently may throw a false decoding error or produce a replacement character. Instead, use an incremental decoder or accumulate byte sequences with well-defined state until complete encoded characters can be decoded.
Many simple NDJSON implementations buffer raw bytes until a line feed, then decode a complete line. This is practical because the newline delimiter itself is a single byte in UTF-8, and a correctly encoded non-ASCII character will not use that delimiter as a continuation byte. Apply a byte-size bound before accumulating untrusted line content. Where the API can send very large JSON values, agree on a deliberate maximum and an explicit oversized-record response.
Normalize line endings only according to the documented format. For NDJSON, a line feed may be immediately preceded by carriage return, so that preceding carriage return may be removed as part of the delimiter handling. Do not indiscriminately strip all control characters or spaces in payload strings. JSON whitespace outside strings is already handled by the parser; arbitrary string normalization can change business meaning.
Watch for a UTF-8 byte-order mark. RFC 8259 says generators of JSON texts exchanged among systems must not add a BOM, though parsers may ignore one rather than treat it as an error. Choose and document a handling policy. Repeated BOM characters embedded before later records are not equivalent to a single optional leading marker in a file.
A parser should report whether it accepted the final line only because an end-of-file policy permits an unterminated last record. In a live streaming protocol, a missing newline may mean the producer was interrupted before committing its last serialized record. If the producer guarantees newline termination, do not silently promote a trailing fragment into a complete business action.
A small state machine is safer than one clever split call
Think of the reader as a bounded state machine with an unfinished byte buffer, a record counter, an error state and a durable acknowledgement cursor. At each read, append bytes up to a configured memory bound; while a complete delimiter is present, remove one complete line, decode it, parse JSON and validate it. The remaining buffer represents an unfinished record, not a failed object.
Differentiate complete-line validation from line-delimiter discovery. The parser can know that a newline has arrived while still rejecting the JSON it frames. An incomplete string might be valid once more bytes arrive, but after a delimiter has completed a line, the same unfinished string is invalid for that record. This distinction avoids waiting forever on a line that is already known to be malformed.
Do not perform remote side effects inside a low-level framing loop without a transaction plan. The decoder should emit a validated record or a structured error to a higher layer. The action layer can then check permissions, duplicate identities, predecessor requirements and retry behaviour. Separating these layers makes testing easier: transport tests need not call payment, email or database interfaces just to exercise byte boundaries.
When a consumer crashes, its in-memory partial buffer may vanish. Resume correctly by following the producer’s explicitly documented cursor or byte-range mechanism, ideally from the last durably acknowledged complete record. If the stream is a single immutable file and the protocol supports stable byte ranges, byte offsets may be meaningful; for a live regenerated stream, an offset from yesterday’s response may be useless.
A durable checkpoint includes a source identity, format version, last committed record identity and appropriate integrity or version checks. It should not claim that a record is applied merely because its line ended in the network buffer. In a multi-step application, acknowledgement comes after the write that makes downstream state recoverable.
Worked case: seven records split across five network reads
Imagine a fictional service sends exactly seven records identified 101 to 107. Their serialized sizes, including delimiter bytes, are 40, 32, 28, 44, 36, 20 and 50 bytes. Total expected transfer is 40+32+28+44+36+20+50 = 250 bytes. For this teaching case the records are all valid and no line contains a delimiter inside a string. The exact JSON contents are not important to the byte arithmetic.
Five transport reads deliver 17, 75, 46, 81 and 31 bytes. Their sum is 17+75+46+81+31 = 250 bytes, so the transport delivered the entire planned file in this example. After the first read there is no complete record and the buffer has 17 bytes. After the second read, 17+75 = 92 bytes have arrived; two records of 40 and 32 bytes complete, leaving 20 bytes toward the third.
After the third read, cumulative arrival is 138 bytes. The first three records require 100 bytes and the fourth requires another 44. Because 138−100=38, the decoder can safely emit the first three records and retain 38 bytes of the fourth. After the fourth read, cumulative arrival is 219 bytes: the first six records require 40+32+28+44+36+20 = 200 bytes, so six complete records have been framed and 19 bytes remain of record seven.
The last read adds 31 bytes, making the retained final record 19+31=50 bytes including its newline. The decoder emits record seven. Seven records were framed, seven validated and, assuming the higher application layer commits each and records receipt, seven may be acknowledged. A naive decoder treating each of the five reads as a JSON record would produce the wrong number and possibly errors even though every byte was eventually delivered.
Now change one condition: the stream stops after the fourth read. The source delivered 219 of 250 bytes, or 87.6% of the expected total bytes. Six of seven complete records were received, approximately 85.71% of the records. These percentages have different denominators. The trailing 19 bytes are not a seventh complete record. A reliable report states six complete records plus one unfinished fragment rather than claiming seven.
If the application had committed only the first five records when the connection stopped, its safe acknowledgement count is five, not six. Framing is a syntax outcome; committing is an application outcome. After reconnect, the agent must reconcile the status of record 106 before deciding whether to apply or skip it. It must not multiply a high byte-completion percentage into a fabricated business-completion claim.
Offsets, sequence numbers and idempotency are separate tools
A byte offset names a position in one particular encoded representation. A sequence number names an event’s order under a documented producer numbering rule. A business record ID identifies one logical entity or action. These are not interchangeable. Compression, regenerated timestamps and different serialization whitespace may change byte positions without changing logical records. A new stream can restart sequence numbers or include updated versions.
If a source guarantees immutable content and supports range reads, a consumer can checkpoint the byte offset after the last accepted newline. But before resuming it must verify that the source is the same representation, for example by a strong validator where appropriate, rather than applying an old offset to different content. Byte-resume safety is addressed separately in untrusted file ingestion and related transfer guides.
Sequence numbers are useful for detecting missing or repeated events. Suppose IDs 10, 11, 13 arrive. A consumer that expects a contiguous sequence must not assume event 12 never existed. It should mark a gap and use a documented recovery procedure. Conversely, 10, 11, 11, 12 may contain a duplicate that must be distinguished from a legitimate correction using stable event IDs or versions.
Deduplication should generally use a scope-qualified identity such as source plus event ID plus version. Deduplicating merely by the JSON line’s text can merge two distinct legitimate events with identical fields; deduplicating by arrival timestamp will fail to detect replay. The right key depends on producer semantics and must be documented, not guessed from convenience.
For any irreversible side effect, an idempotency key or transactional state machine can help prevent duplicate application after reconnect. But exactly-once effects are an end-to-end property requiring sender, receiver and external service guarantees. A newline, message ID or retry counter alone cannot establish it.
Line-size, decompression and computational limits
Streaming a file does not automatically make it safe for memory. A producer can send one giant line without a newline and force a naive consumer to buffer forever. Configure a maximum encoded line size based on legitimate workloads and reject excess bytes before the buffer grows without bound. Also cap total records, nesting depth and processing time where the environment makes those limits relevant.
A compressed response can be small on the wire but huge after decompression. Apply separate compressed-size and expanded-size limits, and understand whether the streaming library exposes bytes before or after decompression. The record framing policy should be defined over the decoded NDJSON body, not over the arbitrary compressed wire segments. Otherwise the decoder may search for newline bytes in content that is not yet plain JSON.
JSON strings can contain long numbers or deeply nested structures. Standard JSON syntax alone does not guarantee a parser will preserve arbitrary numeric precision or tolerate any depth safely. For money, identifiers and scientific quantities, validate representation and range, then use appropriate numerical types. A valid JSON number may still be unacceptable for a downstream system with limited integer precision.
Operational limits must include the higher action layer. Ten thousand valid records can induce ten thousand API calls; even safe framing can overwhelm a downstream service. Use a bounded queue, rate limits and a backpressure contract. If the consumer cannot keep up, avoid acknowledging records it has not persisted. Error messages should report which records were accepted and which remain unresolved.
A good rejection is explicit: source identity, line ordinal, expected schema or limit, safe diagnostic excerpt and recommended action. Avoid logging a whole untrusted record if it may contain secrets or personally identifying information. Observability is valuable when it preserves the distinction between a parse error and a prohibited downstream action.
Untrusted JSON is data, not an instruction channel
A malicious record can be perfectly valid JSON. It might contain a field that says to ignore previous rules, change a payment recipient or contact an unfamiliar URL. Syntax validity does not grant its text authority. Agents should treat imported descriptions, comments and free-text fields as untrusted task data, while their action permissions are established independently by the authorised workflow.
Validate any path, filename, URL and identifier before using it outside parsing. A record that contains a relative path is not automatically allowed to escape an approved workspace. A URL in a record may point at internal services or credential-bearing endpoints. A recipient field may identify someone the operator did not authorise the agent to contact. Domain-specific gates belong after parsing and before side effects.
Schema validation helps, but it cannot settle all semantic risks. An object with exactly the expected keys may contain a mislabelled unit, a stale account reference or a value that is technically in range but still wrong for the current task. Check invariants against trusted external state where consequences demand it. Never treat a generated confidence score as a substitute for those checks.
Provenance should travel with each record: authenticated source, observed stream or file version, original line ordinal, validated schema version and any cryptographic integrity verification supported by the producer. If a record is reserialized, preserve an audit reference to the original source rather than claiming that a new sequence of bytes is the source’s original text.
For sensitive workflows, keep ingestion and execution separate. A first phase can parse, validate and stage records in a restricted environment; a second phase reviews and applies only permitted operations. This separation makes interrupted processing recoverable and limits the impact of malicious or merely mistaken content.
When events arrive out of order or change meaning
NDJSON preserves the sequence in which its records are serialized within a given stream. It does not promise that the records describe real-world events in chronological order. A producer may batch delayed messages, re-send missed events or include corrections with earlier event timestamps. A consumer that sorts records by the displayed timestamp can silently reorder dependencies and break the intended event semantics.
Define whether the source’s order is authoritative, whether records are independent, and whether there is a monotonic version or sequence number. If record 103 updates an object created by record 102, it may need to wait for 102. If two records are independent observations, safe concurrent processing may be possible. Framing alone does not decide this.
Corrections deserve an explicit policy. A source may send version two of record 101 after version one was already processed. Deduplication by record ID alone would discard a legitimate update, whereas unguarded application could overwrite a newer state with an older replay. Pair identity with version or effective-time rules suitable to the source.
Separately, a stream may carry a mixture of data records and control records. If a producer supports a final summary record with counts or a checksum, treat that as part of the documented schema rather than as ordinary application data. The absence of an expected final summary may indicate a truncated transfer even if the last observed data record had a newline.
The event-ordering guide explores gaps, duplicates and projections. Here the distinct reader problem is narrower: how raw UTF-8 bytes become correctly framed NDJSON records and when those records can legitimately enter an event-processing system.
Recovery after connection loss must start from durable truth
A disconnection may occur after the server sent a complete record but before the client persisted it, or after the client committed it but before the acknowledgement returned. These ambiguous cases are routine in distributed systems. A safe reconnect plan always starts by querying or replaying against durable state, not by trusting the last line seen in a terminal window.
For a finite immutable NDJSON export, the client can track an expected object version and last committed byte offset if the storage protocol supports stable range reads. On restart, it verifies source identity and resumes from the next complete record boundary. A byte-range request to a changed representation is unsafe even if the offset is numerically plausible.
For a live server that issues opaque cursors, follow its documented cursor semantics. Some cursors are only temporary; some apply to a particular filter and some expire. Do not invent a cursor by incrementing a timestamp or guessing that an event ID is contiguous. If the cursor is invalid, recovery may require a fresh snapshot plus event reconciliation.
Backfill is not always free. Replaying all records may be safe for a pure read-only cache but dangerous if each record triggers an external email or payment. An application should store stable record identities and side-effect completion receipts. A replayed record can then be recognised, checked and skipped or explicitly reprocessed according to policy.
At the end, compare committed record counts, error counts and any producer-provided manifest. A mismatch should be surfaced with the affected range and unresolved status. Do not claim the stream is complete merely because the network connection closed cleanly; EOF without a documented completion condition is just EOF.
Common failure patterns and precise fixes
Failure one is splitting each network chunk into JSON values. Repair it with a persistent bounded buffer and delimiter-based framing. Failure two is decoding every partial chunk independently into text and silently replacing broken UTF-8 sequences. Repair it with incremental UTF-8 decoding or byte-oriented delimiter framing followed by strict decoding of completed lines.
Failure three is accepting a trailing incomplete line as a completed event without a documented end-of-file policy. Repair it by treating the fragment as unresolved until source completion is confirmed. Failure four is discarding blank lines silently when the ingestion contract requires them to be errors. Repair it with an explicit configurable blank-line policy and a test that states the selected behaviour.
Failure five is assuming valid JSON means an authorised action. Repair it with a schema gate, provenance check and independent permission layer. Failure six is allowing unbounded record lengths or decompression to exhaust resources. Repair it with byte, depth, count and time limits, plus graceful rejection that does not overwrite prior committed state.
Failure seven is resuming from an offset against modified content. Repair it by validating source version and range semantics. Failure eight is confusing receipt with durable application. Repair it by recording the last committed identity and reconciling side effects before acknowledging or replaying.
Failure nine is treating stream order as event-time order. Repair it by explicitly documenting and testing the source’s ordering contract. Failure ten is logging complete sensitive records as debugging context. Repair it with scoped, redacted diagnostics that still identify the failed line and reason.
The most useful diagnostic question is often not ‘What went wrong with JSON?’ but ‘At which layer did the agreement fail?’ Framing, decoding, parsing, schema, ordering, authorisation, commit and acknowledgement are distinct checkpoints. Identifying the first broken invariant keeps the repair narrow.
Transfer problem: NDJSON records become student feedback messages
Now imagine an invented tutoring system that imports a daily NDJSON export containing 120 feedback records. The system must not send a message until the record’s student reference, intended recipient and review status are verified. The export reaches the service in 15 network reads of varying sizes. Those 15 reads are transport artifacts, not fifteen feedback records.
Suppose 120 complete lines are framed, but two fail schema validation, three repeat already processed event IDs and one valid record awaits a required teacher approval. There are 120 framed records, 118 schema-valid records, 115 unique schema-valid record IDs after removing the three duplicates, and at most 114 currently eligible records if the one awaiting approval belongs to that unique set. These numbers are hypothetical and the denominators differ deliberately.
An implementation that reports ‘118 messages sent’ based on valid JSON count is wrong. The 118 includes duplicate events and may include an unauthorised recipient. An implementation that reports ‘114 messages sent’ merely because 114 are eligible is also premature: eligibility precedes actual email acceptance, delivery and receipt. The application must separately verify the downstream send state.
Change one condition: the producer repeats two of the records with a higher version representing corrected text. Deduplication by ID alone would now be too coarse. The consumer needs a source-defined version policy and perhaps a requirement to re-approve changed content. The line-framing component remains the same while the business record contract changes.
The transferable answer is that NDJSON makes incremental parsing convenient, not human-facing decisions automatic. The safest pipeline preserves counts and identities at every layer and requires actual external evidence before claiming that a student or parent has received anything.
What this format guarantees—and what it does not
Correctly formatted NDJSON supplies a practical record delimiter and independent JSON validity for each line. It does not guarantee record schemas, event uniqueness, chronological order, reliable network delivery, durable storage or trustworthy sources. Those properties require additional protocols and application logic. It is misleading to call an NDJSON pipeline exactly once simply because each physical line contains one object.
Line-delimited JSON is convenient for append-only logs and batch exports. It may be a poor fit when a document must be updated atomically as one unit, when records contain extremely large binary payloads or when a richer streaming protocol supplies explicit event names, retries and acknowledgements. Choosing it should follow reader needs, not familiarity alone.
Different projects use slightly different end-of-file and blank-line conventions. Treat the actual producer contract as authoritative and test edge cases, especially incomplete final lines, embedded escaped line breaks, Unicode boundaries and oversized records. A permissive parser can mask producer bugs by silently dropping malformed content.
This discussion does not imply a model will parse any real stream correctly from its language understanding. Reliable ingestion depends on deterministic byte-processing code, documented limits and independently observed outcomes. Agent reasoning can help design and diagnose that code, but the actual records and side effects require machine-verifiable checks.
The cases and measurements in this guide are fictional. The NDJSON specification and IETF JSON standard govern the format statements; the operational proposals are engineering controls whose fit must be tested in the actual environment.
Frequently asked questions
Is NDJSON one large valid JSON document? Usually no. It is a sequence of individually valid JSON texts separated by lines, not an array enclosed in square brackets. A parser expecting a single standard JSON document for the entire stream will generally reject multiple top-level texts.
Can a JSON string contain a newline? A logical newline can be represented by an escape in the serialized string. An unescaped literal line break is not valid inside a JSON string under the standard grammar, so proper serialization preserves delimiter meaning.
Does a network read always contain a complete line? No. Chunks can end anywhere, including in the middle of a JSON token or multibyte UTF-8 character. Accumulate bytes until the declared delimiter arrives.
What about a final line without newline? Follow the producer contract. Some JSON Lines tools accept it for a finite file, but strict NDJSON serialization specifies a newline after each JSON text. Treat an unfinished live stream conservatively.
May blank lines be ignored? NDJSON parsers may ignore empty lines if the behaviour is documented and configurable. A more restrictive system can reject them, especially when every physical record is expected to be accounted for.
Does valid NDJSON make data safe to act on? No. Schema, provenance, identity, permissions and application-specific invariants must be checked separately.
How should a stream resume after a disconnect? From a producer-supported cursor or an immutable file version with stable offsets, after reconciling the last durably committed record. Do not guess from the final characters seen.
Can NDJSON ensure exactly-once processing? Not by itself. Exactly-once effects require coordinated record identities, durable state and appropriate external operation semantics.
How do I find a malformed line? Record its safe source identity and line ordinal, then report decoding, parsing or schema failure distinctly. Avoid exposing whole sensitive payloads in logs.
Learning routes and primary documentation
If you write streaming clients, begin with a bounded byte buffer, a precise line-ending rule and a strict UTF-8 decoder. Unit-test a single line split at every possible byte boundary, including within a Unicode character, before enabling action tools. Add recovery tests that crash between frame completion, validation, durable commit and acknowledgement.
If you operate an agent workflow, document record identity, validation and approval separately from transport receipt. Review how an attacker could use one valid record to trigger an unauthorised action. Compare with untrusted uploads and parser isolation and event ordering for adjacent but distinct layers.
Primary technical sources: the maintained NDJSON specification; the IETF RFC 8259, The JavaScript Object Notation Data Interchange Format; and the RFC 3629 UTF-8 encoding definition. They support the format and encoding distinctions; they do not certify any particular agent application’s delivery guarantees.
For a wider reading pathway, visit How Super Intelligence Works, the How X Works hub and the education-to-life learning framework. The worked scenario remains illustrative, and no external stream was processed to produce its numbers.