2026-09-13 · 8 min read
- ndjson
- json
- streaming
Newline-delimited JSON, commonly called NDJSON or JSON Lines, represents a sequence as one complete JSON value per line. It supports incremental production and consumption without buffering one large JSON array.
The record boundary belongs to the format, while chunks belong to filesystems, compression libraries, HTTP transports, and runtime buffers. Any chunk can contain several lines or only part of one encoded character.
Specify the wire profile
Document UTF-8, accepted line endings, blank-line behavior, whether each record must be an object or may be any JSON value, the final-newline rule, maximum record bytes, and maximum total stream work.
- Distinguish LF from an escaped newline inside a JSON string.
- Reject raw control characters that JSON does not permit.
- Decide whether a byte-order mark is rejected or accepted once.
- Version any required envelope fields independently from framing.
Decode and frame incrementally
Use a streaming UTF-8 decoder so an incomplete multibyte sequence carries into the next chunk. Append text to a bounded buffer, extract complete lines, and retain only the unfinished suffix.
Enforce the byte limit while data accumulates, not only after a delimiter arrives. A producer that never sends a newline must not cause unbounded memory growth.
Define EOF, errors, and recovery
At clean EOF, either accept one nonempty unterminated line as the final record or reject it according to the profile. Keep clean EOF distinct from timeout, cancellation, decompression failure, and truncated transfer.
- Choose fail-fast or skip-invalid-record behavior explicitly.
- Report record number and bounded byte or character location.
- Do not log complete records or credentials in parser errors.
- Propagate backpressure and cancellation to the source.
Maintain chunk-boundary fixtures
Use Flashman's JSON formatter for synthetic records, diff for expected record sequences, hash for exact public bytes, units converter for limits, and timestamp converter for harmless event fixtures.
Split the same fixture at every byte boundary and test CRLF, blank records, escaped newlines, multibyte text, malformed JSON, oversized lines, missing final delimiters, compressed streams, slow sources, cancellation, retries, and downstream backpressure.