2026-09-10 · 8 min read
- json
- unicode
- interoperability
JSON string escapes use hexadecimal UTF-16 code-unit notation. A supplementary Unicode character is represented by a high-surrogate and low-surrogate pair, but the grammar can also admit an escape containing only one half. RFC 8259 notes that behavior for such non-Unicode sequences is unpredictable across implementations.
This distinction matters because JavaScript and some other runtimes can store lone surrogate code units in strings, while Unicode scalar-value APIs, UTF-8 encoders, databases, and downstream protocols may reject or replace them.
Model the representation layers
Document the source bytes, decoded text, JSON token spelling, runtime string representation, reserialized JSON, and destination bytes independently. A sequence containing the six ASCII characters backslash-u-D-8-0-0 is different from a runtime string containing code unit 0xD800.
- Treat U+0000 through U+D7FF and U+E000 through U+10FFFF as scalar values.
- Require a low surrogate from 0xDC00 through 0xDFFF after a high surrogate.
- Reject an isolated low surrogate as well as an isolated high surrogate.
- Count code units, code points, scalars, graphemes, and bytes according to the actual limit.
Validate before irreversible transformations
Validate incoming strings before normalization, replacement, truncation, hashing, signing, persistence, or identifier comparison. Once an encoder substitutes U+FFFD, the original malformed input and a genuine replacement character can become indistinguishable.
Apply the rule recursively to object names and string values. Reject duplicate object names according to the API policy after decoding escapes, because differently escaped names can resolve to the same runtime string.
Define canonical and cryptographic boundaries
If JSON participates in a signature or digest protocol, follow its named canonicalization profile and supported input domain. Deterministic serialization does not rescue malformed Unicode, and silently repairing text before verification changes authenticated bytes.
- Specify UTF-8 for JSON exchanged between systems.
- Preserve raw authenticated bytes until verification completes.
- Use maintained canonicalization and cryptographic libraries.
- Version any legacy contract that preserves arbitrary UTF-16 units.
Build a cross-runtime fixture matrix
Use Flashman's JSON formatter for public escaped fixtures, diff for serialized forms, Base64 for synthetic byte representations, number-base converter for code-unit values, and hash for stable fixture labels. These tools help inspect data; production acceptance rules must run in the receiving service.
Test every high and low boundary, valid pairs, repeated surrogates, literal escape text, raw supplementary characters, normalization-sensitive text, object names, database and queue round trips, logs, schema validators, canonicalizers, and all supported language runtimes.