2026-09-10 · 6 min read · Rahul Chitturi
- json
- unicode
- interoperability
A JSON string can contain an escaped UTF-16 surrogate such as \uD800 without its matching low surrogate. Some parsers preserve that code unit, some serializers escape or replace it, and UTF-8 encoders commonly substitute the replacement character because a lone surrogate is not a Unicode scalar value.
The result is an interoperability bug that hides behind valid-looking escapes: two services can parse the same text yet hash, store, display, or reject different output.
Locate the boundary that changes the value
Preserve a tiny public fixture and record its source characters, JSON escape spelling, parsed code units, reserialized JSON, and emitted UTF-8 bytes at every runtime boundary. Do not rely on a console glyph, which may already show a replacement character.
- Test an isolated high surrogate and low surrogate.
- Compare with one valid supplementary character pair.
- Separate literal backslash-u text from an actual code unit.
- Check parser, serializer, database driver, and message broker behavior.
Adopt an explicit Unicode policy
For interoperable APIs, validate strings as sequences of Unicode scalar values and reject unpaired surrogates before storage, signing, or transport. If a legacy protocol deliberately preserves UTF-16 code units, document that narrower contract and avoid pretending arbitrary units are valid UTF-8 text.
Do not repair malformed values silently at only one hop. Replacement changes data and can make identifiers, signatures, length checks, and audit records disagree.
Compare safe fixtures in Flashman
Use the JSON formatter to inspect escaped public fixtures, diff to compare serialized forms, Base64 to display synthetic byte samples, and number-base converter to inspect code-unit values. Browser tools reveal representation changes; production validation still belongs at the service boundary.
Add regression cases for paired and unpaired surrogates, literal escapes, non-BMP characters, UTF-8 conversion, database round trips, canonicalization, hashing, truncation, and every supported runtime.