2026-09-02 · 8 min read
- xml
- encoding
- integration
XML parsers begin with bytes, infer or read an encoding, and then construct Unicode character data. Problems arise when a file declaration says UTF-8, a byte order mark indicates UTF-16, an HTTP header names another charset, or an application decodes bytes before giving text to the parser.
Editors often hide these differences. Two documents can display identically while having different byte sequences, declarations, line endings, and signature outcomes.
Define one byte-to-text boundary
Decide whether the XML parser receives raw bytes or already decoded Unicode. Prefer raw bytes when the parser should apply XML's encoding detection; otherwise configure the transport decoder and pass text through an API intended for characters.
- Preserve the original bytes for incident fixtures.
- Record Content-Type and charset metadata.
- Reject contradictory encodings instead of guessing silently.
- Avoid decoding bytes with a platform default charset.
Understand declarations and byte order marks
A UTF-8 byte order mark is permitted but not required. UTF-16 documents normally need a byte order mark or an explicit transport-level distinction so the parser can determine byte order. The XML declaration must be at the beginning apart from an allowed marker.
Do not strip arbitrary leading bytes to make parsing succeed. A prefix may indicate logging contamination, concatenated records, a compressed stream, or the wrong protocol rather than harmless whitespace.
Preserve parsed meaning and security
Once decoded, create text and element nodes through XML APIs instead of concatenating strings or pre-escaping values. Configure limits and disable unsafe external entity resolution unless a tightly controlled application explicitly requires it.
Byte conversion and XML canonicalization are different operations. If a signature profile applies canonicalization, verify according to that profile and retain the signed representation; reserializing first can alter namespace, whitespace, or encoding details.
- Bound document size, depth, and entity expansion.
- Validate schema only after secure parsing.
- Keep binary content in a schema-defined encoding such as Base64.
- Log error class and offset without dumping confidential XML.
Create byte-aware fixtures
Use Flashman's XML formatter after successful decoding, Base64 tool to carry safe exact-byte fixtures, diff for character and serialization comparisons, hash tool to label byte variants, and HTML entities tool to inspect text escaping.
Test UTF-8 with and without a marker, UTF-16 byte orders, declaration and header conflicts, non-ASCII text, invalid and truncated sequences, line endings, CDATA, namespaces, schema validation, signatures, and each transport used in production.