MQTT UTF-8 Validation: Reject Invalid Topic Strings Before They Reach the Broker
Validate an MQTT topic in layers: encoded byte length, well-formed UTF-8, MQTT’s prohibited and discouraged code points, then Topic Name grammar. A generic UTF-8 decoder alone is insufficient because U+0000 is valid Unicode encoding but forbidden in MQTT UTF-8 strings. Conversely, some well-formed code points are discouraged rather than universally forbidden, so keep protocol requirements separate from a stricter sender policy.
This guide pins the OASIS MQTT 5.0 Standard, 7 March 2019, sections 1.5.4 and 4.7, and RFC 3629, November 2003, for UTF-8 byte syntax. The original reference fixture ran with Python’s strict decoder. It is not an embedded decoder, broker test or validation of arbitrary binary message payloads.
Count encoded bytes before writing the prefix
MQTT UTF-8 strings carry a two-byte length giving the number of encoded string bytes. The maximum is 65,535 bytes, excluding that length prefix. Count bytes after encoding; character count, display width and C-string allocation size are different quantities.
The hypothetical topic line/é contains six Unicode code points and seven UTF-8 bytes. Its length prefix is hexadecimal 00 07. A buffer terminator used by a local C API is not part of the MQTT field and must not be transmitted as an extra character.
Check the enclosing packet’s limits separately. A string that fits its two-byte length does not prove the complete packet fits local buffers or the peer’s Maximum Packet Size. Never truncate an identifier to make it fit, especially in the middle of a multibyte sequence.
Reject malformed UTF-8 without replacement
RFC 3629 limits UTF-8 to valid one-through-four-byte sequences representing Unicode scalar values. Surrogate code points U+D800 through U+DFFF are excluded. Reject overlong encodings, isolated continuation bytes, truncated sequences and encodings beyond U+10FFFF.
Do not decode a topic with replacement-character error handling and then publish the repaired text. That changes the identifier and can route data to an unintended topic. A printable diagnostic label can be derived separately, but it must not become the accepted wire identity.
| Hexadecimal bytes | Classification |
|---|---|
| C0 AF | Overlong form; malformed UTF-8 |
| ED A0 80 | Encoded surrogate; malformed UTF-8 |
| F4 90 80 80 | Beyond Unicode maximum; malformed UTF-8 |
| E2 82 | Truncated sequence; malformed UTF-8 |
| 00 | Well-formed U+0000; prohibited by MQTT |
| EF BB BF | U+FEFF; preserve as part of the MQTT string |
MQTT treats received ill-formed UTF-8 and U+0000 as malformed packets. The complete error-handling action belongs to the packet parser and section 4.13; a local encoder should prevent constructing such a packet in the first place.
Label the stricter sender policy
MQTT says strings should not contain control characters U+0001 through U+001F, U+007F through U+009F, or Unicode noncharacters. A receiver may treat those as malformed. This is different normative strength from the mandatory prohibition of U+0000.
A practical sender can reject all of these categories to avoid peer-dependent acceptance. Report that as an explicit interoperability policy. Do not describe every such character as an ill-formed UTF-8 sequence.
Unicode Corrigendum 9 clarifies that noncharacters do not make Unicode strings ill-formed. The permanent set consists of U+FDD0 through U+FDEF plus the final two code points in each plane, totaling sixty-six. MQTT adds its own protocol-level treatment of that set.
Keep error categories useful: malformed encoding, prohibited null, discouraged-code-point policy, length overflow and topic grammar should not all become “invalid Unicode.” That separation also makes conformance testing less ambiguous.
Do not strip a byte-order mark
MQTT explicitly requires EF BB BF to be interpreted as U+FEFF wherever it occurs in a string; a packet receiver must not skip or strip it. A text-file loader that automatically consumes a leading BOM is therefore unsuitable as the MQTT field decoder.
The fixture preserved a leading U+FEFF followed by an ASCII label through length-prefix encoding. If an application wants to prevent invisible characters when users create new topics, that must be a separate admission policy that rejects the input, not a receiver transformation that silently changes an existing identifier.
Apply Topic Name rules after string validation
A logical Topic Name must be nonempty and cannot contain the wildcard characters plus or hash. Those characters have defined uses in Topic Filters, which need a separate grammar. A generic UTF-8 validator should not attempt to implement both roles through one undocumented flag.
Spaces and a topic consisting only of slash are permitted. Leading or trailing slashes create distinct names. MQTT topic matching is case-sensitive and prohibits normalization or substitution of unrecognized characters. Preserve the original accepted representation through configuration, persistence and encoding.
MQTT 5 can encode the Topic Name as a zero-length string in a PUBLISH using a valid Topic Alias. The Topic Name field remains present, including its two-byte zero length. That packet-level optimization does not create a valid empty logical topic. Keep alias-state validation outside this logical-name fixture rather than weakening its nonempty rule.
Executed conformance vectors
The local reference fixture rejected eleven malformed byte vectors and every one of the 2,048 surrogate encodings. It separately classified U+0000, sixty-four discouraged control characters and all sixty-six noncharacters under the stricter sender policy.
Additional assertions checked a valid four-byte scalar, BOM preservation, the seven-byte example prefix, exact 65,535-byte acceptance and 65,536-byte rejection, generic empty-string handling versus Topic Name rejection, wildcard rejection, slash and trailing-space acceptance, and preservation of two different accented representations.
These tests rely on Python’s strict UTF-8 implementation and are not exhaustive proof of a C decoder. Reuse the vectors against the actual encoder and decoder, including input split at every byte of representative multibyte sequences. Verify bounded buffering and no application dispatch before the complete field has passed validation.
Acceptance means rejected input leaves the prior valid topic unchanged, accepted bytes survive exactly, and every error names the violated layer. No broker connection is needed to catch these local defects, although final compatibility still requires the selected client library and broker versions.
