MQTT Fixed Headers: Reject Invalid Flags and Nonconforming Remaining Lengths
Validate the MQTT packet type and fixed-header flags before accumulating its advertised body. Decode Remaining Length with a four-byte limit and minimal-encoding check, then compare the complete packet size against the effective receive bound using checked arithmetic. An incomplete prefix should remain bounded; a fourth continuation byte is already invalid and does not justify waiting for a fifth.
This guide pins the MQTT 5.0 Standard, 7 March 2019, sections 1.5.5, 2.1 and 3.3.1. It covers fixed-header admission, not complete MQTT decoding or session legality. An original C11 fixture tested every first-byte value and selected length boundaries under GCC 14.2 with AddressSanitizer and UndefinedBehaviorSanitizer.
Treat the low nibble according to packet type
The first byte combines the control-packet type in its high nibble with type-specific flags in its low nibble. Type zero is reserved. Most packet types require a zero low nibble, but three require 0x2 and PUBLISH has meaningful flags.
| Packet types | Required low nibble | Admission check |
|---|---|---|
| CONNECT, CONNACK, PUBACK, PUBREC, PUBCOMP, SUBACK, UNSUBACK, PINGREQ, PINGRESP, DISCONNECT, AUTH | 0x0 | Reject any other reserved-bit combination |
| PUBREL, SUBSCRIBE, UNSUBSCRIBE | 0x2 | Do not accept zero as a generic default |
| PUBLISH | DUP, QoS and RETAIN fields | Reject QoS bits 11; require DUP zero when QoS is zero |
The 0x2 nibble on PUBREL is fixed-header syntax, not a configurable PUBLISH-style QoS field. A retransmitted PUBREL does not acquire PUBLISH’s DUP bit. Keep the table in one reviewed decoder path instead of duplicating partially different rules across handlers.
Decode a canonical Variable Byte Integer
Remaining Length counts bytes in the variable header and payload. It excludes the first byte and its own encoded bytes. Each length byte contributes seven data bits; the high bit indicates continuation. MQTT 5 requires the smallest number of bytes that represents the value.
The length field uses one through four bytes, with a maximum value of 268,435,455. Boundary vectors should cover zero, 127, 128, 16,383, 16,384, 2,097,151, 2,097,152 and that maximum.
For example, bytes 0x80 0x00 numerically decode to zero but are not a permitted minimal representation. A decoder that only adds seven-bit groups can accept this malformed form unless it separately checks canonical length.
At the fourth length byte, a set continuation bit is sufficient to reject. Do not read a fifth byte, multiply the accumulator again or use the advertised continuation as a reason to allocate a larger header buffer.
Distinguish incomplete input from invalid input
A prefix ending after one continuing length byte can still become valid when more bytes arrive. Return an explicit need-more-input result and retain only the bounded prefix. At transport EOF, it is incomplete; under a local deadline, expire it through the documented connection policy.
Do not turn a malformed prefix into a scanning problem that searches arbitrary later bytes for a plausible MQTT start. The protocol runs over an ordered byte stream, and losing its framing is a connection error rather than permission to guess where another command begins.
The executed fixture accepts one byte per call and stores no more than five fixed-header bytes: one type/flags byte plus four length bytes. Its terminal rejection state stays terminal. It does not implement deadline handling or reconnection.
Check the complete size before body admission
The total packet size is one plus the number of encoded Remaining Length bytes plus the decoded Remaining Length. A local cap or negotiated Maximum Packet Size concerns that complete packet, not just its payload or Remaining Length value.
In the fixture’s hypothetical hundred-byte cap, a one-byte Remaining Length of ninety-eight produces a total of one hundred and is admitted. Ninety-nine produces one hundred and one and is rejected as too large. The test also rejects a cap smaller than the header itself.
Use subtraction after confirming the cap can hold the header, rather than blindly adding untrusted values into a narrow integer. C integer promotions and storage types deserve explicit review even though MQTT’s maximum value fits in a 32-bit unsigned accumulator.
Admission does not require allocating the entire body. A supported streaming decoder can use bounded storage, but it must still enforce total-size policy and complete field-level validation before application effects.
Executed coverage and its limits
The fixture independently enumerated the expected first-byte combinations and checked all 256 byte values. Twenty-four passed this narrow syntactic gate: fourteen non-PUBLISH combinations and ten PUBLISH combinations. That does not mean every such byte is legal in every direction or connection state.
Eight canonical length boundaries passed. Four nonminimal encodings and one fourth-byte-continuation encoding were rejected. The total-size equality and overflow-of-policy cases behaved as expected, and an unfinished prefix remained incomplete without allocating dynamic storage.
AddressSanitizer and UndefinedBehaviorSanitizer completed the run with leak detection disabled. LeakSanitizer could not run under the execution environment’s tracing restriction; the fixture itself performs no dynamic allocation. No target firmware, network peer or complete MQTT library was tested.
Extend admission into a full decoder qualification
- Keep the fixed-header truth table and minimal Variable Byte Integer vectors as deterministic regressions.
- Add packet-specific body lengths, property legality and identifier rules after this gate.
- Test permitted short forms, including empty DISCONNECT and compact acknowledgements, using their own rules.
- Reject bad input before application dispatch, while retaining the precise local rejection category.
- Verify the stack’s MQTT error response and connection closure under the standard’s error-handling constraints.
- Repeat on the target integer model and transport integration, including incomplete-input expiry.
A valid fixed header is only the first bounded decision. The useful release evidence shows exactly which bytes were admitted, which limit was applied, and why no malformed or oversized prefix could force uncontrolled body accumulation.
