Changing a Production Test Parser: Compare Old and New Results on the Same Raw Fixtures
Compare old and new test parsers on identical raw fixtures, but judge both against an independently specified meaning. Agreement can preserve a shared defect. A justified parser change may deliberately alter a result, especially when the old implementation mishandled units, missing data or invalid numeric text.
This article executes sixteen original host fixtures on CPython 3.12.14 with two small synthetic parser revisions. They are not deployed station software. The output is a semantic-difference record; it does not establish manufacturing yield, calibration equivalence, hardware qualification or which historical units require retesting.
Specify the reading contract before comparing implementations
For the example, a record is an ASCII signed decimal followed by one space and either V or mV. The numeric part contains one to six integer digits and optionally a decimal point with one to six fractional digits. Exponents, surrounding whitespace, commas, underscores, Unicode digits and nonfinite tokens are excluded deliberately.
These restrictions are an illustrative input contract, not universal instrument syntax. A real parser must implement the exact approved protocol, including any framing, encoding and locale requirements. Preserve the received bytes so normalization cannot erase evidence of a malformed input.
Canonical values use volts. The example decision interval is inclusive from 3.200 V through 3.400 V. A valid reading outside that interval is a product-rule failure; malformed or missing input is invalid evidence. No measurement uncertainty or guard band is implied by this simple parser example.
Keep three outputs separate
Record parse status, canonical value/unit and final decision independently. A missing value must not silently become a measured zero. Conversely, an actual zero can be validly parsed and then fail the numeric requirement.
Store the expected fields in a reviewed fixture manifest. Do not populate them by calling the old parser or by copying the new parser’s output after a test fails. For each intentional change, retain the requirement or defect explanation that justifies the difference.
The old synthetic parser intentionally contains two defects: it treats mV numbers as volts and defaults rejected input to zero. The corrected parser rejects invalid grammar and converts mV to volts. Both use the same exact inclusive decision rule once a valid canonical value exists.
Read the paired fixture results
| Raw input | Independent expected meaning | Old parser | New parser |
|---|---|---|---|
| 3.199 V | Valid 3.199 V; fail. | Same meaning. | Same meaning. |
| 3.200 V and 3.400 V | Both exact inclusive boundaries pass. | Both pass. | Both pass. |
| 3.201 V and 3.401 V | First passes; second fails. | Matches. | Matches. |
| 3200 mV | Valid 3.2 V; pass. | 3200 V; fail. | Corrected value and decision. |
| -3300 mV | Valid -3.3 V; fail. | -3300 V; fail. | Corrected value; decision unchanged. |
| 0 V | Valid zero; fail. | Valid zero. | Valid zero. |
| Empty, NaN V or Infinity V | Invalid; no canonical reading. | Defaults to zero and fail. | Invalid, not a measured failure. |
The sixteen-fixture campaign also rejected an underscore, decimal comma, missing separator, leading space and a fullwidth digit. Every corrected output matched its independent oracle. Ten fixtures had approved semantic differences from the deliberately defective old parser.
The negative mV case is important: matching pass/fail categories would hide a thousandfold value error. The invalid-input cases show the reverse problem: a red result alone cannot tell whether the product failed or the test failed to obtain a reading.
Use Decimal without confusing it with a grammar validator
The CPython 3.12.14 decimal reference documents that Decimal string construction accepts forms beyond this example’s grammar, including nonfinite values, whitespace, underscores and other Unicode decimal digits. Decimal(text) alone therefore does not enforce the station protocol.
The executed parser first applies a full-match bytes grammar, then constructs Decimal from the captured ASCII number and checks finiteness. It divides mV by exactly 1000 under sufficient decimal precision for the bounded input. It never routes through binary float.
Decimal construction from a binary float preserves that float’s exact value, which can differ from the intended decimal text. Retain the original numeric token and a deliberate conversion policy rather than using Decimal merely to decorate an already rounded float.
Compare meaning exactly where decisions are exact
The fixture harness compares canonical numeric values separately from their textual formatting. Decimal 3.200 and Decimal 3.2 have equal numeric meaning here; preserving display precision may be a separate requirement in another protocol.
Assert parse category, canonical unit and final decision exactly. Do not use a broad numeric tolerance that allows a below-limit reading to match an above-limit reading while their classifications differ. If a real algorithm requires numerical tolerance, justify it from that algorithm and retain exact decision assertions.
Include the chosen rule’s below, at and above cases, both signs, unit conversions, absent versus zero, malformed and nonfinite inputs. A fixture suite limited to comfortable middle-of-range values can miss the changes that matter most.
Approve differences rather than forcing equality
Group paired results into unchanged-correct, approved-correction, unexplained-change and shared-defect categories. A new parser agreeing with an incorrect old result belongs in the last category, not the first.
Bind the raw fixture files, oracle revision, parser revisions, interpreter version and actual execution report. Keep any original failures and the reviewed correction rationale. The host campaign here used ordinary assertions and retained all sixteen paired records; no pytest installation or production parser execution is claimed.
Only after semantic qualification should another process assess the historical scope of the change. This article stops at explained parser behavior. Whether old raw data remain valid, which units were exposed and whether a new physical observation is required are separate engineering decisions.
