ADR 0015: Codegen determinism and the conformance suite

Status: accepted

Context

The same IR and options must give the same files, byte for byte, on every machine (FR-14, SM-3). Consumers commit generated code and review its diffs, so any nondeterminism shows up as noise in their pull requests, and drift checks become impossible.

Eight server targets in three languages must also prove the same server boundary (amendment A17). Each target maps the contract onto its framework’s own mechanisms and the declared wire format (A30), so the proof must check observable HTTP behaviour, not generated source. FR-35 asks for one black-box HTTP suite, written once, run against every server target and every adapter.

Options

• Generation: targets free to read their environment and format with whatever the consumer has installed, or a pure generate step with pinned formatters. Choice: pure (FR-14, SM-3), since committed output and drift checks need byte-identical files on every machine.
• Proof of the boundary: golden tests of generated source only, a test suite written per target, or one black-box HTTP suite run against every server target, beside golden tests. Choice: the shared suite (FR-35, A17). Targets differ in source but must agree on the wire, and a suite written once cannot drift between targets.

Decisions

Generate is pure

A target’s generate step is a function of four inputs only: the IR without its location side table, the validated target options (formatting and file layout included, below), the link descriptors (ADR 0009), and the target’s own version. It returns a list of files, each a relative POSIX path and its bytes.

Inside generate, a target never reads the file system, the network, environment variables, the clock or a random source, and never calls locale-sensitive APIs (localeCompare, Intl, toLocaleString, locale-default string casing). Sorting uses code-unit order. Paths use / and never contain an absolute prefix.

The TypeScript targets enforce this with Biome’s restricted-imports and restricted-globals rules on their generate modules. The native generator libraries (ADR 0010) follow the same rules; I/O lives in the plugin layer around them, never in the generator.

Writing files

The file writer (in codegen-core, and its equivalent in the native IR packages) owns the output directory:

• files are UTF-8 without a byte-order mark, with LF line endings and a final newline;
• it writes only inside the configured output directory;
• it keeps a manifest in that directory listing each generated file and its hash, and removes files a previous run generated that are no longer produced;
• it refuses to overwrite a file it did not generate, and a generated file whose hash shows a hand edit, naming the file and the fix (move the code next to the generated files, then delete the edited file).
• it refuses a run whose files include two paths equal once case-folded (Tasks.ts, tasks.ts), or two directories that differ only by case (Routes/a.ts, routes/b.ts), or a name used as both a file and a directory, naming both before writing anything, since on a case-insensitive file system the second would overwrite the first or the paths would drift between systems (file.path.invalid, as for a path generated twice). The fold is the same in the four writers: lower case, then upper case, then lower case (a plain case fold or upper-then-lower differ on the capital sharp s and the dotless i). There is no Unicode normalisation: generated names are made of naming segments [a-z0-9]+ and fixed parts, so NFC and NFD forms of one name cannot differ. A file a previous run generated and this run renames only by case is refused with its own message and the fix (delete the old one, then generate again). The shared pairs are in packages/core/spec/vectors/generated-paths/case-fold.json, read by the four writers’ tests.

seampact check with checks.generated (ADR 0011) regenerates into memory and reports any byte difference with the committed files, listing them; at error, the default, the difference fails the check.

The generated header

Every generated file that has a comment syntax starts with one line, in that syntax:

Generated by @seampact/codegen-nestjs 1.2.0 from api "tasks". Do not edit.

The line holds the target package and version and the api name. It never holds a timestamp, a machine path, a user name or a hash of the contract, so a contract change touches only the files it affects. Files without comment syntax (JSON) carry no header; the manifest covers them.

Formatter pinning

• TypeScript targets format with Prettier, pinned to an exact version as each target’s own dependency, called in-process. Its configuration is a target option, set in the app config (ADR 0011), that accepts every Prettier option except plugins, each with Prettier’s default: code style is the consumer’s choice, never imposed (A36). The consumer’s Prettier configuration files, .editorconfig and plugins are never read, so the output depends only on declared options; plugins stay out because their versions would escape the pin.

• The spring-boot generator builds Java with JavaPoet (Palantir’s maintained fork), pinned; its printer is deterministic, and its indent is a target option.

• The PHP targets print with nette/php-generator, pinned, through its PsrPrinter by default (four spaces, as PSR-12, PER and Laravel Pint expect), with the indent as a target option.

• File layout is a target option too: one file per resource or per schema, index files, directory and file naming, each with a default. The naming options of ADR 0007 work the same way.

Changing a formatter version is a target release, and its golden diffs are reviewed like any change. Generated code lives in the app’s one generated directory (__generated__ by default, ADR 0011), which consumers exclude from their own formatters and linters; seampact init writes the ignore entries for the tools it detects, unless --no-ignore-files, and prints each file it changes.

Golden tests

The golden trees of the server targets hold no runtime: generated code imports the runtime package (ADR 0025).

Every target has golden tests:

• inputs: the reference contract (the tasks api with a tenant-scoped route with rights, a signed webhook and a service api with relays) and one fixture per feature the target supports;
• one golden set per wire format the target claims: the core default (bare payloads with problem+json errors) for every target, plus each format pack it supports, such as the api-platform pack for the api-platform target (A30);
• the expected file trees are committed beside the target and compared byte for byte; an explicit update mode rewrites them, and the diff is reviewed in the pull request;
• every golden tree is also compiled with the target’s real toolchain: tsc in strict mode, javac on Java 21 and 25 through Gradle and Maven (A13), PHP lint and PHPStan;
• the naming vectors of ADR 0007 run in each target’s test suite (FR-38).

The determinism matrix

A CI job compiles and generates the reference contract and the scale fixture (FR-18) on Linux, macOS and Windows, with TZ set to UTC and Pacific/Chatham, LANG set to C and tr_TR.UTF-8 (Turkish casing breaks naive case conversion), and a Windows checkout with core.autocrlf=true. It runs each build twice and compares the hashes of the IR and of every generated tree with the Linux baseline. The Node targets and the native generators are both in the matrix (SM-3).

Runtime helpers

Optional runtime helpers (ADR 0009) are hand-written, semver-versioned and unit-tested in their own ecosystem. Generated code never imports them; it imports only the hooks package and, for a server target whose boundary runtime ships as a package, that runtime package (ADR 0025). A helper that has shared vectors, such as the signed-webhook verifier (ADR 0013), passes the same vectors in every ecosystem. The sample apps of the HTTP suite use the helpers where a real application would, so the suite exercises them too.

The shared black-box HTTP suite

@seampact/http-suite is a tool package (ADR 0009), written once in TypeScript (FR-35). It sends HTTP requests to a running sample app and checks status, headers and body. It never reads generated source.

• Sample app. Each target’s test consumer (ADR 0011) builds a sample app from the suite’s contract: the reference contract plus the conformance fixtures. Its handlers return scripted data; its hooks are doubles that answer from a script and record their calls.
• Control routes. The sample app exposes test-only routes to load a script, read the recorded hook calls (for the check order and short-circuit, and the throttle bucket a decide hook receives) and read captured log lines (for redaction). They are framework routes outside the contract, so the sample app allow-lists them in its config with a reason, which also exercises the boot check’s allow-list (CHK-063). The script format is language-neutral JSON, published with the suite, so a Java or PHP sample app implements the same doubles.
• Boot tests. Misconfigured variants of the sample app (a missing authenticate hook, decider, scope resolver or handler, an unlisted framework route) run as separate processes and must refuse to start with the named diagnostic.
• Report. Every case is tagged with the checklist ids it proves; a tag may name the targets it counts for, when the item is scoped to some targets. A case may also list the targets that show its behaviour (targets), when servers meet one fact in different ways (a mount mismatch refused at boot on most servers, at the first request on Express and AdonisJS); it then runs only on those, and the report reads it as refused, naming why, on the others. The suite writes a JSON report keyed by id, which the checklist script reads (FR-18). A proof file may also cite a CI job as <target>: ci: <workflow file>#<job id>: the release gate counts it proven when the newest develop run of that job passed.

Amendment 2026-11-06: the release gate reads only the items in the checklist’s 1.0 scope list and only the targets and profiles a tag or a proof names; an item after 1.0 is left out of it. It runs in the release workflow: blocking in a manual release run, which reads the reports of that commit and fails on a gap, a failed cited CI job or a consumer without a report, and report-only (the step may fail without failing the workflow) on a push, a pull request or a merge group, where it reads develop’s newest reports.

Running the suite needs Node.js on the CI machine of the test consumer. The application under test does not: a Java or PHP sample app runs without Node.js, and only the test runner beside it uses Node.

Suite profiles

A profile selects how the suite reads responses and which cases apply. Cases are written once.

Profile	Run by	What it changes
default format	every server target	reads success bodies as bare payloads and errors as problem+json
one per format pack	each target that claims that pack	reads bodies with the pack’s format reader (for example JSON-LD items, Hydra pages and Hydra errors)
hand-written hooks	every server target	hooks are the scripted doubles; no adapter or authorization library is installed (CHK-098)
adapter:<name>	each adapter (nestjs-casl, laravel-spatie)	the adapter answers the decide hook from library state seeded through the control routes; runs the rights, restricted-field, sort-key and scope cases
boot	every server target and adapter	the boot tests

A pack profile reads with the shape the step expects (item, page or error), from the IR operation; it reports a case it refuses by name.

A server target is done when the default, hand-written and boot profiles pass, plus the profile of every format pack it claims (A17). An adapter is done when its own profile and boot pass.

The validator agreement table

The three TypeScript validator targets share a language-neutral JSON table of payloads, each with the expected accept or reject per schema and direction (FR-15). Their generated validators must agree with it exactly, as a CI gate (CHK-080, SM-9). The native targets meet the same expectations through the HTTP suite’s 400 cases. The table holds, besides the cases of ADR 0003: a root-level union input, a per-subtree unknownKeys, an optional property that is not nullable sent as null, a string whose length counts variation selectors (the class-validator gaps), {} against a list and [] against an object, 1.0 for an integer (the PHP decoding cases), {} against { "publishOn": null } for an optional and nullable property, a nine-digit date-time fraction, unknown keys under ignore and keep, and an opaque value.

When a target’s nativeDocs artefact is on, its golden tests also emit the framework’s own OpenAPI document for the reference contract and compare it with seampact’s: operation ids, response codes and schemas must agree for every contract route (CHK-165). The comparison runs whether or not the app also writes seampact’s document (A43), since seampact’s is always computable from the IR. Merged documents of a root config have golden tests of their own: a fixture monorepo of two apps, with an identical shared element written once, a clash refused, and the same clash accepted under a prefix (CHK-171).

Consequences

• A consumer’s diff after generate contains only changes caused by its contract or by a target upgrade.
• Hand edits to generated files are caught at the next generate or check, never silently overwritten.
• Each target carries golden trees per claimed wire format; this is the cost of supporting formats as data.
• One suite proves eight server targets and two adapters; a new target or adapter is accepted by passing it, not by a new test set.
• The control routes exist only in sample apps, never in generated code for real applications.

Traceability

• FRs: FR-14 (pure, deterministic generate), FR-9 and FR-38 (byte-stable IR and naming vectors), FR-15 (agreement table), FR-16 (the undeclared-status test helper is generated and golden-tested), FR-18 (report keyed by checklist id), FR-35 (the suite); amendments A13, A17, A30.
• Checklist items enabled: CHK-144 (code style options), CHK-165 (native docs agree), CHK-171 (merged documents), CHK-156, CHK-158 and CHK-159 (validator and payload cases), CHK-060 (test helper on undeclared responses), CHK-062 and CHK-063 (boot tests and allow-list), CHK-074 (every server target and adapter passes the suite), CHK-080 (validator agreement), CHK-098 (hand-written hooks, no adapter), CHK-099 (exactly one neutral hook per concern, observed through the recorded calls); every server-category item is proven through the suite’s tags.
• Success metrics: SM-3, SM-7, SM-9.