We enabled deterministic inference on the model behind our policy-ingestion pipeline. 800 of 800 test replies matched a single-request control exactly. We then ran one synthetic banking policy manual through the pipeline four times, resulting in commitment of four different sets of obligations. One run even dropped the rule that escalates an unresolved conflict to the Head of Compliance.

Across all four runs, the model returned the same answer to the same request. The variation came from the pipeline around it, which built different requests from the same document. Any application that writes upload IDs, unordered collections or shared state into its prompts carries the same exposure, regardless of deterministic inferencing. Concordance is the instrument we built to root cause this type of issue, and we are releasing it under Apache 2.0.

What Concordance measures

Concordance tests whether an AI system reaches the same decision when it is given the same work twice. When it does not, it finds the first point where the runs part. It works from the requests and responses alone, so it is model agnostic and does not depend on access to weights. It measures two layers:

• Inference. Does the server return the same text for the same prompt, whatever else is in the batch? Eight fixed prompts per round, shuffled, at concurrency 40, for 100 rounds, compared with a batch size 1 control.
• Pipeline. Does running the same document through the whole application commit the same result? Five runs of one document, each in a fresh organization, with every model call captured. Identical requests are grouped across runs, and the comparison reports the first stage where runs differ and whether the model there received a different request or gave a different answer to the same one.

Model and stack

Qwen3.8-27B (open weights) served with SGLang 0.5.19.dev20260825, deterministic inference enabled (--enable-deterministic-inference), tensor parallel 2, Triton attention. The inference benchmark ran with prefix caching off; the production pipeline ran with it on. The deterministic mode is SGLang's,[[fn: LMSYS]] built on Thinking Machines Lab's work on batch invariance.[[fn: Thinking Machines]] We enabled it and ran SGLang's own verification test before measuring.

Results: inference

With deterministic mode on, Qwen3.8-27B matched its control in 800 of 800 requests over the first 1,000 tokens. Two speculative-decoding configurations run without deterministic mode, Qwen3.8-27B with DFLASH and gpt-oss-120b with EAGLE3, diverged in every request. Deterministic speculative decoding has been proposed separately,[[fn: arXiv:2601.17768]] though we have not tested it.

Results: pipeline

Before the fix, four runs of the same manual committed 32, 31, 33 and 33 obligations, and each run produced a different result. Across those runs, 74 model requests repeated exactly, and the model gave the same answer every time. Each changed answer followed a changed request.

‍

In the request where the escalation was lost, the only difference between run 2 and run 3 was four lines of asset IDs in a 12,525-character prompt. Given that request, run 3 read five conflict-resolution rules as background instead of as rules, and it never wrote the obligation to escalate, whose source text ends "Not resolved. Escalate to Head of Compliance."

‍

Why four lines of IDs changed a decision

The asset IDs, hexadecimal hash strings regenerated on every upload, were irrelevant to the model's task. In run 3, they were the only bytes that differed from run 2. Because the server was deterministic, with those four lines changed and everything else fixed, the model read the same five rules differently.

The sensitivity itself is well documented. Changing only the formatting of a prompt has moved accuracy by up to 76 points on LLaMA-2-13B,[[fn: arXiv:2310.11324]] adding a single space changed more than 500 of 11,000 ChatGPT labels,[[fn: arXiv:2401.03729]] and irrelevant sentences added to math problems sharply cut accuracy.[[fn: arXiv:2302.00093]] Practitioners replacing UUIDs with short integers in prompts have reported roughly eight times fewer errors on an aggregation task.[[fn: BoundaryML]] Sensitivity is highest where a model is least confident,[[fn: arXiv:2410.12405]] and deciding whether a clause states a rule or describes its context is similar in kind.

Those studies compare prompts across sampled outputs. Deterministic serving removes the sampling, which is what let us attribute one lost obligation to one changed span of text in a production pipeline. Exactly how the IDs shifted the model's reading of the rules remains untested.

The fix

1. Identifiers in prompts. Asset IDs were a SHA-256 hash of a per-upload UUID and appeared in 53 of 459 model calls. Prompts now carry short stable aliases.
2. Unordered serialization. A 30-item list was written out in a different order on each run; it is now sorted.
3. Shared state. A topic registry grew between runs, from 21 keys to 22. Each run now starts in a fresh organization.

After the fix, five runs committed the same 34 obligations, with 89 repeated requests and no changed answers.

‍

Where reproducibility stops

Before the fix, we replayed run 3's exact request eight times and got the same wrong answer eight times. Our consistency check missed it, because the misread rules agreed with one another. Pinning the inputs can turn an error that shows up in one run of four into an error that shows up in every run.

Reproducibility lets a team investigate a bad decision after the fact. Whether the decision followed the rules is what we measure next with semantic concordance: whether a system's decisions stay faithful to its rules when the wording of the input changes.

Limitations

• This behavior was empirically observed on one document, five runs, and one pipeline. This is a first measurement.
• "Identical" holds after normalizing per-upload IDs in stored objects. Parse and interpretation outputs are byte-identical without it.
• Ingestion ran at temperature 1.0 with prefix caching on, and replays were stable at both 1.0 and 0.0. The client sent no seed, and SGLang's deterministic mode seeds uniformly so identical requests with identical logits produce identical tokens.
• The server exposed no metrics endpoint, so we could not confirm that every batch reached concurrency 40.

Run it

The protocol, both harnesses, a toy pipeline that reproduces the failure in about a minute, and the run data are at github.com/CTGT-Inc/concordance. If you run it on your own pipeline, we would like to hear what you find.

Citation

Cyril Gorlla, Andrew Noonan, CTGT Engineering, "Deterministic Inference Did Not Make Our Pipeline Deterministic", CTGT Research, October 2026.

BibTeX:

@article{ctgt2026concordance,
 author  = {Gorlla, Cyril and Noonan, Andrew and {CTGT Engineering}},
 title   = {Deterministic Inference Did Not Make Our Pipeline Deterministic},
 journal = {CTGT Research},
 year    = {2026},
 month   = {oct},
 note    = {https://www.ctgt.ai/research/concordance}
}