SAN FRANCISCO, October 21, 2026.CTGT, an applied interpretability lab, today released Concordance, an open-source protocol and tool under the Apache 2.0 license that tests whether AI decisions are reproducible and traces where they diverge when they are not. VentureBeat first reported the release.

In its first test, the model was perfectly consistent. With SGLang's deterministic inference mode on, it matched its control in 800 of 800 replies, and across 74 requests it saw more than once, it gave the same answer every time. The agentic compliance pipeline around it still produced four different sets of obligations (32, 31, 33 and 33) from the same banking policy manual across four runs. In one run, the rule to escalate an unresolved conflict to the Head of Compliance was never written. CTGT traced the cause to four lines of upload IDs, regenerated on every upload, inside a 12,525-character prompt. After the fix, five runs committed the same 34 obligations.

The release also documents a harder problem. Replaying the bad request returned the same wrong answer 8 of 8 times, and a consistency check passed it. Repeatable doesn't mean right.

The finding arrives as more institutions ask whether AI systems can be trusted with consequential decisions. A recent Pentagon case showed how an AI-generated report nearly sent U.S. troops toward a Chinese ship. On September 29, six AI developers signed a White House accord committing to independent evaluators, and days later the White House formed a task force with 120 days to recommend the federal government's role. In banking, SR 26-2, the model risk guidance the Federal Reserve, OCC and FDIC issued in April to replace SR 11-7, leaves generative AI out of scope. An evaluation is only as good as the system's ability to reproduce its own decisions. Concordance measures whether a system reaches the same decision twice and, when it doesn't, where the difference comes from.

“Everyone decided AI output unreliability was solved once the model gave the same answer twice. But even if the model is consistent, the system built around it may not be, and that’s where the decisions actually get made,” said Cyril Gorlla, founder and CEO of CTGT. “When a bank can’t reproduce a loan decision or an analyst can’t show how a report reached its conclusion, the problem isn’t that the model isn’t intelligent enough. It’s that nobody measured the system. We built the instrument, and we’re giving it away, because a fix proves one system is clean, while measurement lets every team check its own.”

“Every serious AI evaluation assumes the system under test will make the same decision twice. That assumption is rarely checked. Measuring it in the open gives independent evaluators something they can stand behind,” said Rumman Chowdhury, CEO and Founder, Humane Intelligence PBC, and Founding Director, Independent AI Evaluation Foundation.

Chowdhury launched the Independent AI Evaluation Foundation at the UN General Assembly in September. She served as the U.S. Science Envoy for Artificial Intelligence, led Twitter's Machine Learning Ethics, Transparency and Accountability team, and was named to TIME's inaugural TIME100 AI list.

In 2025, Thinking Machines Lab showed that setting a model's temperature to zero doesn't guarantee identical outputs, because inference-server batching can change the computation. CTGT achieved byte-identical inference at the model level in production, but found that ordinary application details, including identifiers and metadata embedded in prompts, reintroduced variability above the model. Consistent outputs still don't guarantee correct decisions. What matters is whether a system's judgments stay aligned with the policies and source data they are meant to reflect.

Concordance starts with byte-level determinism: whether the same input produces the same output at every step of a pipeline. The protocol sends the same requests many times, records every model call and prompt, and compares each step to find where results first differ and what changed. CTGT's next release extends the work to semantic concordance: whether a system's decisions stay true to its rules even when the wording of the input changes.

The research write-up, results and example runs are at ctgt.ai/research/concordance, and the code is on GitHub. CTGT encourages AI developers and enterprises to run the protocol against their own systems. Concordance is CTGT's second open instrument in under three months, following LineageEval, an Apache 2.0 tool for measuring whether censorship in a Chinese open model survives distillation into American models, which Semafor reported in July.

About CTGT

CTGT is an applied interpretability lab researching the behavior of frontier intelligence in the world's largest institutions. Modeled after research organizations such as Xerox PARC, CTGT builds instruments that make AI behavior measurable, testable and understandable in production. Its open research spans model behavior, knowledge transfer, censorship, determinism and the systems surrounding AI models. CTGT is backed by Gradient Ventures, General Catalyst, Y Combinator and François Chollet, the creator of Keras.

Media contact:ctgt@bambybig.agency