Skip to content
Publié le November 6, 2026 · par Coinator Team · 4 min de lecture

Same question, same seed, two reports: what makes AI answers reproducible in forensics

Anyone who puts an AI report into a case file must be able to explain how it came about, and a second analyst must be able to retrace it. We measured why the same question to a language model can yield two different reports, what can be done about it, and where the limit lies. The results are in a new whitepaper you can request from us.

#ai #llm #reproducibility #traceability #forensics #privacy #whitepaper

Language models have become everyday tools in blockchain forensics: they summarise transaction chains, draft reports and suggest next steps. They have one property, though, that sits badly with forensic work. The same question does not reliably produce the same answer. Anyone who puts an AI report into a case file must be able to explain in court or to a client how it came about, and if in doubt, a second expert must be able to retrace it with the same means.

We put this question to the test on Coinator's own AI assistant, with a locally hosted model, and learned more than we expected.

An experiment with a surprising outcome

The starting point seemed simple. A language model generates text piece by piece and chooses at every step among several possible continuations. That choice is random, but it can be switched off: temperature 0 always takes the most likely continuation, and a fixed starting value (the seed) makes any remaining randomness repeatable. Both set, the same question asked twice, twelve minutes apart. The result: two reports that named the same facts but had taken different routes, with different intermediate steps, a different order and an additional side finding in the second run.

The search for the cause led not to randomness but to the arithmetic. An inference server keeps the beginning of a prompt in a cache to save compute. The second run found the beginning of the first one there and processed it in a numerically slightly different way. At a point where two continuations were almost equally likely, the decision tipped. Without the cache, the same prompt returned the same answer three times, word for word.

That settled the first lesson: temperature 0 and a fixed seed are only half the answer. A model becomes repeatable only when the processing of the prompt runs the same way every time.

Five sources of deviation on the way from question to report: the random procedure of text generation, the numerics of processing with cache and batching, changes to the prompt, the data state of the blockchain, and the tool path that amplifies every early deviation.
Five sources of deviation on the way from question to report: the random procedure of text generation, the numerics of processing with cache and batching, changes to the prompt, the data state of the blockchain, and the tool path that amplifies every early deviation.

The limit is called blockchain

With the cache switched off, the first two rounds of two runs were identical to the byte. In the third round they diverged again, this time for a reason no setting can fix. One of the queried addresses was the hot wallet of an exchange, and between the two runs new blocks had been confirmed. Balance and last transaction were different, so the model saw different data and planned differently from there.

That is not noise from the model; it is the blockchain itself. An address that has been dormant for months yields the same report today and in a year. A live address yields it only as long as no new block changes its figures. For forensics this means: repeatability of the text is an achievable but narrow goal. The goal that carries weight is traceability: every statement in the report points to a logged query, and that query can be executed again later and compared.

That is exactly how we built it in Coinator. An audit mode sets temperature, seed and cache so that the model's share of a run is repeatable. Every report carries a log of all tool calls, and a replay executes those calls again later and states per call whether the result is identical or whether the data has changed. In our test, 19 of 22 queries from a report of the previous day were identical the next morning; the three deviations belonged to an address that had been spent overnight.

The verification chain: the run leaves a log with calls, arguments, results and versions; the report cites its evidence per statement; the evidence check marks unsupported statements; the replay executes the calls again later and shows per call whether the result is identical, whether the data has changed or whether a deviation exists. The reviewer decides.
The verification chain: the run leaves a log with calls, arguments, results and versions; the report cites its evidence per statement; the evidence check marks unsupported statements; the replay executes the calls again later and shows per call whether the result is identical, whether the data has changed or whether a deviation exists. The reviewer decides.

And the data?

The second half of the question concerns not the answer but the input. Anyone who types addresses, transactions, case context and possibly the names of victims into the chat of a cloud provider transmits personal data to a processor, frequently to a third country, with retention periods and usage rights that are set out in the contract rather than in the interface. That is no reason to forgo language models. It is a reason to check beforehand what is transmitted and where.

Coinator takes a different route here: the model runs on our own servers, reachable only from within the platform, without internet access and without a provider in between. Case data never leaves the infrastructure, there is no training use and no third-party retention. The model also sees no raw data, only abridged tool results; the complete data stays in the log on the user's side.

The whitepaper

We have summarised the full series of measurements, the five sources of deviation, a verification chain for your own use of AI, the pitfalls of working with chat services and a privacy checklist in a nine-page whitepaper: "Reproducibility of AI answers in blockchain forensics". It is written for forensic analysts, investigators and expert witnesses who already use language models such as GPT or Claude or plan to, and it is product-neutral; Coinator appears only in the final section.

You can request the whitepaper free of charge via our contact form. Use the subject "Whitepaper reproducibility" and we will send it to you as a PDF, in English or in German.

This post is currently only available in the language shown above.