field note·mai 2026·Journal

Non-determinism: why AI changes its answer

Non-determinism isn't a bug to fix — it's an architecture decision. What that imposes on a system in production.

Before wiring an AI into a task, one question decides the architecture: must the result be reproducible? This framework gives the rule, the two cases, and the guardrails. Apply it task by task.

The principle

Even at zero temperature — asking for the single most likely answer and nothing else — an LLM served behind an API doesn't always return the same output for the same input. This isn't a forgotten setting: it's a property of how it's served, and it's something you design around rather than fix.

Thinking Machines Lab — "Defeating Nondeterminism in LLM Inference" (Sept. 2025)

The usual explanation — floating-point rounding plus concurrent execution on GPUs — is incomplete. The root cause is lack of batch invariance: a request's output depends on how many other requests are being served alongside it at the same time. The authors proved it by rewriting the compute kernels: a thousand runs, a thousand identical outputs. Nondeterminism is controllable once you control the inference stack — but behind a third-party API, you don't. thinkingmachines.ai

The rule

For every task, before putting an AI on it, one question:

The question that decides it

Must the result be reproducible and auditable?

The two cases

Yes → deterministic. Calculations, business rules, compliance, amounts, auditable decisions. This gets hard-coded — code, automation. AI can prepare it, it never decides alone.
No → nondeterminism, accepted. Writing, exploration, synthesis, judgment under uncertainty. Variability becomes an asset — provided it's contained.

The trap

Putting an LLM on a deterministic task is the costliest architecture mistake I run into.

Containing nondeterminism (when you keep it)

Constrained output — enforce a strict schema or response format.
Deterministic guardrails — validation, bounds, checks wrapped around the call.
Systematic evaluation — an AI judge that scores outputs against a rubric, to catch drift before production, not in production.

What to remember

Takeaway

The right question isn't "which model?" but "does this task need to be reproducible?" That dividing line — not the model's raw power — is what separates a POC that impresses from a system that holds.