A paper on arXiv proposes measuring model consistency rather than capability alone

A language model may answer the same prompt well once, then drift on the next run despite an identical question. This variance does not show up clearly in many benchmarks that focus on best-of or average scores. A new preprint deposited on arXiv proposes that output consistency, rather than general capability alone, should serve as an independent metric when evaluating systems.

Titled "Grouping the Stochastic Machine", the paper was submitted by George Andrikopoulos on August 19 in the artificial intelligence category. The author borrows an archery metaphor: capability concerns where the shots land on average, while reliability concerns how tightly grouped the shots are. Applied to models, he terms this clustering "precision".

The difference between a good hit and reliable repetitionThe researcher argues that benchmarking culture often measures the central tendency of results, meaning what a top or average response can achieve. The practical distinction between advanced systems, he contends, may instead lie in output variance across repeated identical prompts. The paper presents this not as an established fact across all models, but as a measurement hypothesis to be tested.

The author proposes running a fixed set of deterministically evaluable tasks multiple times at a constant temperature, then calculating the consistency of the outcome for each task. He stresses that the methodology requires no secondary model to judge responses, as placing a model in the evaluation loop can introduce a fresh layer of variance or circular reasoning.

Error type changes the operational decisionThe paper distinguishes between consistent failures, where results cluster far from the target, and scattered failures with wide dispersion. In the first case, the researcher suggests the issue may respond to adjustments in prompt engineering or operating rules. In the second, the team may need to change the model or sampling strategy. It is a practical division: knowing that a system failed is insufficient, as teams need to know whether it fails the same way every time.

The paper outlines a metric it calls grouping and a harness to measure it, presenting an initial experiment that the author notes was later replicated. According to the abstract, one intervention closed a measured gap from zero out of five to five out of five, whereas a task suite designed from the same rules yielded no additional value because an advanced model already embodied good explicit practices. This is a finding reported by the author from his own experiment, not an independently validated benchmark.

What matters for teams deploying models?The proposal offers no off-the-shelf verdict on any product, but it encourages testing tied to day-to-day operations: repeat the same task, ensure deterministic evaluation, and examine outcome variance before attributing an issue to the user or the tool. For teams building workflows around a language model, this layer of measurement may prove more useful than comparing a single successful snapshot. As for the validity of the metric and its wider applicability, both will require broader testing to substantiate.