ArcadeBench v0

Tech Lab Report 002: ArcadeBench v0

Most evaluations score an AI's output against an opinion: a human rater, a written rubric, a second model acting as judge, or a test that someone had to author. Those instruments are useful, and they are also approximate — the judgment is part of the measurement.

Classic board games allow something rarer. For a small enough game, every legal move in every reachable position carries an exact value, because the game tree can be searched in full. An evaluation built on that foundation has no rubric to argue with and no judge to persuade.

ArcadeBench is a versioned benchmark framework that scores decisions against exact ground truth and reports the computational capacity each decision consumed. Tech Lab Report 001: The Joule Hypothesis established that capacity accounting. ArcadeBench does not define Joules; it uses them as a denominator.

This brief report states the framework, its v0 scoring function, and its first results. The related work this builds on, what transfers to other games, the reproduction commitments, and the effort-allocation result are in the detailed report. Eligibility rules, exclusions, the evidence schema, and per-decision metrics are in Appendix A.

Research Question

Given a state and the available actions, how much value did a player preserve or destroy by the action it chose — and what did that decision cost?

The question is deliberately narrower than "which model reasons best." It is answerable, and the answer is checkable by anyone holding the same match record.

Why The Outcome Signal Is No Longer Sufficient

The motivating observation is that winning has stopped being informative. In the primary dataset — seven complete tournaments, 265 matches, 2,156 scored decisions across 17 model labels — 176 of 265 matches ended in a draw (66.4%), which is the game-theoretically correct result under mutual optimal play. 94.3% of all decisions were optimal.

Tournament standings make the consequence concrete. Of 73 series played, 19 (26.0%) were decided by a coin toss, including one grand final: in v2026.08.20-0x0000, the final between GPT-5.6 Luna and Claude Sonnet 5 reached game 7 of a best-of-7 still level, and the title went to heads.

Move regret does not break that tie either. Two players who both play perfectly both score zero, and that is the correct answer. What distinguishes them is not how they played but what they spent to play that way. Over the same decisions, the fraction of capacity that purchased an optimal decision ranged from 100.0% to 57.3%.

The Primitive: Move Regret

ArcadeBench does not score matches. It scores decisions. For a state s and a chosen action a:

Move Regret(s,a) = V*(s) - Q(s,a)

V*(s) is the value of the best available action and Q(s,a) the value of the action chosen. For Tic-Tac-Toe both come from exhaustive minimax over the complete game tree, so both are exact. Values are +1 for a won position, 0 for a drawn one, and -1 for a lost one, giving regret a range of 0 to 2. A regret of 2 — converting a won position directly into a lost one — is recorded separately as a catastrophic error.

Regret is exact, independent of the opponent, and separates the size of an error from its frequency. Match results have none of those three properties.

The Scoring Function: Efficiency Rating

Efficiency Rating = 1 - (Joules spent on non-optimal moves / total Joules)

Equivalently: the fraction of a player's settled capacity in the match that purchased a zero-regret decision. The rating is bounded to [0, 1] for every match, so it is comparable across matches and models without any population-relative normalization.

Optimal move rate, average regret, and catastrophic error rate remain reported as separate diagnostics. They are deliberately not folded into a single composite score in v0.

What Was Measured

• Game: Tic-Tac-Toe, standard rules. No other game is in scope for v0.
• Matches: 7 bracket tournaments, 73 series, 265 matches, all AI versus AI, exported 2026-08-20.
• Scored decisions: 2,156, across 17 distinct model labels.
• Ground truth: exhaustive minimax, computed independently of the application that recorded the matches.
• Capacity: settled per request from the gateway-issued usage receipt, converted at the joule-capacity-v1 baseline from Report 001.
• Settings: reasoning_effort was minimal on all 2,179 requests. No player had access to a solver, a search tool, or the minimax values used to score it.
• Exclusions: 11 decisions were system fallbacks and are excluded from every quality and capacity figure, because they measure the harness rather than the player.

Results

Efficiency Rating and optimal move rate do not order the field identically. z-ai/glm-5.3 answered 97.7% of its decisions optimally yet ranks below openai/gpt-oss-120b at 90.0%, because its errors landed on expensive decisions. That divergence is the point of weighting by capacity.

Five models answered every decision optimally, so their Efficiency Ratings are identical at 1.0000. Their capacity consumption is not:

Identical measured quality, a 43-fold spread in capacity consumed. This is precisely what the v0 Efficiency Rating does not capture: it measures the fraction of spend that was wasted, and none of these players wasted any. A cost-premium layer that penalizes absolute consumption is deferred to a later scoring version.

The per-model table reports what the instrument produces. It is not a model ranking, and no conclusion about a model author or provider is supported by it.

What This Establishes

Report 001 predicted that results reveal more when the resource budget is held constant or reported alongside them. This is the first test of that prediction against exact labels, and it held in a specific, measurable form:

Across 265 matches on a solved game, the outcome signal collapsed. Two thirds of matches ended in a draw, 94.3% of decisions were optimal, and 26% of series had to be resolved by coin toss — including one grand final. Over the same decisions, capacity-weighted decision quality still separated players across a 42.7 point range, and separated them in a different order than accuracy alone.

Reporting capacity alongside the result recovered discrimination the result alone had lost. That is a narrow confirmation of one of Report 001's four predictions, on one game, and it is the claim this report makes.

A solved game also makes a second question askable: whether capacity was spent where the decision mattered. Across the 2,155 decisions carrying reasoning-token counts, the correlation between exact position difficulty and reasoning effort was −0.044, with a median within-model correlation of −0.036. At minimal reasoning effort, additional inference spent on a decision is not being directed by how much the decision matters. The detailed report states the prediction, its refutation condition, and the robustness checks.

What This Does Not Establish

• One game. Tic-Tac-Toe only, chosen because its labels can be perfect rather than because it is difficult. No measurement here has been repeated in any other game, and v0 specifies no method for estimated values.
• Joules are an accounting policy, not a physical measurement. Capacity is converted from provider-reported usage cost. It is not measured electricity.
• Uneven samples. Per-model scored decisions range from 28 to 352, and a bracket structure gives stronger players more games.
• No human data. Every scored decision was produced by an AI player.
• Retrospective, not behavioral. This report scores records that already existed. It cannot test whether visible capacity changes how a participant behaves, which remains open from Report 001.
• Tic-Tac-Toe is the most memorizable board game there is. A high optimal rate here may reflect recall as much as derivation.
• Reasoning was not scored. Only the action chosen was evaluated. A model's stated justification is preserved verbatim, but the procedure that produced the action is not available, and a stated rationale is an output to evaluate rather than a record of how a system decided.
• The measured system is a model at fixed settings under one harness. Board state was supplied as text in one fixed format, at one reasoning setting. These figures describe models under this harness rather than in general.

Further Detail

• Detailed report — related work, the tie-breaking analysis, what transfers to other games, reproduction commitments, and the effort-allocation result in full.
• Appendix A: Dataset, Method, and Scoring — eligibility rules, exclusions, the full evidence schema, per-decision metrics, and latency measurements.
• Tech Lab Report 001: The Joule Hypothesis — the capacity accounting this report consumes.
• Evaluating Claims About Reasoning — the framework this report is written to satisfy.