The score tripled without changing the model
It started with something that did not add up. Here is what came of digging into it.
The same model was oddly low on this one benchmark
GPT-5.6 Sol has solved longstanding open problems in mathematics such as the cycle double cover conjecture, and it has beaten Pokémon FireRed. Yet on ARC-AGI-3, a benchmark of 2D puzzle games, it stopped at 7.8% — and GPT-5.5 could barely play at all, at 0.4%.
Are 2D puzzles abnormally hard for a model, or is something else going on? That is the starting point.
OpenAI's premise is blunt: benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting.
13.3% became 38.3%
The investigation found that turning on two API settings already used in ChatGPT and Codex — retained reasoning and compaction — tripled the score on the public task set and cut output tokens by 6x.
In numbers: 13.3% with the standard harness, 38.3% with the two settings on.
The figures need care, though. It is not published which task set or which configuration produced the 7.8% quoted at the top. What the official write-up sets side by side is only the 13.3% and 38.3% pair on the same public task set. "7.8% became 38.3%" is not a reading the source supports.
The two harnesses compared (as published by OpenAI)
| Aspect | Standard (official) harness | Harness rebuilt on the Responses API |
|---|---|---|
| Reasoning | Discarded after every action | Retained across tool calls and turns |
| History | Oldest messages dropped past 175,000 characters | Summarized and folded down (compaction) |
| Score (public task set) | 13.3% | 38.3% |
| Output tokens | — | One sixth |
| Thinking time per action | Long | Short |
GPT-5.6 Sol has solved longstanding open problems in mathematics like the cycle double cover conjecture and beaten games like Pokémon FireRed. But on ARC-AGI-3, a benchmark of 2D puzzle games, GPT-5.6 Sol scored just 7.8%, and GPT-5.5 could barely play the games at all, scoring a paltry 0.4%. / Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting. In the case of ARC-AGI-3, we discovered that turning on two API settings we use in ChatGPT and Codex—retained reasoning and compaction—tripled scores and cut output tokens by 6x on the public task set. / With the official harness, GPT-5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%. — From the initial scores, the effect of the two settings, and the comparison on the public task set
What the score actually measures
This is not an accuracy rate. It is RHAE (Relative Human Action Efficiency), a metric comparing how efficiently the model acts against a human baseline.
Based on official gameplay logs, the average human tester is estimated at 48%. So 38.3% is a level that still falls short of the 48% human average.
The conditions are strict, too. The model is not told how it will be scored and cannot see its score along the way. Actions return only a text representation of each frame and what level it is on.
On one game's official leaderboard, no frontier model gets past the first level. With OpenAI's harness, GPT-5.6 Sol solves all six.
Scores measure Relative Human Action Efficiency (RHAE)—a metric comparing model performance to a human baseline. Based on official gameplay logs, we estimate the average human tester scored 48%. Models are not told how they will be scored, and cannot see their score throughout—actions only return a text representation of each frame and what level they are on. / On the leaderboard for this game, no frontier model solves any level beyond the first. With our harness, GPT-5.6 Sol solves all six. — From the RHAE metric, the human baseline, the information given to models, and the leaderboard note in the opening video caption
What was holding the model back
There were two causes, and both come down to the same design question: what does the model get to keep in mind?
Its thinking was thrown away after every action
The first is discarded reasoning. After each game action, all of the model's private reasoning was gone.
What follows from that: the model was asked to figure out the game anew every single turn. It could still see a record of past moves and brief notes, but not the plans, insights, or thoughts that led to them. The moves remained; the reasons for them vanished.
History was being dropped, oldest first
The second is truncation. The harness used a scheme that pushes out the oldest entries first, so older actions became invisible as the history grew.
So the model could not remember its past thinking, and was losing memory of its past actions too. Put those together, OpenAI argues, and struggling to learn over time is what you would expect.
Watching the attempts, the model dwelled a long time on each action and struggled to make progress. The write-up also notes that ARC's analysis of GPT-5.5's shortcomings is what prompted the closer look.
First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT-5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them. / Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT-5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too. / Together, these two features of the harness—discarding reasoning and rolling truncation—helped explain why GPT-5.6 Sol was struggling to learn over time. / Inspired by ARC's analysis of GPT-5.5's shortcomings, we examined some of the GPT-5.6 Sol's attempts. Like ARC, we saw that the model didn't appear too bright. It dwelled a long time on each action and struggled to make progress. — From the two causes and what prompted the closer look at the attempts
The benchmark side had its reasons too
This is not a one-sided complaint. The write-up presents ARC-AGI-3's intentionally generic harness along with the reasoning behind it.
ARC's position is that a simple harness makes model shortcomings more visible and makes model comparisons more fair. Commercial developers, by contrast, optimize harnesses for each model's features and quirks.
Both positions hold. Do you want to measure raw capability, or performance in deployment? Different goals call for different measurements.
ARC-AGI-3 is a benchmark designed to measure how well AI agents learn and reason. Agents explore unfamiliar 2D games and infer how they work without explicit instructions. / ARC-AGI-3 uses an intentionally generic harness, without tools or special features. ARC's reasoning was that a simple harness makes model shortcomings more visible and makes model comparisons more fair. Commercial developers, by contrast, optimize harnesses for each model's features and quirks. — From the benchmark's purpose and the reason it uses a generic harness
The fix: from discarding to folding down
The response came in two stages. First keep the thinking, then change how history is handled.
Keeping the reasoning made it think less
OpenAI's models are trained to think with private reasoning messages before producing replies or tool calls. Those thinking messages are retained as part of the conversation history, and when a conversation grows too long it is summarized and continued. That is how ChatGPT and Codex run.
To match the production setup, the ARC-AGI-3 harness was reimplemented on the Responses API. For GPT-5.6, passing the previous response ID automatically retains reasoning across tool calls and turns.
Two changes followed. First, it spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn. Second, it became much better at learning over time and employing coherent strategies.
Less thinking, better results. That combination captures the nature of the fix: nothing made the model smarter — it just stopped being forced to re-interpret everything.
The 175,000 limit was folded, not cut
Next came truncation. The standard harness discarded the oldest messages once the conversation context exceeded 175,000 characters.
Two drawbacks are named. First, the model loses earlier observations and actions. Second, it spends much of the task operating with a fuller context window, which can slightly impair performance.
So truncation was replaced with compaction, another Responses API setting. With it enabled, GPT-5.6 Sol preserved what it had learned about each game across longer runs, and reached a higher score with fewer output tokens.
It is worth knowing how much context your own implementation carries. A character count will tell you how large your prompts and histories have grown.
Our models are trained to think with private reasoning messages before they output replies or tool calls. These private thinking messages are retained as part of the conversation history. If a conversation grows too long, we summarize it and continue. / Our API makes it easy to manage context: for GPT-5.6, passing the previous response ID automatically retains reasoning across tool calls and turns. / With reasoning retained, we noticed two big changes. First, GPT-5.6 Sol spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn. Second, when it was able to remember its past thoughts, GPT-5.6 Sol was much better at learning over time and employing coherent strategies. / The ARC-AGI-3 harness addresses context limits with rolling truncation. When the conversation context exceeds 175,000 characters, the oldest messages are discarded. / Rolling truncation has two drawbacks. First, the model loses earlier observations and actions. Second, it spends much of the tasks operating with a fuller context window, which can slightly impair performance. / The next improvement came from replacing rolling truncation with compaction, another setting in the Responses API. / When we enabled compaction on ARC-AGI-3, GPT-5.6 Sol was better able to preserve what it had learned about each game across longer runs, and achieved a higher score with fewer output tokens. — From how reasoning is retained, why truncation was replaced with compaction, and the effect of the switch
The three things OpenAI recommends
The conclusion returns to the premise: evals rarely measure models in isolation. They measure a bundle of less visible choices about API settings, harness design, and prompting. And, the write-up notes, this is not the first time a surprising low score on a public benchmark turned out to come from an eval runner using a generic harness that dropped reasoning messages.
From there, three recommendations for API developers chasing performance: use the Responses API rather than the legacy Chat Completions API, retain reasoning, and use compaction. When comparing models, it suggests relying on evaluations that use these settings.
Near the end, the write-up thanks ARC — for years of work on AGI evaluation, and for the analysis that prompted this investigation.
We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. This isn't the first time we've been surprised by low scores on a public benchmark and then discovered that the eval runner was using a generic harness that dropped reasoning messages. / If you're an API developer trying to maximize performance, we recommend using the same settings that we deploy in our own products: / Use our Responses API, not our legacy Chat Completions API / Retain reasoning / Use compaction / We are grateful to ARC for their years of creative work on AGI evaluation, and for their analysis that inspired us to take a closer look here. — From the caution about what evals measure and the three recommendations for API developers
Conclusion: a benchmark number is model plus plumbing
What to take from this is not that a score tripled. It is that the same model lands on 13.3% or 38.3% depending on how it is set up.
And what worked was not clever engineering. Do not throw away what it thought. Fold old history instead of cutting it. That is all. Which also means plenty of implementations may be tying a model's hands without knowing it.
The same word — harness — sat at the center of Anthropic's disclosure of real-world breaches during evaluation. There, a weak harness caused an incident; here, a weak harness hid performance. In how Google set up AI to fix Chrome vulnerabilities, what raised accuracy was the surrounding build-out rather than a model swap.
In practice this is continuous with context engineering. What to keep, what to fold, what to drop. Redirecting a slice of the effort spent on model selection and price comparisons toward this design is well worth it.
