GPT-5.6 Sol's ARC-AGI-3 score went from 13.3% to 38.3%
With retained reasoning and compaction turned on, the same GPT-5.6 Sol scored 38.3% on the ARC-AGI-3 public set, up from 13.3% under the official harness, according to OpenAI. The numbers come from a research post by Ilan Bigio and Ted Sanders, which explains step by step how OpenAI got from one to the other.
The post opens with a result OpenAI says puzzled it at first. GPT-5.6 Sol has solved long-open math problems such as the cycle double cover conjecture, and it has beaten Pokémon FireRed with a vision-only harness, Slay the Spire through Codex computer use, and the opening stages of Baba Is You.
On ARC-AGI-3, a collection of 2D puzzle games, OpenAI says the model managed only 7.8%, and GPT-5.5 scored just 0.4%. OpenAI wanted to know whether 2D puzzle games were especially hard for its models or whether something else was wrong.
The post does not say which task set or settings gave the 7.8%. The two figures it puts side by side are 13.3% and 38.3% on the same public set, so the result is not a jump from 7.8% to 38.3%.
The table sets the two harnesses against each other on four points OpenAI describes. The chart after it plots both scores beside OpenAI's estimate for human testers.
The two ARC-AGI-3 harnesses, as OpenAI describes them
| Point | Official ARC-AGI-3 harness | OpenAI's Responses API harness |
|---|---|---|
| Private reasoning | Discarded after every game action | Kept across tool calls and turns |
| Long history | Oldest messages deleted past 175,000 characters | Summarized by compaction, with the limit counted as 175,000 tokens |
| Time spent on each action | The model dwelled a long time on each move | Less time thinking before each action |
| Score on the public set | 13.3% | 38.3% |
GPT-5.6 Sol's ARC-AGI-3 public set scores and the human estimate, as reported by OpenAI (RHAE)
The third bar is OpenAI's estimate for the average human tester, based on official gameplay logs.
The model was GPT-5.6 Sol in both runs, so OpenAI puts the whole gap down to what each harness keeps and what it discards. OpenAI adds that the two settings together used 6x fewer output tokens on the public task set.
"But on ARC-AGI-3, a benchmark of 2D puzzle games, GPT‑5.6 Sol scored just 7.8%, and GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%."／"Were 2D puzzle games unusually difficult for our models? Or was something else going on?"／"In the case of ARC-AGI-3, we discovered that turning on two API settings we use in ChatGPT and Codex—retained reasoning and compaction—tripled scores and cut output tokens by 6x on the public task set."／"With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%." — from the opening of OpenAI's post, July 29, 2026View official source →
OpenAI estimates the average human tester scored 48% on RHAE
From the official gameplay logs, OpenAI estimates the average human tester at 48%, still above GPT-5.6 Sol's 38.3%. ARC-AGI-3 scores are given in RHAE, short for Relative Human Action Efficiency. RHAE measures a model's performance against a human baseline.
The post also lists what the models can and cannot see while playing. They are not told how they will be scored and never see their score during a run, and each action returns only a text version of the current frame and the level they are on.
"Scores measure Relative Human Action Efficiency"／"a metric comparing model performance to a human baseline"／"we estimate the average human tester scored 48%."／"Models are not told how they will be scored, and cannot see their score throughout"／"actions only return a text representation of each frame and what level they are on." — from the paragraph on scoring in OpenAI's postView official source →
ARC-AGI-3 leaderboard game: OpenAI's harness solved all six levels
According to OpenAI's video caption, GPT-5.6 Sol clears all six levels of one ARC-AGI-3 game with OpenAI's harness, while no frontier model on that game's leaderboard gets beyond the first level. Frontier models are the most advanced models from the leading AI labs. The sped-up video shows the official harness on the left and OpenAI's Responses API harness on the right, and the six-level claim rests on that caption alone.
The benchmark is open for anyone to try. OpenAI's post links 25 demo games at arcprize.org/tasks and ends by inviting readers to play the public games and see how they compare with frontier models.
"A sped-up video of GPT‑5.6 Sol attempting to solve puzzles in the ARC-AGI-3 benchmark, with the official harness (left) and our Responses API harness (right), which retains reasoning and enables compaction."／"no frontier model solves any level beyond the first. With our harness, GPT‑5.6 Sol solves all six."／"You can play 25 demo games at arcprize.org/tasks"／"If you want to test your own mettle against frontier models, try the public games yourself at arcprize.org/tasks" — from the video caption, the ARC-AGI-3 section and the last line of OpenAI's postView official source →
Why the standard harness held GPT-5.6 Sol back
Much of GPT-5.6 Sol's confusion on ARC-AGI-3 came from two harness settings, discarded reasoning and rolling truncation, rather than from the model, OpenAI says. Together they help explain why the model had trouble learning as a game went on. The harness is the code around the model that runs each task and decides what stays in the model's view.
OpenAI started digging after reading ARC's analysis of where GPT-5.5 fell short (ARC is the team that runs the benchmark). Some of GPT-5.6 Sol's attempts showed what ARC had described: long stretches on each action with little progress. A closer look turned up the two settings below.
"Inspired by ARC’s analysis of GPT‑5.5’s shortcomings"／"we examined some of the GPT‑5.6 Sol’s attempts. Like ARC, we saw that the model didn’t appear too bright. It dwelled a long time on each action and struggled to make progress."／"But as we looked deeper, we discovered much of the model’s confusion was not inherent to the model itself, but due to settings in the harness."／"Together, these two features of the harness—discarding reasoning and rolling truncation—helped explain why GPT‑5.6 Sol was struggling to learn over time." — from the section on what OpenAI found in the attemptsView official source →
Discarded thinking left the agent relearning the game every turn
The official harness threw away all of the model's private reasoning after every game action, so GPT-5.6 Sol had to figure the game out again on each move. Private reasoning is the thinking a model does before it answers or calls a tool.
Some record did survive: the model could still see a log of its earlier moves and the short notes attached to them. What disappeared were the plans, insights and thoughts behind those moves, so every action started without the reasoning that led to it.
"First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them." — from the first of the two problems OpenAI describesView official source →
Rolling truncation pushed the oldest context out of view
Rolling truncation is the official harness's way of dealing with a long history: once the conversation goes past 175,000 characters, the oldest messages are deleted and early actions fall out of view. So besides losing its past thinking, GPT-5.6 Sol was also losing sight of its past actions.
OpenAI points to two costs. The model loses observations and actions from earlier in the game, and it spends much of each task with a fuller context window (the text the model can see at once), which OpenAI says can slightly hurt performance.
"Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT‑5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too."／"The ARC-AGI-3 harness addresses context limits with rolling truncation. When the conversation context exceeds 175,000 characters, the oldest messages are discarded."／"Rolling truncation has two drawbacks. First, the model loses earlier observations and actions. Second, it spends much of the tasks operating with a fuller context window, which can slightly impair performance." — from the second problem and the note on context limits in OpenAI's postView official source →
The benchmark uses a generic harness on purpose, for fair comparisons
ARC-AGI-3 keeps its harness generic on purpose, with no tools or special features, because, as OpenAI reports it, ARC holds that a simple harness makes model weaknesses easier to spot and comparisons fairer. Commercial developers do the opposite, OpenAI notes, tuning their harnesses to each model's features and quirks. The post closes by thanking ARC for its years of work on evaluating AGI (artificial general intelligence) and for the analysis that prompted this closer look.
"ARC-AGI-3 uses an intentionally generic harness, without tools or special features. ARC’s reasoning was that a simple harness makes model shortcomings more visible and makes model comparisons more fair. Commercial developers, by contrast, optimize harnesses for each model’s features and quirks."／"We are grateful to ARC for their years of creative work on AGI evaluation, and for their analysis that inspired us to take a closer look here." — from the ARC-AGI-3 section and the closing lines of OpenAI's postView official source →
The fix: retained reasoning first, then compaction
OpenAI fixed the problem by making the harness behave the way its models are trained and deployed, with reasoning kept in the history and an overlong conversation summarized and continued. Its models are trained to think in private reasoning messages before replying or calling a tool, and those messages stay in the conversation. The same setup runs in ChatGPT and Codex, according to OpenAI.
"Our models are trained to think with private reasoning messages before they output replies or tool calls. These private thinking messages are retained as part of the conversation history. If a conversation grows too long, we summarize it and continue."／"This is how our models are trained, and also how they are deployed in ChatGPT and Codex." — from the section "Agents do best when they remember what they’ve done"View official source →
Passing the previous response ID kept thinking across turns
OpenAI rebuilt the ARC-AGI-3 harness on its Responses API to match production, so that passing the previous response ID carries GPT-5.6's reasoning across tool calls and turns. The Responses API is the interface developers use to call OpenAI's models. If each new call includes the previous response's ID, the reasoning comes along with no extra work.
Keeping the reasoning changed two things. GPT-5.6 Sol spent less time thinking before each move, because it no longer had to work the game out from scratch every turn. With its earlier thoughts still in view, it also learned far better as a game went on and played with coherent strategies.
"To better match our production setup, we implemented the ARC-AGI-3 harness with our Responses API"／"Our API makes it easy to manage context: for GPT‑5.6, passing the previous response ID automatically retains reasoning across tool calls and turns."／"With reasoning retained, we noticed two big changes. First, GPT‑5.6 Sol spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn. Second, when it was able to remember its past thoughts, GPT‑5.6 Sol was much better at learning over time and employing coherent strategies." — from the section "Agents do best when they remember what they’ve done"View official source →
Compaction replaced rolling truncation and used fewer output tokens
Compaction, a second Responses API setting, took the place of rolling truncation, and OpenAI says it let GPT-5.6 Sol hold on to what it had learned about each game over longer runs while scoring higher with fewer output tokens. Compaction summarizes the conversation so the run can continue, instead of deleting the oldest messages. Together, OpenAI says, the two settings roughly tripled the score of GPT-5.6 Sol (max) while using 6x fewer output tokens.
An animation in the post shows how each harness fills GPT-5.6 Sol's 175K context window as the model works through several puzzles. With more of its past in memory, the model thinks less per action and gets through the puzzles much faster, OpenAI says.
The two runs measure the limit in different units. The official harness cuts off at 175,000 characters and OpenAI's own run at 175,000 tokens, which OpenAI calls quite similar here because most of the text is action grids that its tokenizer turns into tokens one to one.
"The next improvement came from replacing rolling truncation with compaction"／"another setting in the Responses API."／"When we enabled compaction on ARC-AGI-3, GPT‑5.6 Sol was better able to preserve what it had learned about each game across longer runs, and achieved a higher score with fewer output tokens."／"With better memory of its past, GPT‑5.6 Sol thinks less per action and proceeds much faster. Note: our implementation uses a limit of 175,000 tokens instead of characters, but this ends up being quite similar, as the vast majority of text is action grids which are tokenized at a 1:1 ratio by our tokenizer."／"Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens." — from the compaction paragraphs and the animation note in OpenAI's postView official source →
OpenAI advises API developers to copy its production settings
For the best performance, OpenAI tells API developers to use the settings it runs in its own products: the Responses API instead of the legacy Chat Completions API, retained reasoning, and compaction. For comparing models, it recommends evals that use these settings. All the figures here are OpenAI's own, from its write-up.
Evals are the tests used to score and compare models. OpenAI prefers evals run with these settings because they come closest to how its models are used in ChatGPT and Codex.
"If you’re an API developer trying to maximize performance, we recommend using the same settings that we deploy in our own products:"／"And if you’re comparing models, we recommend relying on evals that use the settings above, which best match real-world use in ChatGPT and Codex." — from the section "Conclusion and recommendations"View official source →
Responses API over Chat Completions, plus reasoning and compaction
The first recommendation is about which API to call. Chat Completions is the older interface, now labeled legacy by OpenAI; retained reasoning and compaction are the two settings that moved the ARC-AGI-3 score.
This was not the first time, OpenAI adds: it had once been surprised by low scores on a public benchmark and then found the eval runner used a generic harness that discarded reasoning messages. An eval runner is the program that runs a benchmark's tasks against a model.
"Use our Responses API, not our legacy Chat Completions API"／"Retain reasoning"／"Use compaction"／"This isn’t the first time we’ve been surprised by low scores on a public benchmark and then discovered that the eval runner was using a generic harness that dropped reasoning messages." — from the section "Conclusion and recommendations"View official source →
Benchmark scores also reflect API settings and prompting
Evals rarely measure a model alone, OpenAI concludes, because a score also reflects quieter choices about API settings, harness design and prompting. On ARC-AGI-3, according to OpenAI, two of those choices, both API settings, were enough to triple the same model's score on the public set.
One question is still open. The post never says which task set or settings produced the 7.8% GPT-5.6 Sol score it cites at the start.
NVIDIA AVO Scores 100 on the ARC-AGI-3 Public Set covers a different result on the same benchmark, built on persistent memory. For deciding what Claude Code keeps in view, see Context Engineering: Claude Code's Prompt Cut 80%+.
As OpenAI tells it, an ARC-AGI-3 score measures the harness as much as the model. Before trusting a model's ARC-AGI-3 number, check which harness produced it.
"Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting."／"We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting." — from the opening and the conclusion of OpenAI's postView official source →
FAQ
- Which two settings tripled the ARC-AGI-3 score?
- Retained reasoning and compaction, two Responses API settings that OpenAI already uses in ChatGPT and Codex. Retained reasoning keeps the model's private reasoning in the conversation history, and compaction summarizes a long conversation instead of deleting its oldest messages. OpenAI says turning both on tripled scores and cut output tokens by 6x on the public task set.
- How much did GPT-5.6 Sol's score change?
- On the ARC-AGI-3 public set, OpenAI reports 13.3% with the official harness and 38.3% with retained reasoning and compaction. The 7.8% in the opening of OpenAI's post is given without a task set or settings, so it is not the starting point for the 38.3%.
- What does an ARC-AGI-3 score measure?
- Scores are given in RHAE, short for Relative Human Action Efficiency, a metric that compares a model's performance with a human baseline. Based on official gameplay logs, OpenAI estimates that the average human tester scored 48%, which is above GPT-5.6 Sol's 38.3%.
- What does OpenAI recommend to API developers?
- OpenAI recommends the settings it deploys in its own products: the Responses API instead of the legacy Chat Completions API, retained reasoning, and compaction. For comparing models, it suggests evals that use these settings, which it says best match real-world use in ChatGPT and Codex.
Related articles
GPT-5.6 Sol Ultrafast: Up to 14x Faster OutputGPT-5.6 Sol Ultrafast outputs up to 750 tokens per second on Cerebras, up to 14x faster than Standard. A limited OpenAI API preview since August 13, 2026.
ChatGPT for Academic Researchers: Free for 100,000ChatGPT for Academic Researchers will bring free GPT-5.6, deep research and Codex to 100,000 researchers. Who qualifies, what's included and how to apply.
GPT-5.6 availability by plan: Sol, Terra and LunaGPT-5.6 availability: generally available since July 9, 2026, but Sol, Terra and Luna reach different plans and tools. Here's who gets what and what it costs.
Codex context window cut to 272K: why that numberOpenAI cut the Codex context window from 372K to 272K on July 18, 2026. Above 272K, GPT-5.6 bills the whole request at 2x input. Sessions now compact sooner.