ARC-AGI-3のスコアがOpenAIのハーネス設定2つで3倍になった

"GPT‑5.6 Sol has solved longstanding open problems in mathematics like the cycle double cover conjecture" ／ "and beaten games like Pokémon FireRed. But on ARC-AGI-3, a benchmark of 2D puzzle games, GPT‑5.6 Sol scored just 7.8%, and GPT‑5.5 could barely play the games at all, scoring a paltry 0.4%." ／ "Were 2D puzzle games unusually difficult for our models? Or was something else going on?" ／ "Benchmarks rarely measure AI models in isolation. They also measure less visible choices about API settings, harness design, and prompting." — OpenAI 研究発表（2026年7月29日）の冒頭の段落より

"ARC-AGI-3 is a benchmark designed to measure how well AI agents learn and reason. Agents explore unfamiliar 2D games and infer how they work without explicit instructions." ／ "You can play 25 demo games at arcprize.org/tasks" — OpenAI 研究発表「ARC-AGI-3」の節より

"With the official harness, GPT‑5.6 Sol scored 13.3% on the ARC-AGI-3 public set. With retained reasoning and compaction, it scored 38.3%." ／ "tripled scores and cut output tokens by 6x on the public task set" ／ "It dwelled a long time on each action" ／ "all private reasoning was discarded" ／ "When the conversation context exceeds 175,000 characters, the oldest messages are discarded." ／ "spent less time thinking before each action" — OpenAI 研究発表の冒頭の段落と各節より

"Scores measure Relative Human Action Efficiency" ／ "a metric comparing model performance to a human baseline" ／ "we estimate the average human tester scored 48%" ／ "Models are not told how they will be scored, and cannot see their score throughout" ／ "actions only return a text representation of each frame and what level they are on" ／ "A sped-up video of GPT‑5.6 Sol attempting to solve puzzles in the ARC-AGI-3 benchmark, with the official harness (left) and our Responses API harness (right), which retains reasoning and enables compaction." ／ "no frontier model solves any level beyond the first. With our harness, GPT‑5.6 Sol solves all six." — OpenAI 研究発表の冒頭の動画の説明と段落より

"Inspired by ARC’s analysis of GPT‑5.5’s shortcomings" ／ "It dwelled a long time on each action and struggled to make progress." ／ "much of the model’s confusion was not inherent to the model itself, but due to settings in the harness" ／ "Together, these two features of the harness—discarding reasoning and rolling truncation—helped explain why GPT‑5.6 Sol was struggling to learn over time." — OpenAI 研究発表でハーネスの問題を調べた段落より

"First, we noticed that after each game action, all private reasoning was discarded. This meant that with each action, GPT‑5.6 Sol was asked to figure out the game anew, unable to remember its past thinking. The model could still see a record of past moves and brief accompanying notes, but it could not see the plans, insights, or thoughts that led to them." — OpenAI 研究発表で1つ目の問題を述べた段落より

"Second, we saw that the harness used a rolling truncation window, causing older actions to become invisible as the history grew. So not only was GPT‑5.6 Sol unable to remember its past thinking, it was losing memory of its past actions too." ／ "The ARC-AGI-3 harness addresses context limits with rolling truncation. When the conversation context exceeds 175,000 characters, the oldest messages are discarded." — OpenAI 研究発表で2つ目の問題と切り捨ての上限を述べた段落より

"ARC-AGI-3 uses an intentionally generic harness, without tools or special features. ARC’s reasoning was that a simple harness makes model shortcomings more visible and makes model comparisons more fair. Commercial developers, by contrast, optimize harnesses for each model’s features and quirks." — OpenAI 研究発表「ARC-AGI-3」の節より

"To better match our production setup, we implemented the ARC-AGI-3 harness with our Responses API" ／ "The next improvement came from replacing rolling truncation with compaction" — OpenAI 研究発表「Agents do best when they remember what they’ve done」の節より

"Our models are trained to think with private reasoning messages before they output replies or tool calls. These private thinking messages are retained as part of the conversation history. If a conversation grows too long, we summarize it and continue." ／ "This is how our models are trained, and also how they are deployed in ChatGPT and Codex." ／ "Our API makes it easy to manage context: for GPT‑5.6, passing the previous response ID automatically retains reasoning across tool calls and turns." — OpenAI 研究発表「Agents do best when they remember what they’ve done」の節より

"another setting in the Responses API" ／ "Rolling truncation has two drawbacks. First, the model loses earlier observations and actions. Second, it spends much of the tasks operating with a fuller context window, which can slightly impair performance." ／ "When we enabled compaction on ARC-AGI-3, GPT‑5.6 Sol was better able to preserve what it had learned about each game across longer runs, and achieved a higher score with fewer output tokens." ／ "When the conversation context exceeds 175,000 characters, the oldest messages are discarded." — OpenAI 研究発表でコンパクションへの置き換えを述べた段落より

"With reasoning retained, we noticed two big changes. First, GPT‑5.6 Sol spent less time thinking before each action, because it no longer had to interpret the game from scratch every turn. Second, when it was able to remember its past thoughts, GPT‑5.6 Sol was much better at learning over time and employing coherent strategies." ／ "Together, retaining reasoning and compaction allow GPT‑5.6 Sol (max) to achieve roughly 3x the score with 6x fewer output tokens." ／ "Note: our implementation uses a limit of 175,000 tokens instead of characters, but this ends up being quite similar, as the vast majority of text is action grids which are tokenized at a 1:1 ratio by our tokenizer." — OpenAI 研究発表で推論の保持とコンパクションの効果を述べた段落より

"We hope these experiments serve as a reminder that evals rarely measure models in isolation—they also measure a bundle of less visible choices about API settings, harness design, and prompting. This isn’t the first time we’ve been surprised by low scores on a public benchmark and then discovered that the eval runner was using a generic harness that dropped reasoning messages." — OpenAI 研究発表「Conclusion and recommendations」の節より

"If you’re an API developer trying to maximize performance, we recommend using the same settings that we deploy in our own products:" ／ "Use our Responses API, not our legacy Chat Completions API" ／ "Retain reasoning" ／ "Use compaction" — OpenAI 研究発表「Conclusion and recommendations」の節より

"And if you’re comparing models, we recommend relying on evals that use the settings above, which best match real-world use in ChatGPT and Codex." ／ "We are grateful to ARC for their years of creative work on AGI evaluation, and for their analysis that inspired us to take a closer look here." ／ "If you want to test your own mettle against frontier models, try the public games yourself at arcprize.org/tasks" — OpenAI 研究発表「Conclusion and recommendations」の節より