Two Settings Tripled GPT-5.6 Sol's ARC-AGI-3 Score

"How enabling two settings tripled our scores on the ARC-AGI-3 benchmark" is the title OpenAI put on its own post on July 29, 2026, and the figures underneath it are 13.3% rising to 38.3% on the benchmark's public evaluation set.

The two settings are retained reasoning and context compaction, both features of the Responses API. The model on either side of that jump is the same GPT-5.6 Sol, same weights, same checkpoint. What changed is how the evaluation harness handed the model its own history between turns.

That distinction is doing an enormous amount of work, and most of the write-ups repeating "tripled" have skipped it.

Both the 13.3 and the 38.3 came out of OpenAI's own harness

ARC Prize Foundation administers ARC-AGI-3 and publishes the official leaderboard. OpenAI's post is not a leaderboard submission. It is a vendor writing up a configuration finding on its own model, using its own evaluation loop, against the benchmark's publicly downloadable task set.

There is nothing improper about that. Harness bugs are real, they are common in agentic evals, and publishing the fix is more useful than quietly re-running. But it means the baseline is also OpenAI's. A before-number you produced yourself, in a harness you later found to be dropping state, is the easiest kind of number to beat.

Three figures are circulating this week and they are not the same measurement:

I can't fully reconcile the 7.78% with the 13.3% from what's public. They may be different game subsets, different retry policies, or different attempt budgets. If you see someone present 7.78% and 38.3% as a clean five-fold improvement, they've stitched two runs together that OpenAI's post describes separately.

The other headline claim is efficiency: roughly six times fewer tokens to reach the higher score, per the same post. Hold that one for later, because it behaves very differently on your bill than it does in a benchmark table.

Retained reasoning stops the agent re-deriving its plan at every frame

ARC-AGI-3 is interactive. Unlike the grid-pair puzzles of ARC-AGI-1 and 2, the agent is dropped into an unfamiliar game, sees a frame, picks an action from a small discrete set, gets the next frame, and repeats. Progress is measured by levels and games completed. A single task is not one API call. It is hundreds of them, chained.

Which is exactly the shape of workload where dropping reasoning state is catastrophic.

Here is the mechanism, and it's plain Responses API behaviour rather than anything exotic. When a reasoning model responds, it emits reasoning items alongside its output. If you build your next request by hand, appending only the assistant's visible output and your new user turn to a growing array, those reasoning items never go back to the model. Every turn, the model wakes up having forgotten why it did the last thing it did. On a 300-step game loop, the agent spends its budget re-forming the same hypothesis about the game's rules, over and over, instead of testing it.

Retained reasoning means passing those items back. Two supported paths:

• Chaining. Send store: true and reference the prior turn with previous_response_id. OpenAI keeps the reasoning items server-side and threads them for you. Least code.
• Stateless with encrypted reasoning. If you're on zero-data-retention or you don't want turns stored, request include: ["reasoning.encrypted_content"] and pass the returned reasoning item back verbatim in the next request's input array. The content is opaque to you; the model can still read it.

Context compaction is the second half. A long game run will hit the context window, and the naive responses to that are both bad: crash, or truncate from the front and throw away the earliest turns, which on ARC-AGI-3 are the turns where the agent worked out what the game's controls do. Compaction summarises older history while keeping recent turns and the live reasoning chain intact, so the run survives past the window without amnesia at the point where amnesia costs the most.

So the literal reading of "tripled" is this: on one interactive benchmark, with one model, an agent that remembers its own reasoning across turns completes roughly three times as many levels as an agent that doesn't. That is a finding about loop plumbing. It is not a finding about GPT-5.6 Sol's ceiling, because nothing about the model moved.

This is not an OpenAI-specific quirk, either. Anthropic requires thinking blocks to be passed back into Claude's context during tool-use loops for the same reason, and warns that stripping them degrades multi-step performance. Anyone who has hand-rolled an agent loop against a reasoning model has probably shipped this bug.

The public set is not the leaderboard, and 38.3 is not close to human

Three exclusions matter more than the delta.

Held-out tasks. ARC Prize evaluates official results on tasks that are not public, precisely so that public-set familiarity can't inflate scores. OpenAI's 38.3% is on the public evaluation set. Across ARC-AGI's history, public-set scores have run ahead of semi-private ones, sometimes by a wide margin. Until ARC Prize publishes a verified run under its own conditions, 38.3% is a self-reported number on the easier half of the benchmark.

Cost per task. ARC Prize has scored prior versions against a cost ceiling, not just accuracy, and reports dollars per task on its leaderboard. The six-times-fewer-tokens claim is directionally relevant to that, but tokens aren't dollars. Reasoning tokens bill at output rates, cached input prefixes bill at a discount, and retained reasoning increases what you send per request even as it reduces how many requests you need. Without cost-per-task from the same runs, efficiency here is an inference.

The human number. ARC-AGI-3's games are built so that people learn the rules in a couple of minutes and finish them. A model at 38.3% on the public set is still failing most of what an untrained human clears. Tripling from a low base leaves you at a higher low base.

One more thing the number can't tell you: how much of the gain came from retained reasoning versus compaction. They shipped as a pair in the write-up. On a workload with short runs that never approach the window, compaction does nothing, and the honest expectation is that retained reasoning is carrying most of the lift.

Chat apps get nothing from this; long agent loops get most of it

Whether any of this shows up in your own numbers depends almost entirely on turn count.

You will see little or nothing if: your calls are single-turn, you're doing retrieval-augmented question answering, you're using a non-reasoning model, or you already pass previous_response_id and never touched the raw input array. The Agents SDK handles session state for you, so SDK users are largely already in the fixed configuration.

You are a candidate if you built the loop yourself, your agent runs dozens or hundreds of turns, and it interleaves reasoning with tool calls. Browser agents, long code-editing sessions, multi-step data pipelines, game and simulation loops.

Finding out whether your loop is dropping reasoning

1. Look at how you construct the next request. If you append assistant text and tool results to a list and resend the list, reasoning items are being discarded. That's the bug.
2. Turn on chaining first, since it's the smallest change: store: true plus previous_response_id on every subsequent call in a run.
3. If your compliance posture forbids storage, switch to include: ["reasoning.encrypted_content"] and echo the reasoning item back unmodified. Do not attempt to parse, trim, or reorder it.
4. Add compaction only once your runs are long enough to approach the window, and decide explicitly what survives a summarisation pass. The early turns where your agent learned the environment's rules usually should.
5. Measure tokens per completed task, not tokens per call. Retained reasoning raises the per-call input count while cutting the number of calls. Only the per-task figure tells you which effect won on your workload, and that's the axis OpenAI's six-times claim lives on.

Run it over ten real tasks before and after. The direction of the token change is genuinely not obvious in advance: it depends on your prefix stability for cache hits, your average run length, and how often your agent currently gives up and restarts.

If ARC Prize publishes a verified GPT-5.6 Sol run on the held-out set and it lands nearer 13% than 38%, then what these two settings fixed was OpenAI's harness on public games, and the tripling generalises no further than that.