IBM's AI agent memory study: eight models on the AppWorld benchmark
IBM Research published a study on August 18, 2026 that tested eight models on the AppWorld benchmark with and without memory. The question was how much memory an agent should carry, and the answer depended on the model.
AppWorld is a set of 585 multi-step tasks: 168 test_normal and 417 test_challenge. They run across 9 simulated apps such as calendars, messaging and payments. All gains are percentage points on the test_normal tasks, as IBM reports them.
AppWorld reports two scores. TGC (task goal completion) counts finished tasks. SGC (scenario goal completion) is stricter: every variant of a scenario must pass.
AI agent memory means agent guidelines distilled from past runs
In this study, the memory is a guideline set: strategies that worked, mistakes to avoid and edge cases, written down from the agent's own earlier runs. Failed runs count as well as successful ones, and the agent receives these guidelines rather than a replay of past chats.
No model weights are updated. The learning loop changes only the guidance placed in front of the agent, so IBM could use one pipeline on all eight models without retraining any of them.
"It means a guideline set — strategies that worked, mistakes to avoid, and edge cases — distilled from the agent's own prior trajectories." ／ "No model weights are updated. The learning loop changes the guidance available to the agent, not the underlying model — which is exactly why it's cheap to adopt and portable across the eight models we tested." — from the IBM Research postView official source →
ALTK-Evolve test: full agent guidelines versus curated retrieval
The study compared three setups: no memory as the baseline, the full guideline set injected on every step, and curated retrieval. The toolkit behind the guidelines is IBM's ALTK-Evolve, and a step here is one turn of the agent's ReAct loop of thinking, acting and observing. Curated retrieval hands the agent a tight, high-confidence core plus a few guidelines picked for the task at hand.
Both memory configurations draw on one guideline set, mined once from the AppWorld training split. No test-split data goes into building it. Delivery is therefore the only variable that changes between those two runs.
"Both memory configurations draw from the same guideline set, mined once (via the loop above) from AppWorld's training split only. What changes between them is only how that one set is delivered — the full guideline set injects all of it every step, while curated retrieval delivers a selected subset — never how the guidelines were produced, and no test-split data ever goes into building it." ／ "We evaluated on AppWorld — 585 multi-step tasks (168 test_normal + 417 test_challenge) across 9 simulated apps (calendars, messaging, payments, and so on)." — from the IBM Research postView official source →
Agentic memory results: GPT-5.5, Claude Opus 4.6 and DeepSeek-V3.2
IBM's headline finding is that agentic memory is a dose: the best amount differed from one model to the next, and the full guideline set was not always the best. The table below lists the five models whose scores IBM publishes, with baseline and best-memory scores for both metrics. Curated retrieval was the best configuration for gpt-oss-120b (117B MoE), and the full guideline set was the best-memory configuration for the other four, including DeepSeek-V3.2 (671B MoE) and GLM-5 (745B MoE).
| Model | Baseline TGC / SGC | Best memory TGC / SGC | Gain TGC / SGC |
|---|---|---|---|
| gpt-oss-120b | 39.9 / 21.4 | 56.0 / 37.5 | +16.1 / +16.1 |
| DeepSeek-V3.2 | 79.8 / 64.3 | 89.3 / 80.4 | +9.5 / +16.1 |
| Claude Opus 4.6 | 90.5 / 87.5 | 94.6 / 94.6 | +4.1 / +7.1 |
| GPT-5.5 | 92.3 / 82.1 | 95.2 / 89.3 | +2.9 / +7.2 |
| GLM-5 | 87.5 / 80.4 | 87.5 / 80.4 | 0.0 / 0.0 |
IBM describes three patterns: strong models with headroom want the full guideline set, weaker ones want a selected subset, and saturated ones show no measurable gain. The SGC gain is larger than the TGC gain for three of the five models in the table: DeepSeek-V3.2, Claude Opus 4.6 and GPT-5.5. GPT-5.5 and Claude Opus 4.6 already score above 90 on TGC, yet they still gain +7.2 and +7.1 points on SGC.
"Figure 1. Representative models in the three observed patterns. Bars show TGC on AppWorld test_normal for baseline vs. the best-memory configuration; the x-axis begins at 40% to make differences visible. TGC alone understates the larger SGC gains — see the SGC columns in the table below." ／ "And the effect doesn't disappear at the top of the range: GPT-5.5 and Opus, both near the ceiling on TGC, still gain +7.2 and +7.1pp SGC respectively." ／ "Agentic memory is not a feature you switch on. It's a dose you calibrate to the model." (the cells in the table above are the values published in that post) — from the IBM Research postView official source →
DeepSeek-V3.2 and gpt-oss-120b want different amounts of memory
DeepSeek-V3.2 did best with every guideline it had mined, while gpt-oss-120b did best with a small selected subset. DeepSeek-V3.2 received rare edge-case lessons too, and its task completion rose from 79.8 to 89.3, a gain of +9.5 points. IBM says models like this one have the capacity to absorb and apply the whole set.
gpt-oss-120b went the other way: with curated retrieval its task completion rose from 39.9 to 56.0, a gain of +16.1 points. The full set gained less for this model and cost about 50% more tokens. IBM says smaller or weaker models get drowned by a large guideline set.
"Strong models with headroom want the full guideline set — every guideline, including rare edge-case lessons. They have the capacity to absorb and apply all of it. DeepSeek-V3.2 (671B MoE) climbed +9.5 percentage points in task completion when given its full self-mined guideline set." ／ "Smaller or weaker models get drowned by a large guideline set. For these, a tight, high-confidence core plus a handful of task-relevant guidelines retrieved per task works best. gpt-oss-120b (117B MoE) gained +16.1pp with this selective approach — while the full guideline set gained less and cost ~50% more tokens." — from the IBM Research postView official source →
GLM-5 and Claude Opus 4.6: no gain for one, +4.1 points for the other
GLM-5 gained nothing from memory, while Claude Opus 4.6 gained 4.1 points of task completion. GLM-5 scored 87.5 on task completion and 80.4 on scenario completion both without memory and with it.
IBM calls this the saturated pattern and adds that the label describes what it observed, not a proven cause. IBM offers three possible explanations: the model was already close to the best score it could reach on these tasks, the guidelines did not address its remaining failures, or it did not apply the guidance well.
Claude Opus 4.6 started high and still moved: the full guideline set took it from 90.5 to 94.6 on task completion and from 87.5 to 94.6 on scenario completion. IBM states that parameter count alone does not put a model in one pattern or another, and GLM-5, at 745B parameters, sat in the saturated one.
"Already-saturated models show no measurable gain. We call this the saturated pattern — the label describes what we observed, not a proven cause. The model may already have been near its ceiling on these tasks, the guidelines may not have addressed its remaining failures, or it may not have applied the guidance effectively. GLM-5 (745B MoE) sat here in our runs." ／ "What puts a model into one pattern rather than another isn't simply parameter count. Benchmark headroom, context-window size, architecture, guideline quality, and task distribution all appear to shape where a model lands, and separating those factors is ongoing work." — from the IBM Research postView official source →
Prompt caching for agent guidelines: full set adds 51–78% tokens
Sending the full guideline set on every step raised DeepSeek-V3.2's tokens per task by 78%, and IBM names prompt caching as the way to cut that cost in production. The guidelines are re-sent on each turn, so the input of every step grows.
| Model and delivery | Baseline | With memory | Overhead |
|---|---|---|---|
| DeepSeek-V3.2, full set | 148K | 263K | +78% |
| gpt-oss-120b, full set | 110K | 166K | +51% |
| gpt-oss-120b, curated retrieval | 110K | 116K | +5% |
For gpt-oss-120b, curated retrieval was the most accurate configuration and also the cheapest of the two memory runs: +16.1 points of task completion at only +5% tokens. The figures in the table are average tokens per task, accumulated across agent steps. DeepSeek-V3.2 took about 18 to 19 ReAct steps on average with or without memory, so its runs did not get longer and the added bill is input-token inflation.
The static part of the guideline set is identical across steps, which means it can be cached. IBM recommends designing prompts so the shared guideline prefix stays stable and stays cacheable. The post gives no cached-cost figure, only the statement that caching cuts effective cost substantially.
Two related reads look at context from the product side. Context Engineering: Claude Code's Prompt Cut 80%+ covers a trimmed system prompt, and Codex context window cut to 272K: why that number covers a smaller window.
"A practical concern: injecting a full guideline set inflates every ReAct step's input, because the guidelines are re-sent each turn." ／ "Memory doesn't blow up the reasoning loop. DeepSeek runs about the same number of ReAct steps with memory as without (≈18–19 on average), so the added cost is input-token inflation, not longer trajectories." ／ "Curated retrieval keeps cost near baseline. For weaker models, where selection wins on accuracy, it also wins on cost — the best of both worlds (+16.1pp TGC at only +5% tokens for gpt-oss-120b). Better performance here does not require more inference cost." ／ "The real efficiency lever in production is prompt caching: the static portion of the guideline set is identical across steps and can be cached, cutting effective cost substantially. Cache-aware prompt design — keeping the shared guideline-set prefix stable so it stays cacheable — is worth engineering for." ／ "Table 1. Average token use per task, accumulated across agent steps, measured against the no-memory baseline." (the rows above are Table 1) — from the IBM Research postView official source →
What the single AppWorld agent benchmark does not settle
IBM validated the results on AppWorld alone, and the post lists what is still open. AppWorld is, in IBM's words, a rigorous multi-step benchmark, but a single one. Broader agent benchmarks and real-world deployments are in progress, and the post gives no dates for them.
The post names three more limits. Context-window size is a suspected factor in which pattern a model lands in, but IBM has not yet run controlled experiments that isolate it.
Retrieval ranks guidelines by cosine similarity, a measure of how alike two texts are, and IBM says the ranking does not perfectly predict which of them help a given task. Below a minimum capability level, distilling guidelines from a model's own runs lacks signal, and IBM calls teacher-distilled memory for very weak models a separate problem it is exploring.
In this experiment, the best amount of agentic memory differed by model and the full guideline set was not always the best. For gpt-oss-120b, the cheapest delivery was also the most accurate.
"We have not yet run controlled experiments isolating this factor." ／ "Our current retrieval ranks guidelines by cosine similarity, which we've shown doesn't perfectly predict which guidelines help a given task." ／ "Below a minimum capability baseline, self-distillation lacks signal. Teacher-distilled memory for very weak models is a separate problem we're exploring." ／ "These results are validated on AppWorld — a rigorous multi-step benchmark, but a single one. Broader agent benchmarks and real-world deployments are in progress." — from the IBM Research postView official source →
FAQ
- What does memory mean in this study?
- Not a replayed transcript. It is a set of behavioral guidelines that the agent distilled from its own earlier runs, both the successful and the failed ones, and receives again at inference time. No model weights change, which is why the same method carried across all eight models tested.
- Does a bigger model always want more memory?
- No, and parameter count alone does not decide it. IBM lists benchmark headroom, context-window size, architecture, guideline quality, and task distribution as factors that appear to matter, and says separating them is still open work. GLM-5 at 745B parameters gained nothing, while gpt-oss-120b gained the most task completion among the models in IBM's table.
- How expensive is it to run an agent with memory?
- It depends on how the guidelines are delivered. Injecting the whole guideline set on every step raised tokens per task by 78% for DeepSeek-V3.2 and 51% for gpt-oss-120b. For gpt-oss-120b, curated retrieval cost 5% more than the no-memory baseline. The overhead is input inflation rather than longer runs, and IBM names prompt caching as the way to cut that cost in production.
Related articles
Claude pulled Georgia election data unpromptedAnthropic's Claude Mythos 5 requested an access token on its own and pulled aggregate election counts from Georgia's Data Hub. The state says it is free.
Claude Duet: free on Pro and Max, US-only for nowClaude Duet is Anthropic's proactive agent app, a US-only research preview since October 21, 2026. It's free on Pro and Max but draws on your usage limits.
Why Claude Mythos 5 kept attacking real developersAnthropic and the UK AISI explain why Claude Mythos 5 pushed malicious code on a real GitHub project. Told it faced real people, it stopped 5% of the time.
Claude Platform adds OpenTelemetry agent tracingClaude Platform now exports OpenTelemetry traces in public beta. Managed Agents, the Messages API and SDKs send them to Datadog, Langfuse or any OTLP tool.