What IBM measured on the AppWorld benchmark
The study runs on AppWorld. Setup matters as much as result here, because the detail that usually goes missing in memory research is what was actually sitting in the context window.
The evaluation, in four numbers
Memory means distilled guidelines, not stored transcripts
The word memory does a lot of unhelpful work in agent discussions, so IBM pins it down. What the agent gets back is a written brief on itself — what worked, what went wrong, which edge cases caught it out — pulled from its own earlier attempts, the failed runs as much as the successful ones, and boiled down into something it can reuse. Nothing in the weights changes. The whole loop runs outside the model, which is why the same pipeline could be pointed at eight different models without retraining any of them.
"It means a guideline set — strategies that worked, mistakes to avoid, and edge cases — distilled from the agent's own prior trajectories." ／ "No model weights are updated. The learning loop changes the guidance available to the agent, not the underlying model — which is exactly why it's cheap to adopt and portable across the eight models we tested." — from the IBM Research postView official source →
Two delivery methods, one agentic memory set
The toolkit is IBM's ALTK-Evolve, and it ran three configurations against each other: no memory at all, the full guideline set injected on every ReAct step, and curated retrieval — a fixed high-confidence core plus a few task-relevant guidelines pulled per task. Both memory runs draw on one and the same pool of guidelines, and that pool was mined only from the training split of AppWorld, never the test split. What varies is delivery. That single design choice is what lets the study attribute differences to dose rather than to one model having better lessons than another.
"Both memory configurations draw from the same guideline set, mined once (via the loop above) from AppWorld's training split only. What changes between them is only how that one set is delivered — the full guideline set injects all of it every step, while curated retrieval delivers a selected subset — never how the guidelines were produced, and no test-split data ever goes into building it." ／ "We evaluated on AppWorld — 585 multi-step tasks (168 test_normal + 417 test_challenge) across 9 simulated apps (calendars, messaging, payments, and so on)." — from the IBM Research postView official source →
The right memory dose depends on the model tier
Three patterns came out of the sweep, and they do not line up with model size.
| Model | Pattern | Baseline TGC / SGC | Best-memory TGC / SGC | Best config | Δ TGC | Δ SGC |
|---|---|---|---|---|---|---|
| gpt-oss-120b (117B MoE) | Weak / selective | 39.9 / 21.4 | 56.0 / 37.5 | curated retrieval | +16.1 | +16.1 |
| DeepSeek-V3.2 (671B MoE) | Strong w/ headroom | 79.8 / 64.3 | 89.3 / 80.4 | full guideline set | +9.5 | +16.1 |
| Claude Opus 4.6 | Strong w/ headroom | 90.5 / 87.5 | 94.6 / 94.6 | full guideline set | +4.1 | +7.1 |
| GPT-5.5 | Strong (near-ceiling) | 92.3 / 82.1 | 95.2 / 89.3 | full guideline set | +2.9 | +7.2 |
| GLM-5 (745B MoE) | Saturated | 87.5 / 80.4 | 87.5 / 80.4 | full guideline set | 0.0 | 0.0 |
Read the last two columns together. The stricter metric usually moves further than the headline one, because guidelines help most with the variant of a scenario that used to fail. DeepSeek gained +9.5pp on task completion and +16.1pp on scenario completion from the same configuration. Even GPT-5.5 and Claude Opus 4.6, already up against the top of the TGC range, added about 7 points on the strict one.
"Figure 1. Representative models in the three observed patterns. Bars show TGC on AppWorld test_normal for baseline vs. the best-memory configuration; the x-axis begins at 40% to make differences visible. TGC alone understates the larger SGC gains — see the SGC columns in the table below." ／ "And the effect doesn't disappear at the top of the range: GPT-5.5 and Opus, both near the ceiling on TGC, still gain +7.2 and +7.1pp SGC respectively." ／ "Agentic memory is not a feature you switch on. It's a dose you calibrate to the model." (the cells in the table above are the values published in that post) — from the IBM Research postView official source →
What a large guideline set does to a weaker model
The headline result belongs to the smallest model in the table. gpt-oss-120b went from 39.9 to 56.0 on task completion with curated retrieval. Hand it everything instead and it improves by less, for roughly half again as many tokens. More context did less for it than a smaller, better-chosen set did. Strong models with room to improve went the other way. DeepSeek-V3.2 did best when it got back every guideline it had mined, rare edge cases included.
"Strong models with headroom want the full guideline set — every guideline, including rare edge-case lessons. They have the capacity to absorb and apply all of it. DeepSeek-V3.2 (671B MoE) climbed +9.5 percentage points in task completion when given its full self-mined guideline set." ／ "Smaller or weaker models get drowned by a large guideline set. For these, a tight, high-confidence core plus a handful of task-relevant guidelines retrieved per task works best. gpt-oss-120b (117B MoE) gained +16.1pp with this selective approach — while the full guideline set gained less and cost ~50% more tokens." — from the IBM Research postView official source →
Saturated is a label for an observation, not an explanation
GLM-5 moved zero points on both metrics, and IBM does not claim to know why. It sets out three possibilities and picks none of them: the model may have little headroom left on this benchmark, the guidance may not touch whatever is still failing, or GLM-5 may just not act on what it is handed. Copy that restraint when you read the row. What got measured is an absence of gain, not a property of the model.
"Already-saturated models show no measurable gain. We call this the saturated pattern — the label describes what we observed, not a proven cause. The model may already have been near its ceiling on these tasks, the guidelines may not have addressed its remaining failures, or it may not have applied the guidance effectively. GLM-5 (745B MoE) sat here in our runs." — from the IBM Research postView official source →
Deciding how much to inject means comparing two prompts that differ only in their guideline block. Lining the two versions up and reading the changed spans is faster than diffing them by eye.
Paying less for agent memory can also score higher
Injecting a guideline set on every step means paying for it on every step, and the measured overhead is large enough to change a budget.
| Model | Config | Tokens/task (baseline) | Tokens/task (+ memory) | Overhead |
|---|---|---|---|---|
| DeepSeek-V3.2 | full guideline set | 148K | 263K | +78% |
| gpt-oss-120b | full guideline set | 110K | 166K | +51% |
| gpt-oss-120b | curated retrieval | 110K | 116K | +5% |
For the weaker model the two axes point the same way: curated retrieval was both the most accurate configuration and the one that barely moved cost. You do not often get to pick the cheap option and the accurate one at the same time.
"A practical concern: injecting a full guideline set inflates every ReAct step's input, because the guidelines are re-sent each turn." ／ "Table 1. Average token use per task, accumulated across agent steps, measured against the no-memory baseline." (the rows above are Table 1) — from the IBM Research postView official source →
Prompt caching is the lever for LLM agent context cost
What you pay for is a bigger input, not a longer run. DeepSeek's trajectories came out the same length either way, around 18 to 19 ReAct steps, so the loop never got heavier. The same text just went over the wire again on every turn. That is what prompt caching exists for, because the static part of the guideline block does not change from one step to the next. IBM's engineering advice follows: park that block where nothing will shift it, and the cache keeps paying off. It is the same lever as context engineering in Claude 5, pulled in the opposite direction from Codex cutting its context window.
"Memory doesn't blow up the reasoning loop. DeepSeek runs about the same number of ReAct steps with memory as without (≈18–19 on average), so the added cost is input-token inflation, not longer trajectories." ／ "Curated retrieval keeps cost near baseline. For weaker models, where selection wins on accuracy, it also wins on cost — the best of both worlds (+16.1pp TGC at only +5% tokens for gpt-oss-120b). Better performance here does not require more inference cost." ／ "The real efficiency lever in production is prompt caching: the static portion of the guideline set is identical across steps and can be cached, cutting effective cost substantially. Cache-aware prompt design — keeping the shared guideline-set prefix stable so it stays cacheable — is worth engineering for." — from the IBM Research postView official source →
What to take from one agent memory benchmark
The practical instruction is short: measure your model with and without its guideline set before deciding how much to inject, and measure the strict metric as well as the headline one, because that is where the gain often sits.
The post states its own limits. It is one benchmark; wider ones and real deployments are still running. Retrieval currently ranks candidates by cosine similarity, and IBM says the ranking is an imperfect predictor of which guideline will actually help on a given task. Context-window size is a suspected factor nobody has isolated yet. And there is a floor: under some level of capability, a model distilling lessons from itself has nothing worth distilling, which IBM treats as a separate problem.
"We have not yet run controlled experiments isolating this factor." ／ "Our current retrieval ranks guidelines by cosine similarity, which we've shown doesn't perfectly predict which guidelines help a given task." ／ "Below a minimum capability baseline, self-distillation lacks signal. Teacher-distilled memory for very weak models is a separate problem we're exploring." ／ "These results are validated on AppWorld — a rigorous multi-step benchmark, but a single one. Broader agent benchmarks and real-world deployments are in progress." — from the IBM Research postView official source →
