2026-09-10 –, Room 2 (350)
This talk is about the art of forgetting, a design pattern for long-context LLM pipelines where you drop the noise from the context on purpose, and the model calls a retrieval tool to pull back anything it actually needs. You pay for half the tokens, and the model capacity to answer is maintain, sometimes even improved.
I'll walk through the pattern end-to-end in about forty lines of Python, content-type routing (image, code, plain text, etc), query-aware relevance scoring, compressed-store markers, and retrieve-tool wiring.
Then, the part that matters most to anyone actually shipping this: how to test whether forgetting-on-purpose helps your workload.
I'll share a reproducible evaluation recipe, including the naive-baseline control that separates what works and what doesn't, and the checks that catch benchmarks measuring the wrong compressor.
You'll leave with a provider-neutral architectural pattern you can drop into any Python LLM pipeline, and a reusable methodology for deciding whether it's worth it on your own production traces.
Reference implementation: the open-source Headroom library and my benchmark repository with all code and evaluation assets will be public.
What attendees would learn
A concrete architectural pattern, query-aware compression with a lossless retrieval fallback, for cutting long-context LLM cost without degrading quality, paired with a reproducible evaluation methodology so attendees can reproduce the benchmark on their own workload.
Scope
In scope: the pattern itself (query-aware compression + a retrieval tool as safety net); content-type-aware routing across JSON, code, and prose; compressed-store markers and retrieve-tool; evaluation methodology including naive-baseline controls, analysis of compression pipelines, and deployment-path verification.
Target audience
Python engineers, ML engineers, and data scientists building LLM applications, agents, or multi-step pipelines that accumulate long context. Provider-neutral, applies to any provider and open models.
Required background
Comfort with Python and basic LLM API concepts (messages format, tool calling). No ML or compression-algorithm background required. Familiarity with evaluation concepts helps but isn't required.
Presentation approach
The idea is to be conceptual with code-illustrated examples. A simple reference implementation in Python.
Empirical results serve as illustrations of the pattern's properties rather than the headline, the talk would be prepared to stay relevant as model families evolve.
Outline. 30 min
Why long context is expensive and counterproductive, the case for forgetting. 5 min
The pattern: lossy compression + a retrieval tool as the safety net. 5 min
Under the hood: content-type routing, relevance scoring, markers, retrieve-tool wiring. 7 min
The art is in the forgetting. Evaluating a compression pipeline honestly: naive baselines, content type analysis. 8 min
When forgetting helps, when it hurts, open questions. 3 min
Q&A. 2 min
