This talk is about the art of forgetting, a design pattern for long-context LLM pipelines where you drop the noise from the context on purpose, and the model calls a retrieval tool to pull back anything it actually needs. You pay for half the tokens, and the model capacity to answer is maintain, sometimes even improved.
I'll walk through the pattern end-to-end in about forty lines of Python, content-type routing (image, code, plain text, etc), query-aware relevance scoring, compressed-store markers, and retrieve-tool wiring.
Then, the part that matters most to anyone actually shipping this: how to test whether forgetting-on-purpose helps your workload.
I'll share a reproducible evaluation recipe, including the naive-baseline control that separates what works and what doesn't, and the checks that catch benchmarks measuring the wrong compressor.
You'll leave with a provider-neutral architectural pattern you can drop into any Python LLM pipeline, and a reusable methodology for deciding whether it's worth it on your own production traces.
Reference implementation: the open-source Headroom library and my benchmark repository with all code and evaluation assets will be public.