PyData Amsterdam 2026

This talk is for Python developers moving agents from prototype to production. It explains four architecture patterns for reducing inference cost and task completion time for agents using managed, open-weight models, drawing on real long-context, tool-using workloads.

1. Model qualification: Build a representative evaluation set, define task-specific quality and latency requirements, and select the serving configuration with the lowest measured cost per successfully completed task. Model size, throughput, generation length, retries, and failure rate are treated as parts of the same decision.
2. Context design: Separate stable and volatile context, serialize prompts and tool definitions, and structure conversation history to maximize safe prefix-cache reuse. Observed cached tokens are measured, not inferred.
3. Runtime routing: Route only across models that have passed the quality gate. Use task type, complexity, risk, conversation phase, and generation budget to select the fastest and most economical qualified model, with explicit fallback and escalation policies.
4. Targeted adaptation: Fine-tune or distill a smaller model when the workload is repeated, measurable, and stable. Compare the fixed cost of data preparation, training, evaluation, and deployment with the recurring savings from lower latency, lower inference cost, shorter prompts, or fewer retries.

The four patterns are connected through a Python reference architecture using OpenAI-compatible endpoints, deterministic prompt construction, task evaluation, routing policies, fallbacks, and request-level telemetry. A common scorecard tracks task success, structured-output validity, latency, cached tokens, retries, and inference cost.

Attendees will learn how to choose between closed APIs, local models, and managed open inference; how to apply the four patterns in a practical order; and how to measure whether an optimization improves the cost and response time of successful task completion without reducing quality. The session is scoped to 25 minutes and uses architecture diagrams, short code examples, and a single running case rather than a live demo.

Outline (25 minutes + 5 minutes Q&A)

1. Why open weights and managed cloud inference (4 minutes): Compare closed APIs, local models, self-managed serving, and managed open inference.
2. Pattern 1, model qualification (4 minutes): Define quality gates and identify the optimal balance of quality, cost and latency.
3. Pattern 2, context design and prefix reuse (4 minutes): Implement stable prefixes, deterministic serialization, prefix caching, and request-level cache measurement.
4. Pattern 3, runtime routing (4 minutes): Select the fastest and most economical qualified model for each task, with fallback and escalation policies.
5. Pattern 4, targeted adaptation (4 minutes): Evaluate when adaptation changes the inference economics enough to justify its fixed costs.
6. Python reference architecture (3 minutes): Connect prompt construction, evaluation, routing, endpoints, fallbacks, and telemetry.
7. Decision framework and takeaways (1 minute): Apply the patterns in order and select the appropriate deployment model.
8. Token Factory and invitation (1 minute): Explore Nebius Token Factory as a managed implementation of the architecture and explain how attendees can try it.
9. Q&A (5 minutes).