PyData London 2026

We'll explore the architecture of production-grade semantic layers, demonstrating how Semantido enables reliable text-to-SQL applications by providing LLMs with rich contextual understanding of database schemas, relationships, and business logic. Attendees will learn practical patterns for implementing semantic layers that bridge the gap between user intent and database queries by building a semantic layer for a fictional company.

The repos for this tutorial:
https://github.com/hikarilabs/pydata-london-2026-ui.git
https://github.com/hikarilabs/pydata-london-2026-sql.git

The deck:
https://docs.google.com/presentation/d/1uyN-xblr6d6cKb9HgHcgfd4ZafnN3mLRcVTuiKsYL-E/edit?usp=sharing

Unlocking the full potential of AI starts with your data, but real-world documents come in countless formats and levels of complexity. This session will give you hands-on experience with Docling, an open-source Python library designed to convert complex documents into AI-ready formats. Learn how Docling simplifies document processing, enabling you to efficiently harness all your data for downstream AI and analytics applications.

Your model is well-calibrated on average, but is it calibrated for every subgroup of your users? In this hands-on tutorial you will learn what multicalibration is, why standard calibration methods leave systematic errors hidden in subpopulations, why this matters for ML models in production, and how to fix it in a few lines of code using MCGrad, an open-source Python library that has been battle-tested on hundreds of production models at a large tech company. Attendees will leave with a working notebook they can immediately apply to their own projects.

Want to write your own GPU algorithms, but not sure how to get started or keep them portable? Come to this hands-on session to learn tile programming with CUDA Tile and cuTile Python: you will build an accurate mental model of tiles and thread groups, write and debug real GPU kernels in a browser-based JupyterLab (no installation), profile and tune performance with NVIDIA Nsight, and see how the same tile code applies across DL and HPC examples like LLM inference and conjugate gradient, including when to use tiles vs SIMT and how to mix both.

AI agents are moving into production in 2026, but when something goes wrong (a tool call fails silently, an LLM takes 13 seconds to respond, token costs spike overnight) teams struggle to diagnose issues across multi-step agentic workflows. In this hands-on tutorial you will solve a real problem on the island in Animal Crossing with a FastMCP Model Context Protocol (MCP) server in Python, instrumenting it with OpenTelemetry following the emerging GenAI and MCP semantic conventions and visualising end-to-end traces in a local Jaeger instance. Did I mention that events on the island occur in real time and are collected and processed using Apache Kafka?

You will learn how distributed tracing captures the hierarchical relationship between agent conversations, tool executions and MCP protocol messages, and how to use that visibility for debugging, cost analysis and performance optimisation (including picking the right model and checking if you’re drowning in serialisation overhead). You will leave with a fully instrumented MCP server, a Docker Compose real-time observability stack and the knowledge to bring production-grade observability to your own agentic AI systems.

This hands-on tutorial takes participants from zero to confident use of tabular foundation models. Using real datasets, we will run TabICL-style models, benchmark them rigorously against XGBoost and Random Forest, diagnose their behavior, and build intuition for when they help and when they don't.

LLMs are increasingly being used to take actions, call APIs, and write code. But giving AI agents the ability to run code opens up a surprisingly tricky question: how much control do you actually hand over?

There's a full continuum here, from structured tool calling at one end to full computer use at the other, but most developers don't realise how many interesting options live in between. That gap matters, because the extremes both have serious trade-offs: pure tool calling is safe but sequential and limiting, while full sandboxes or computer use are powerful but complex, slow, and often a hard sell to enterprise security teams.

This talk introduces Monty, a minimal Python interpreter written in Rust, purpose-built for running AI-generated code safely. Unlike traditional sandboxing approaches that start with full access and try to lock things down, Monty starts from zero and requires you to explicitly grant each capability — meaning the LLM can only interact with the outside world through functions you wrote, control, and can audit. It's a new paradigm: not AI using your tools, but AI writing its own programs to coordinate your tools.

In this talk, you will learn how to think about the control-capability trade-off when building AI agents, where Monty sits on that spectrum and why, and how to use it with Pydantic AI to replace sequential tool calls with expressive Python — complete with a live demo traced through Logfire.

Basic familiarity with Python and LLM tool use is helpful but not required. No prior knowledge of Rust or sandboxing concepts needed.

Most machine learning methods give you a prediction but not a measure of how much to trust it. Bayesian Additive Regression Trees (BART) combine the flexibility of tree ensembles (e.g. random forests, boosting) with full uncertainty quantification—every prediction comes with a probability interval, not just a point estimate. This hands-on tutorial introduces BART for regression and classification. Using pymc-bart, participants will learn to fit flexible models that automatically capture non-linear relationships while providing honest uncertainty estimates. We emphasize practical interpretation throughout: visualizing predictions with uncertainty bands, understanding variable importance, and interpreting model output.

AI systems that can autonomously navigate websites, fill forms, extract data, and complete multi-step workflows; are one of the most exciting and practical applications of large language models in 2026. Libraries like browser-use (60k+ GitHub stars) and Skyvern have demonstrated their potential, but their abstractions can obscure the surprisingly approachable fundamentals underneath.

In this 90-minute hands-on tutorial, attendees will build a browser agent entirely from scratch using only Python, Playwright, and an LLM API. No agent frameworks, no magic; just the core building blocks: extracting and structuring the DOM into an LLM-friendly representation, capturing screenshots for vision-based reasoning, building the observe-think-act agent loop, and handling real-world challenges like dynamic content, multi-tab navigation, and error recovery.

By building from first principles, attendees will gain a deep understanding of how browser agents actually work; knowledge that transfers directly to using, debugging, and extending any browser agent framework. Every participant will leave with a working agent that can autonomously complete tasks on live websites.

This tutorial is aimed at Python developers and data scientists who are curious about AI-driven browser automation. Basic Python proficiency and familiarity with async/await are expected. No prior experience with Playwright, browser automation, or agent frameworks is required.

Test-Driven Data Analysis is a methodology for reducing errors in data and data analy. It is also an open-source Python package for supporting key aspects of the methodology. This tutorial will provide hands-on experience using the library to validate data and write tests (manually or automatically) for analytical processes. It will also highlight approaches to avoiding errors in specific areas not amenable to software support.

[Preflight check]: You may want to have a look at the repo and pre-download some libraries or models in advance: https://github.com/Cheukting/lighteval-exercises

Large Language Models (LLMs) are becoming central to modern applications, yet effectively evaluating their performance remains a significant challenge. How do you objectively compare different models, benchmark the impact of fine-tuning, or ensure your LLM responses adhere to safety guidelines (guard-railing)? This hands-on workshop addresses these critical questions.

Production AI systems improve through a data flywheel: teams create training examples from curated source material, those examples shape model behavior, production usage reveals what the model still needs, and those usage signals become the next round of improvement. This hands-on tutorial focuses on the data pipelines behind that flywheel: how to generate, validate, and anonymize training data without relying on one-off prompt scripts.

Participants will build a reproducible training-data pipeline using NVIDIA NeMo Data Designer and NeMo Anonymizer. We'll start by working through text-based examples that introduce the basics of Data Designer: defining the shape of a dataset, connecting generation to source records, creating structured outputs, and filtering generated rows with judge-based quality checks. Then we'll extend the same pattern to multimodal document understanding with rich synthetic business document images, VLM-classified visual focus areas, and VLM-generated visual QA examples.

Finally, we'll shift from workshop-generated data to production-style usage data. Using Anonymizer, participants will detect and transform sensitive fields so usage logs can safely become source material for the next training iteration.

By the end, participants will understand a practical pattern for multimodal training data with privacy-safe feedback: source data -> generate -> validate -> anonymize feedback -> improve.

Posterior predictive checks are a key step within Bayesian modeling workflows where we compare model predictions with the data used to fit the model. By focusing on distributional comparisons instead of point estimates, they offer valuable insights about our models, where they fail and inform model improvements. Knowing a model is not completely right is relatively easy, knowing why that is the case and how to fix it are a whole other question which will be the focus of the tutorial. This tutorial will provide data scientists and researchers with multiple strategies for posterior predictive checks to allow their use in continuous, discrete or categorical data, and for homogeneous or heterogeneous data.

Join us after Day 1 of PyData London for an evening of drinks, conversation, and community at Fleets • Bar & Kitchen, a stylish bar and social space just steps from St Paul’s Cathedral. It’s the perfect opportunity to unwind after a full day of talks, connect with fellow attendees, speakers, and organizers, and keep the PyData conversations going in a relaxed setting. Whether you’re looking to network, catch up with colleagues, or simply enjoy a great London evening with the community, we’d love to see you there.

Thank you to our sponsor PDFTA & Coefficient, find John Carney and John Sandall and give them a big thank you!

Venue: 44–46 Ludgate Hill, London EC4M 7DE
Learn more: Fleets website

The tech industry rises and falls in cycles, like the tide follows the moon. Every few years, the ground shifts and we all have to learn to walk again. It happened to me twice, when I moved from being an award-winning cartoonist to an underemployed web developer during the Recession, and again as a React Core engineer who washed up in AI after layoffs. I went from peak influence to the absolute bottom rung each time. Both times, what saved me wasn't genius or a special gift. It was embracing and being embraced by my new community.

This is a keynote about which skills travel across every shift, what we owe each other as the waves keep coming, and how, in a rising tide era, to find your boat and row with the crew. Whether you've been here a while, have just arrived, or are watching the water rise from the shore, there's a place in this boat.

Every team runs on unwritten rules. Habits that shape how decisions get made, how failure is handled, and what is safe to say. This talk provides a framework for reading those rules, understanding the collective self-image that drives team behaviour, and influencing culture from any position. With a look at how AI adoption is forming new rules in real-time, you will leave knowing how to decode the system you are in and start shifting it.

Multi-agent AI systems promise autonomous reasoning, but most tutorials stop at prototypes. This talk shares hard-won lessons from deploying a production multi-agent RAG platform on serverless AWS , covering agent orchestration patterns, cross-region LLM routing, vector search cost optimisation, and the observability strategies that keep it all running reliably.

You'll learn concrete patterns for coordinating multiple RAG-enabled agents via SQS and Lambda, the cost/latency trade-offs between managed and self-managed vector search (including how to achieve 90% storage savings), and practical observability strategies using Langfuse and dead-letter queues. Whether you're scaling your first RAG system or architecting multi-agent workflows, you'll leave with actionable patterns you can apply immediately.

Adopting a streaming architecture as a Python developer often means abandoning the tools and abstractions you know: DataFrames, batch processing, familiar data workflows, in favour of an entirely different mental model. After ten years of tackling this problem across multiple companies, I've learned it doesn't have to be that way.

In this talk, I'll show how to treat Kafka not as a stream of individual messages but as a source of micro-batches, and how to deserialize those messages, whether JSON or Protobuf, into Arrow-backed DataFrames. The result: your processing code looks the same whether the data comes from a Parquet file or a Kafka topic.

No heavy framework required. Using confluent-kafka and Apache Arrow, I'll walk through how to build this from the ground up, so you understand every layer of the stack.

Sensors operating in complex environments produce noisy data. Determining exactly when a system transitions between states — and what values it is recording — is surprisingly hard: vibrations, environmental changes, and gradual shifts all conspire against simple threshold approaches. This talk walks through a real-world Python pipeline that solves this problem, starting with classical signal processing, exposing its failure modes, and then building a principled solution using a Kalman filter for noise reduction coupled with a Hidden Markov Model (HMM) for state inference. Attendees will leave understanding how to frame sensor problems as state estimation tasks and how to apply these techniques in Python using necessary libraries.

Come build something with the PyMC development team.

Code sprints are collaborative working sessions where contributors of all experience levels tackle meaningful open issues side by side. Whether you want to squash a long-standing bug, sharpen the documentation, build a worked example, or simply understand how a major open-source project operates from the inside — there's a place for you here.
PyMC is the most widely used probabilistic programming library in Python, and the people who build it will be in the room. Bring your laptop; we'll handle the rest.

When building high-performance systems for analytical workload, we often focus on the efficiency of the algorithm, like reducing Big-O complexity or optimising numerical routines. Yet in real world workloads, the decisive factor is not just the algorithm but the shape of how the data is laid out, traversed, and distributed across processes.

This talk will cover aspects of mechanical sympathy, focussing on how structures in memory can benefit from cache-sensitive, SIMD-enabled (vector instructions) CPUs, constrained by memory bandwidth and optimised for predictable, contiguous access.

We will use real-world examples to show how minimising serialisation overhead and enabling efficient cross-process and cross-language data exchange reduces the cost of data movement across systems. Beyond single-system performance, we will examine why Arrow’s standardised, zero-copy columnar format is a critical enabler of distributed execution. We will see how columnar formats support scalable computation across threads, processes, and distributed nodes.

Building a demo agent with hundred billion parameters and beyond can be easy. Deploying reliable, cost-effective agents in production is hard. This talk provides a comprehensive roadmap for taking AI agents from prototype to production, with a focus on migrating from expensive frontier LLMs to efficient small language models (SLMs).

We'll explore the entire lifecycle of production agent development: test-driven development practices adapted for non-deterministic AI systems, agent architectures and migration strategies from large to small models, CI/CD considerations for agents, and observability frameworks which capture what matters and assist in remediating failures.

Whether you're running agents at scale or planning your first deployment, you'll leave with actionable strategies and concrete tools to build reliable, maintainable agent systems with small language models.

Training large language models requires massive, high-quality text corpora—but web-scale datasets like Common Crawl contain significant near-duplicate content that degrades model performance and wastes compute. Existing solutions like Spark MLlib's MinHashLSH suffer from UDF serialization overhead and shuffle explosion, causing out-of-memory failures at scale.

We present a partition-aware MinHash LSH system that co-locates similar documents within Spark partitions, dramatically reducing cross-partition shuffles during similarity computation. Our approach combines vectorized MinHash generation using mathematical permutation tricks, band-based candidate filtering with configurable collision limits to handle edge cases like boilerplate false positives, and GraphFrames-based connected components for transitive deduplication.

Benchmarks on Common Crawl 253.4 million documents, generating 2.1 billion candidate pair
completed in under five hours on a 9-node r5d.8xlarge EMR cluster. We discuss key optimizations including partition-aware MinHash LSH and band collision filtering for common boilerplate content.
Attendees will learn partition-aware LSH design patterns, strategies for handling boilerplate-induced false positives, and how to integrate deduplication into existing Spark ETL pipelines. The system will be open-sourced, enabling practitioners to deploy production-ready deduplication pipelines for their own LLM training workflows.

Decarbonising UK’s home heating is one of the greatest challenges of the Net Zero transition, yet it currently relies on individual household decisions supported by government incentives. To help accelerate the local delivery, we are building a tool that maps the most suitable low-carbon heating for clusters of properties at a neighbourhood level.

In this talk we will walk through our end-to-end data science pipeline, covering processing of large-scale geospatial data, the nuances of modelling where ground truth data does not yet exist, and how to translated local authorities needs into a functional product. We will present our Python tech stack and will conclude with a showcase of the user interface.

Whether you're interested in geospatial data engineering, machine learning for social good, or how to work within a multidisciplinary team, this talk offers a blueprint for building data products with real-world impact.

JupyterLite is a JupyterLab distribution that runs entirely in the web browser, backed by in-browser language kernels. Using it you can run Python, R and C++ in your browser via WebAssembly, use git and vim in a terminal, and access AI agents in a safe, sandboxed environment.

This talk will present a comprehensive summary of all things JupyterLite, and provide live demonstrations of many of its key features and how easy it is to deploy.

The talk assumes basic familiarity with JupyterLab but not necessarily JupyterLite. It will be of benefit to anyone who wishes to learn about this emerging technology and its potential for scalable, accessible interactive computing.

As AI agents become more popular, one question becomes increasingly important: how do you actually know if your agent is performing well? Multi-turn conversations are hard to evaluate because because there is rarely one right answer and at any given turn multiple responses can be correct. In this talk, we'll walk through a structured approach to evaluating complex conversations. We'll cover what makes a good conversation, techniques for evaluating multi-turn conversations where multiple outcomes are simultaneously valid, and how to scale evaluation pipelines. Finally, we'll discuss practical frameworks for continuous improvement and building confidence in your agent's real-world behaviour.

Modern lakehouse architectures promise flexibility and scale — but governance is often an afterthought. While we version data and evolve schemas, we rarely version or test access policies.

This talk explores how to implement governance-as-code in a lakehouse using the REST Catalog from Apache Iceberg, applying Zero Trust principles and enforcing fine-grained policies with Open Policy Agent (OPA) and Cedar.

Attendees will learn how to move from static IAM and implicit trust to centralized, engine-agnostic, policy-driven governance.

Dynamic Path Analysis (DPA) extends survival analysis with a causal, time-varying perspective. This allows causal effects to be decomposed into direct and indirect pathways that evolve over time. The perspective is particularly valuable when interventions (exercise) act through mediators (weight loss) whose influence changes dynamically in time, because we get to distil when each driver of our survival probabilities are active and whether their combined effects are harmful or positive.

Despite its conceptual appeal, DPA remains niche, with existing implementations limited to frequentist R packages and no Bayesian or Python-native alternatives. In this talk, I present a Bayesian, generative implementation of Dynamic Path Analysis using PyMC. By discretising time and modelling cumulative hazard effects with smooth spline priors, we obtain interpretable time-varying causal effects with coherent uncertainty quantification. I benchmark the approach against canonical dpasurv examples and discuss why DPA focuses on hazards rather than survival curves.

This talk is aimed at Python users interested in survival analysis, causal inference, and Bayesian modelling.

"Join us for a relaxed lunch gathering to meet this year's handpicked scholars - a group of exceptional people bringing fresh perspectives to our community.

This is a chill bring-your-plate space to meet some of the PyData 2026 diversity team and welcome some fine folks with diverse interests & experiences. Let's find conversation over lunch and a shared table.

Space permitting, all are welcome, and speakers and allies are encouraged to squeeze in!"

Python became the language of data science because it made hard work feel possible. It gave scientists, analysts, engineers, and researchers a shared way to express ideas without forcing them to become software specialists first. The result was a style of software that made powerful systems easier to learn, easier to combine, and easier to trust.

In this keynote, Jeremiah Lowin explores what it means for software to be Pythonic: simple, composable, readable, and easy to reason about. Those qualities helped Python become the default language for data science, and they matter even more now that software is beginning to build software.

The next generation of agentic systems will be judged by whether people can understand them, change them, and trust them. This talk argues that the lesson of PyData is also the challenge for AI: build systems that remain easy to reason about as they become more powerful, and use emerging interfaces like MCP to make data easier to communicate, explore, and act on.

What if you could watch an AI’s thought take shape? For years, LLMs have been impenetrable "black boxes," but we are finally beginning to find ways to see how the ghost in the machine actually works.

This talk explores mechanistic interpretability, a subfield of AI that aims to understand the internal workings of neural networks. Mapping these internal "circuits" is not only just a philosophical curiosity - or duty: it is a high-stakes engineering necessity for safety, debugging, and trust.

In science and engineering, we are frequently challenged by the inability to manipulate environmental variables—a key component of the scientific method. For example, we cannot simply stop a hurricane in its tracks or change the temperature of the Sun. Instead, we heavily rely on "Forward Models"—numerical simulations that predict data from physical parameters. However, these models are often massively computationally expensive.

Emulators (or surrogate models) present a solution. Whether solving a single time-sensitive equation or searching a high-dimensional inference space, emulators can accelerate simulation results by orders of magnitude. In this talk, we show how these machine-learning tools are revolutionising research across STEM disciplines, from inferring input parameters to developing digital twins and augmenting foundational models.

Model Context Protocol is a standard for defining tools that can be made available to LLMs and AI applications. There’s a lot of noise out there about what you should use to get the best results from AI, so in this talk, I will provide some guidance on when you should use MCP, and when you should reach for some other tool. I will describe cases where MCP is the right tool for the job, and when other things, like skills or other context files, are better. I will also devote attention to questions of security and authentication, which are important for MCP, and provide concrete examples of how MCP servers can be used to unlock agentic workflows while also strengthening data governance. This talk is intended for those who are interested in using LLMs for workflows involving data. No prior experience with MCP is required.

Your company launches a loyalty program — but not everywhere at once. Ten stores get it in January, another ten in March, the rest later. Leadership asks: "Did it work? By how much?" You compare before and after... and get a number that's wrong. Phased rollouts break naive pre/post comparisons, and standard regression quietly gives misleading answers.

This talk shows a practical Python workflow for getting it right. Using a realistic store-rollout example and CausalPy (an open-source library), I'll demonstrate how to produce event-study plots that show when and how much an intervention takes effect — with uncertainty estimates your stakeholders can actually act on. Whether you're measuring feature flags, marketing campaigns, or policy changes, you'll leave with a reproducible notebook and a step-by-step workflow you can apply tomorrow.

Join our chill space, unwind, chat about Feminist AI and contribute to the PyData London DIY collage zine.

Choosing a cloud instance type for a DS/ML/AI workload is still largely a heuristic exercise. While public pricing and hardware specifications are available, they are fragmented, inconsistently structured, and challenging to compare across cloud providers -- especially once real workload performance is taken into account.

In this talk, we present Spare Cores Navigator, a Python-queryable benchmark dataset that covers thousands of cloud server types from multiple vendors, with standardized performance and cost-efficiency metrics. We demonstrate how instance selection can be expressed as a simple data query, e.g. filtering by workload characteristics, hardware or compliance constraints, and budget, then ranking candidates by price-performance.

The Python data ecosystem is migrating from NumPy-based arrays toward Apache Arrow. Polars is built entirely on Arrow, and Pandas is heading in the same direction. Yet differences in string encoding, missing values, schemas, and index metadata make interoperability between the two formats surprisingly costly and error-prone. This talk examines these challenges through a case study of how ArcticDB, the open-source client-side dataframe database, navigated this same migration.

Your intelligent application is your castle, and your security practices are the moat that protects it. Inside your castle, you must aim for full visibility into what you’re running and why, with freedom to iterate. Your moat creates your security perimeter, ensuring no proprietary data leaves your castle and enforcing best practices including data provenance, cryptographically signed models, evaluation tools, build pipelines and reproducible environments.

Build on your infrastructure, answer to your requirements, scale on your terms.

Multilingual embeddings are often assumed to place different languages into a shared semantic space. In practice, that alignment breaks down in systematic ways.

This talk explores where multilingual embeddings work, where they fail, and why. Using examples across multiple languages, I show how tokenisation, training data imbalance, and semantic ambiguity shape embedding behaviour in practice, along with practical diagnostics for evaluating multilingual embeddings.

Scientific publishers tag millions of articles with author-provided keywords, but these keywords are noisy, inconsistent, and semantically ambiguous. "Machine learning," "ML," and "machine-learning" all mean the same thing, while other terms shift meaning across disciplines.

This talk presents a production pipeline that extends OpenAlex's 4-level hierarchy with a fifth in-house Concept layer, producing a 115K-concept scientific taxonomy.

SPECTER2 embeddings model semantic similarity, and per-field Leiden clustering with CPM resolution groups 100K+ concepts via mutual kNN graphs — with hyperparameters selected through grid search and custom pair-based evaluation. Qdrant enables vector-based hierarchical attachment.

LLMs are deployed at five targeted stages — granularity filtering, field classification, cluster renaming, explanation generation, and topic-assignment validation — while deterministic methods handle everything else, ensuring scalability and reproducibility.

The resulting taxonomy powers a paper-tagging pipeline where SPECTER2 retrieves ~150 candidates per paper across multiple text-splitting strategies, deterministic filters prune by field/subfield distribution and near-synonym merging, and an LLM reranker selects the final 5–8 concepts. These assignments enable applications such as temporal trend detection over emerging research topics and more.

Attendees will learn when to integrate LLMs in large-scale NLP pipelines, how to scale graph clustering to 100K+ nodes, and how to design hybrid embedding–LLM systems that turn noisy metadata into reliable scientific intelligence.

Coding agents such as Cursor and Claude Code are fundamentally changing software development workflows. Most teams, however, still rely primarily on proprietary frontier models.
In this demo-driven session, I will show how to pair modern coding agents with high-performance open models running on Nebius Token Factory, with a focus on developer experience, model behavior, considerations relevant to production use.

Attendees will receive platform credits for getting started immediately with open-model-powered coding agents.

As coding agents grow in popularity, open source project documentation is increasingly consumed by LLMs. When people build things with your open source library their agent will read your documentation and write code based on what it discovers there. To ensure your users have a good experience we need to start thinking about how to write and publish our documentation to make sure agents produce the best code possible.

Coding agents are now on the critical path for making decisions around which libraries to use. For open source developers it’s important to market your projects to LLMs as well as humans. Publishing material about the project in a way that is easy to discover and parse for models is key to increasing adoption.

This talk will cover key things you need to know to make your project successful in a coding agent world:

• SEO for the LLM age
• Publishing your docs in context efficient formats like markdown
• Providing plentiful examples that ensure agents produce idiomatic code for your library
• Adding LLM specific information to the documentation to help shape behaviour

Multi-agent GenAI systems don’t fail because models lack intelligence, they fail because they lack memory.

As LLM applications move from demos to production, semantic memory becomes the defining systems challenge. Agents must remember user preferences, share context across roles, preserve conversational state across sessions, and evolve over time, all without exploding token costs or losing observability.

In this talk, I’ll explore semantic memory as a data engineering problem rather than a prompt engineering trick. Drawing on real-world experience from the Azure Cosmos DB engineering team, we’ll examine how to design layered memory for multi-agent systems in Python: short-term conversational state, episodic event logs, declarative and procedural memory, and retrieval-driven personalization.

Using a practical multi-agent travel planner built with LangGraph, we’ll implement patterns such as session-level versus per-turn persistence, hybrid retrieval design (structured filters plus semantic signals), memory lifecycle management (write, retrieve, summarize, supersede, expire), and checkpointed workflows for reproducibility and debugging.

You’ll leave with practical design heuristics for building agent systems that become more reliable, more efficient, and more explainable over time.

All demonstrations will be in Python and applicable to production-scale systems.

Keep the PyData London energy going as we wrap up an incredible two days of talks, learning, and connection. Join us for our closing social immediately following the final sessions on Saturday from 5:00–6:00 PM, right at the conference venue. This is a great chance to continue conversations sparked during the day, connect with speakers and fellow attendees, and celebrate another fantastic PyData London with the community. Grab a drink, mingle, and enjoy one last opportunity to network and reflect on the ideas and insights shared throughout the conference before heading out for the evening.

Lightning talk sign up will take place at the NumFOCUS booth all day Saturday.

The EU AI Act is often seen as a legal concern, but many of its requirements directly affect everyday ML workflows. This talk shows data scientists and ML engineers where the regulation impacts the machine learning lifecycle and presents concrete, low-overhead patterns to make ML systems more AI Act–ready, without slowing down development.

Tesco is applying AI and Data Science at scale to solve some of the most complex problems in retail. From personalisation to optimisation and decision support, our systems power millions of customer interactions and operational decisions every day. In this talk, we highlight how these capabilities come together in modern AI-driven customer experiences, and why Tesco is at the forefront of applying AI in real-world, high-impact settings.

We briefly introduce Tesco’s Meal Planner to highlight the technical challenges behind AI-driven customer experiences. A key challenge behind the scenes is translating recipes into products that customers can actually buy. We approach this by connecting recipes, ingredients, and products in a way that enables the system to move from meal ideas to a ready-to-shop basket. This requires balancing richer reasoning over customer needs and preferences with the practical realities of a live retail environment, such as a constantly changing product catalogue, cost, and availability.

We then turn to one of the most important aspects of deploying AI systems at scale: Evaluation and how it helps to ensure that the system behaves reliably. When AI assistants support customer journeys, even small errors can degrade the experience or lead to incorrect outcomes. We present our evaluation framework, which combines multiple techniques to assess both system behaviour and response quality. This allows us to identify issues early, enforce consistent standards, and continuously improve performance.

Overall, this talk offers a practical view of how Tesco applies AI and Data Science to real-world problems, combining strong technical foundations with robust evaluation to deliver reliable and impactful customer experiences.

Text-to-SQL makes great demos, but in real systems generating queries is rarely the hard part - understanding data is. Modern data is increasingly S3-first and multimodal, where meaning is defined by Python workflows, not table schemas.

To work reliably, both agents and people need data context across multiple layers: storage context (what exists and where), metadata context (what’s inside files), dataset context (how files are grouped and versioned), and code context (the transformations that define semantics).

In this talk, I’ll share a practical framework for building these context layers in Python-first systems, and show how DataChain makes multimodal workflows agent-ready in domains like Physical AI and biotech.

Birds of a feather to share what’s working well for us to do the best Python engineering and datascience that we can, while leading the way for our teams.

We rely on dashboards to tell us if our RAG system is working. But most standard metrics, Cosine Similarity, BLEU, and even BERTScore, are fundamentally broken for measuring factual correctness. They measure text overlap or semantic drift, not truth.

This means you can have a "90% Accurate" system on paper that hallucinates dangerous misinformation in production. This talk dismantles the current state of RAG evaluation. We will look at why "Golden Datasets" are often contaminated, why "LLM-as-a-Judge" is biased towards its own output, and how to build a robust, adversarial evaluation pipeline that actually catches failures before your users do.

Large SQL codebases inevitably accumulate duplication, inconsistency, deep nesting, and subtle logic errors, making refactoring slow, risky, and often unrealistic to do by hand. This talk shows how Python metaprogramming can turn SQL itself into data that can be analyzed and transformed safely and automatically.

Instead of relying on fragile regex patterns or manual inspection, we use Python to parse queries into Abstract Syntax Trees (represented as nested dictionaries) using libraries such as sqloxide. Once SQL itself is encoded as data, entirely new workflows become possible.

The session walks through practical examples of treating SQL programmatically via tree operations in Python: computing subquery depth for linting, wrapping all denominators in NULLIF() with a simple AST rewrite, auto‑aliasing aggregate expressions, and generating dependency graphs of temporary tables used across pipelines, among others. Each example highlights how metaprogramming enables precise, automatable refactors that would be error‑prone or impossible through text manipulation alone. This talk is designed for analytics and data engineers who work with large SQL codebases.

Britain has an invisible clean energy graveyard. Over 3,800 clean energy projects have been cancelled in the UK since 2010, representing enough capacity to power millions of homes. This talk presents the Clean Energy Graveyard - an open-source python pipeline & interactive web visualisation that transforms the government's Renewable Energy Planning Database (REPD) into a story about what's blockoing out energy transition.

One of the hardest parts of applied NLP has always been breaking down complex business problems into machine learning components. It's so hard because it requires domain expertise and reasoning about the specific use case, and it's the one thing technology couldn't fix. But what if we could take some of the learnings from AI-powered coding assistants and apply them to solving real-world NLP problems? In this talk, I'll show how we've built powerful assistants and tools to help developers solve NLP tasks using open-source software, and create modular solutions that are small, fast and fully data-private.

Data work often gets blocked by the unglamorous parts: brittle pipelines, unclear ownership, slow deployments, and systems that are hard to trust. This talk is about deliberately making data infrastructure “boring” — predictable, observable, and easy to change — so that the data itself can be used in lots of exciting ways.

Climate Policy Radar is a non-profit building open, credible databases and AI powered tools to support informed climate, nature, and development action.

Using a real-world journey from an unreliable ingest to a steadier, federated platform, this talk will walk through the principles and trade-offs that matter most: resilience over heroics, incremental delivery over big-bang rewrites, and transparency over intuition. The focus is not on specific tools, but on the engineering moves that turn data pipelines into dependable systems: orchestration that supports recovery, interfaces that unblock downstream teams, quality signals that can be acted on, and a shared layer (data lake/warehouse) that aligns definitions and reduces duplication.

Attendees will leave with a practical mental model for taking maturing data flows and making them boring — in a good way.

Explainable AI is frequently invoked to make machine learning systems understandable and trustworthy. In real applications, however, explanations are often expected to justify decisions and support action. Drawing on experience with remote sensing–based risk monitoring, this talk examines the gap between the guarantees of explainability methods and the expectations placed on them by different users. It discusses how explanations can inform practice, how they can be misinterpreted, and when focusing on explainability may obscure deeper problems in models or data.

In this unconference session, hear what reviewers actually look for in proposals for PyData and how to frame your idea so it stands out.

Are you a PyData Meetup Organizer? Come join us in the boardroom at lunch to mingle with other leaders.

Large language models (LLMs) have taken the world by storm since the public launch of ChatGPT 3 in November 2022, sparking a huge number of LLM-powered tools, products and start-ups. Since then LLMs have gained reasoning and tool use capabilities, and have been integrated into more autonomous agentic workflows, leading to significant increases in their usefulness for software engineering work. However, despite being readily accessible to us all, these models and their agentic wrappers remain black boxes to many of us using them in our daily work.

Martin will demysitify LLMs by providing an intuitive understanding of how they build upon key prior advances to successfully cross the "uncanny valley" of text generation and achieve almost flawless fluency. He will explain what makes these models "foundational", illustrating how this "one weird trick" of next word prediction results in models that can be easily fine-tuned for conversation, coding and reasoning, and we'll take a peek under the hood of how LLMs have been extended to integrate private data sets, call external tools and support more autonomous agentic workflows.

This talk won't make you an expert on deep neural networks, transformers, fine-tuning or agentic workflows, but it will give you a peek behind the curtain of how these seemingly magical models work and hopefully give you enough intuitive understanding to explain them to friends and family.

Our video streaming service hosts vast catalogue of content, but producing tailored marketing assets is slow, manual, and costly and therefore limited to the most popular shows with the biggest budgets. This talk describes how we’re using python to automate the creation of thousands of marketing assets to promote our full catalogue on and off-platform. The system combines audience data, programme metadata, machine learning, and automated rendering in Adobe After Effects. For editorial safety, we’ve built AI-assisted QA layers, automated Slack messaging, and plotly dash apps to allow controlled human review and intervention. All using python (mostly!)

Polars is a dataframe library which has taken the world by storm over the last 4-5 years. Because people love benchmarks, people often compare it with SQL-like engines such as DuckDB, PySpark, Daft, and others. But what if, instead of comparing performance, we compared semantics?

This talk will make no mention whatsoever of performance differences. Instead, it will focus entirely on the semantic differences - which don't get nearly enough attention - of Polars vs SQL. Attendees will leave with a heightened appreciation for the differences between the Polars and SQL models, and an understanding of the consequences this has on their code.

A few nights ago I was up to 2am obsessively crafting an LLM plan. ("Just one more prompt!" - famous last words). Yet it still did something inexplicably stupid. 🫠 So yeah: LLMs are both genuinely useful and genuinely destabilising. Focusing on the first and ignoring the second is how people burn out.

This talk is an honest account of what it feels like to be a developer right now, from someone inside it, and some thoughts on what might actually help. My thesis: we've been optimising for model output when we need to be optimising for human experience.

I'll share observations from my work, peers and colleagues. The peculiar fatigue of machine supervision: holding the intent in your head while the machine generates volumes of mostly-correct output that still needs your eyes, your judgement, and your taste. The way the satisfying part of the work shrank while the exhausting part grew. The isolation of pair-programming with a machine, and the loss of real human learning, interconnection and collaboration. And underneath all of it: uncertainty. About market conditions, about employability, about whether the skills we've spent years building will still matter.

The second half is about what's been working for me, and what hasn't. On the human side: encouraging pairing and teamwork even when the tools push you toward isolation, sharing the pain openly, naming the uncomfortable thing. On the technical side: structuring your environment to collaborate with LLMs more deliberately — writing plans, configuring project-specific rules. Learning when to stop prompting and just write code. And critically: rebalancing the push and pull of information so that you're directing your attention, not feeling at the mercy of the model's output. More Star Trek, less Black Mirror.

Leave with concrete strategies for recalibrating your workflow, challenges to discuss and the reassurance that if you're finding this hard, you're not broken. The feedback loop is. And we can start fixing that.

Data scientists, analysts and engineers are all feeling the pressure — but what's actually changing, and what's hype? This session brings us together to share real experiences of integrating LLMs and agents into data workflows, honestly assess which skills still matter, and tackle the uncomfortable question: are we building the tools that replace us?

Large Language Models are rapidly changing how we think about recommendation systems. Traditional pipelines based on collaborative filtering or matrix factorization are being complemented and sometimes replaced by embedding-based and LLM-driven approaches.

In this talk, we explore how modern recommendation systems can be built using LLM embeddings, vector databases, and hybrid architectures that combine classical ML with generative models. We will discuss practical design patterns for personalization, retrieval, ranking, and user modeling, focusing on real-world constraints such as latency, cost, and evaluation.

The session emphasizes hands-on insights from production systems and highlights where LLMs add real value and where they don’t. Attendees will leave with a clear mental model for designing scalable, LLM-powered recommendation systems beyond toy examples.

Most RAG demos stop at retrieval and summarisation. In practice, we also need to measure the understanding of users, models, and the source material. This talk introduces a reusable evaluation pattern that turns any document into a live-graded “exam engine” using Python tools including Docling, DeepEval, and Marimo.

We will build a stateful application that generates multiple-choice and free-text questions from complex documents, creates realistic distractors, and scores answers in real time using an LLM-as-judge pipeline. The demo is intentionally playful, but each component maps to a production concern: layout-aware ingestion (tables and figures), synthetic QA dataset creation, semantic grading, and interactive evaluation loops.

Attendees will learn how to move beyond passive RAG towards systems that benchmark knowledge, support training workflows, and enable human-in-the-loop evaluation.

Residential energy models like NREL’s ResStock generate the kind of data most humans run from: thousands of buildings, dozens of columns, and at least 8,760 rows per column. Great for research, but difficult for anyone who just wants to ask, “What happens to electricity demand in Texas if homes used solar water heating?” or “How do HVAC upgrades change my annual cooling costs in North Carolina?”

Join us for this session as a University of Texas energy researcher and a Red Hat engineer team up to see what large language models can realistically do with this kind of messy, domain-heavy data using Python. We’ll show how we sample, reshape, and describe large datasets so LLMs can help generate and refine pandas/DuckDB queries, explain upgrade scenarios in plain English, and guide non-experts through “what if” electrification questions. This and more, all while being honest about where the models break down and why humans still need to do the science.

AI coding agents are changing how data professionals work. But an AI agent chat session is a stream, a long conversation that scrolls on and on. A good notebook is something different: a sequence of distinct, well-structured transformations, each with an explanation and a visible result. How do you get from the chat stream to that? And how do you see the visualizations, the tables, charts, and diffs that make data work legible?

We'll trace the historical reasons why the programming notebook style developed, what problems it solves, and what problems it creates. Notebooks intermingle three valuable concepts: a live execution environment, a long-running process that caches state in memory, and a narrative log of exploration steps. The long-running process is the key. It's why data scientists use notebooks instead of Python scripts. But this coupling is also why notebooks are fragile, unreproducible, and impossible to productionize. And the kernel's implicit mutable state is a poor fit for AI agents. Unlike databases (explicit state, declarative interface, introspectable), a notebook kernel degrades as implicit state accumulates across cells.

This talk introduces the Deconstructed Notebook: a system that gives AI-agent-driven data work the structure and visualization of a notebook without the notebook's baggage. Claude writes the instructions in the terminal. The PyData Arrow stack, driven by Ibis and xorq, handles the compute. A browser companion renders tables, charts, diffs, and lineage live as the work iterates, organized into distinct steps, not a scrolling chat log. The key architectural insight is that automatic caching of expression results to disk replaces the notebook kernel's in-memory state, letting each step execute as a self-contained script while preserving the interactive, incremental workflow data scientists depend on. The system is built on xorq, an open-source library built on Ibis and Apache Arrow, but the design principles generalize. We'll demo the full workflow live and share what we learned about building post-notebook tooling for the age of AI agents.

Learn practical techniques for using LLMs to solve the data scarcity problem that plagues real-world ML projects. This talk demonstrates three production-ready approaches: synthetic generation, LoRA fine-tuning, and LLM-powered annotation to augment training datasets when you have abundant data for common cases but almost nothing for edge cases or emerging categories. Using a food review classification scenario, you'll see how to generate high-quality training data, when each technique works best, and critically, how to validate synthetic data to avoid amplifying errors. Perfect for practitioners facing the "we have 10k examples of X but zero for Y" problem.

Target Audience: Data scientists and ML engineers working on classification, NLP, or content moderation tasks who struggle with imbalanced or incomplete training datasets.

Takeaway: A decision framework for choosing between synthetic generation, fine-tuning, and LLM annotation, plus validation strategies to ensure data quality before retraining models.

Forecasts live in a high-dimensional space. They vary by origin date, prediction horizon, scenario assumptions, uncertainty, granularity and decision context. Treating them as a single artefact creates ambiguity, semantic drift and misaligned expectations.

In this talk, I’ll show how we reframed forecasting at Spotify as a structured prediction problem not just a modelling task. I’ll cover practical design patterns for representing forecast objects across origins and scenarios, handling probabilistic outputs, implementing hierarchical reconciliation and tracking lineage and versioning in Python-based systems.

Aimed at data scientists and ML engineers working with production systems, this talk offers a framework for thinking about forecast dimensionality and concrete implementation patterns you can apply in your own forecasting platforms.