PyData Amsterdam 2026

Target Audience

• Roles: Data Scientists, Machine Learning Scientists, Applied Scientists, Economists.
• Experience Level: Intermediate.

Required Background Knowledge
Familiarity with standard A/B testing, introductory linear regression, and basic causal inference concepts (e.g., confounders, selection bias). Basic knowledge of Python’s scientific stack (e.g., scikit-learn).

Talk Type and Approach
Simulation-driven and Practical. The session will blend a live-coded Python simulation with mathematical intuition (DAGs) and a real-world product case study.

Audience Takeaways

• Conceptual: Visually understand why standard observational methods (like Propensity Score Matching or naive Doubly Robust ML) structurally fail for opt-in features due to collider bias.
• Strategic: Learn how to disentangle a product's intrinsic value from its marketing reach, preventing the business from abandoning highly effective features simply due to poor adoption UX.
• Practical: Gain a Python blueprint for implementing Randomized Encouragement Design (RED) and the Interactive Instrumental Variable Model (DoubleMLIIVM) to handle heavy-tailed, highly skewed platform data.

Topic and Relevance
Across online marketplaces, measuring the impact of voluntary features—whether it is a customer opting into a loyalty program, an e-commerce seller adopting a pricing tool, or a vacation host enabling "instant booking"—is notoriously difficult. These opt-in mechanics introduce a severe "Trilemma" for Data Science: Voluntary Adoption (the opt-in barrier), Extreme User Heterogeneity, and Finite Sample Sizes within eligible cohorts. This Trilemma creates a paradox: standard A/B tests severely dilute the Intention-to-Treat (ITT) effect across non-adopters, while observational methods introduce fatal selection bias. This talk provides a rigorous framework to solve this trilemma, allowing organizations to disentangle true product quality from adoption bottlenecks.

Technical Depth & Educational Value
This talk moves beyond theory by using code-driven simulations to visualize causal failures, providing a robust structural solution for each part of the Trilemma:

• Solving Voluntary Adoption (Simulation & RED): We will use a simulation to demonstrate why conditioning on adoption—via Propensity Score Matching or naive ML—fails. By forcing a comparison between treated adopters and control matches, we inadvertently compare a mixed-motivation group against a "super-motivation" group, resulting in Collider Bias. To escape this, we introduce Randomized Encouragement Design (RED) as a valid Instrumental Variable (IV) to bypass the collider and target the Local Average Treatment Effect (LATE).
• Solving Heterogeneity (The DoubleML Engine): While RED provides the causal framework, standard Two-Stage Least Squares (2SLS) assumes linear, additive effects. We will explain why this "Linearity Trap" fails for highly skewed platform data (where the impact on a highly engaged "whale" user is exponentially different than on a casual user). We introduce Double Machine Learning (DoubleML) to handle this extreme non-linear heterogeneity.
• Solving Finite Samples: Because we often cannot simply "run the test longer" for time-bound campaigns or limited user cohorts, we must maximize statistical power. We will showcase how to parameterize the DoubleMLIIVM model using Tweedie Regressors. This specifically models the zero-inflated and right-skewed nature of platform metrics (like spend or total bookings), drastically reducing residual variance to achieve precise causal estimates.

Anticipated Reviewer Questions
Why not just use rank-based tests (e.g., Mann-Whitney U) to handle the extreme variance and outliers of this data? We will explicitly address this in the talk as the "Metric Alignment Problem." While robust to outliers, rank tests evaluate whether the median shifts. In a business context, total volume (the sum of revenue or bookings) is the primary success metric. A rank-based test could show a statistically significant shift by helping thousands of low-activity users, while actual total volume remains flat due to the variance of "whale" users. Furthermore, RED + DoubleML allows us to estimate the causal lift in actual business units, which is strictly required for ROI analysis.

Scope and Structure (30 Minutes Total)

• Minutes 0-5: The Business Context & The Trilemma. Introduce the opt-in paradox across platform ecosystems. Define the "Trilemma" preventing us from measuring true ROI: Voluntary Adoption, Extreme Heterogeneity, and Finite Sample Sizes.
• Minutes 5-15: The Measurement Trap & RED Framework. Using a Python simulation, we visualize the failure of standard A/B tests (dilution) and observational models (Collider Bias). We solve this conceptual hurdle by introducing Randomized Encouragement Design (RED) to create a valid instrument.
• Minutes 15-20: The Variance Problem & DoubleML. We explain why standard IV estimation (2SLS) isn't enough for highly skewed user data. We walk through the DoubleML Python implementation (DoubleMLIIVM), highlighting the use of Tweedie Regressors to handle zero-inflated data and recover the precise LATE.
• Minutes 20-25: Empirical Case Study. We apply the full RED + DoubleML framework to a real-world product decision, demonstrating step-by-step how it recovered the signal from the noise and saved a high-potential feature from being abandoned due to poor UX discovery.
• Minutes 25-30: Summary, Learnings & Q&A. A quick recap of the architectural blueprint (the "how-to" for attendees). Open the floor for Q&A.