Dror A. Guldin
Dror A. Guldin is a Senior Staff Data Scientist at Meta, where he focuses on experimentation methodology and product analytics for networked products since 2018.
Based in Amsterdam, he spoke at PyData Amsterdam 2023 on metrics and KPIs. These days he spends a lot of time thinking about what happens to data roles when AI gets good enough to do most of the execution, and whether that's exciting or terrifying (current verdict: probably both).
Session
Standard A/B tests assume that treating user A has no effect on user B (SUTVA). In highly-networked products - where users message, call, share content with each other or buy/sell from each other - this assumption breaks down. The result: experiments that systematically underestimate, miss entirely, or even produce directionally wrong estimates of the true impact of product changes.
This talk surveys the landscape of approaches for dealing with network effects in experimentation: from simply ignoring them (and when you might get away with it), through bi-directional metrics, translation coefficients, double-sided gating, and cluster-based randomization, to bipartite experimental design. For each approach, we'll discuss the intuition, when it's appropriate, and its practical trade-offs.
Target audience: Data scientists, analysts, or any product person who runs A/B tests and wants to go beyond naive user-level randomization. Familiarity with basic experimentation concepts (treatment/control, statistical significance) is assumed; no background in causal inference or graph theory is required. Concepts are built up visually and intuitively.
Key takeaways:
- Why ignoring network effects can lead to biased, incomplete, or even directionally wrong experiment results
- A practical decision framework for choosing the right measurement approach based on your product's interaction model and constraints
- How bipartite estimators can measure spillover effects while respecting user privacy constraints