2026-09-10 –, Main stage
Standard A/B tests assume that treating user A has no effect on user B (SUTVA). In highly-networked products - where users message, call, share content with each other or buy/sell from each other - this assumption breaks down. The result: experiments that systematically underestimate, miss entirely, or even produce directionally wrong estimates of the true impact of product changes.
This talk surveys the landscape of approaches for dealing with network effects in experimentation: from simply ignoring them (and when you might get away with it), through bi-directional metrics, translation coefficients, double-sided gating, and cluster-based randomization, to bipartite experimental design. For each approach, we'll discuss the intuition, when it's appropriate, and its practical trade-offs.
Target audience: Data scientists, analysts, or any product person who runs A/B tests and wants to go beyond naive user-level randomization. Familiarity with basic experimentation concepts (treatment/control, statistical significance) is assumed; no background in causal inference or graph theory is required. Concepts are built up visually and intuitively.
Key takeaways:
- Why ignoring network effects can lead to biased, incomplete, or even directionally wrong experiment results
- A practical decision framework for choosing the right measurement approach based on your product's interaction model and constraints
- How bipartite estimators can measure spillover effects while respecting user privacy constraints
This talk surveys approaches for measuring network effects in A/B tests, grounded in practical experience running experiments in highly-networked products. Each method is presented with its intuition, applicability, and real-world trade-offs. Not as a linear progression, but as a toolkit where the right choice depends on the product context, technical feasibility, and privacy constraints.
Outline:
Why this matters (2 min). Experiments exist to quantify trade-offs. When network effects are present but unmeasured, we're making launch decisions on incomplete, or misleading, data. Concrete example: a UI change that boosts one behavior while suppressing another through indirect exposure, where a naive A/B test shows a positive result but the true net effect is negative.
The core problem (3 min). SUTVA and why it breaks in networked products. Visual intuition for how treatment "leaks" through user interactions. Why the measured effect in a standard A/B test can be an underestimate, an overestimate, or even directionally wrong.
The toolkit (16 min). Six approaches, each with trade-offs:
- Ignoring network effects (YOLO): when it's a reasonable approximation and when it'll burn you. (2 min)
- Bi-directional metrics: logging interactions from both sender and receiver, then summing effects. Intuitive and practical for 1:1 interactions. (2 min)
- Translation coefficients: estimating multipliers that convert direct impact to ecosystem-level impact. Necessary for 1:many features and engagement metrics. (2 min)
- Double-sided gating: ensuring control users aren't exposed to the feature via interactions with treated users. When it works, when it's technically or otherwise infeasible (3 min)
- Cluster-based randomization: randomizing at the group/region/cluster level to contain spillovers. Trade-offs around power, cluster definition, and residual leakage. (3 min)
- Bipartite analysis: modeling the experiment as a bipartite graph with treatment units on one side and outcome units on the other, using exposure-reweighted linear estimation (Harshaw et al., 2021) to recover the total effect. Will also cover some practical innovations in implementing this approach in practice (4 min)
Decision framework & takeaways (4 min). A practical guide for choosing the right approach based on interaction type (1:1 vs 1:many), technical feasibility, privacy requirements, and statistical power considerations.
Q&A (5 min)
Dror A. Guldin is a Senior Staff Data Scientist at Meta, where he focuses on experimentation methodology and product analytics for networked products since 2018.
Based in Amsterdam, he spoke at PyData Amsterdam 2023 on metrics and KPIs. These days he spends a lot of time thinking about what happens to data roles when AI gets good enough to do most of the execution, and whether that's exciting or terrifying (current verdict: probably both).