2024-07-11 –, Else (1.3)
Conventional A/B testing often falls short in industries such as airlines, ride-sharing, and delivery services, where challenges like small samples and subtle effects complicate testing new features. Inspired by its significant impact in leading companies like Uber, Lyft, and Doordash, we introduce the switchback design as a practical alternative to conventional A/B testing.
By addressing small sample size limitations and the need to detect subtle effects quickly, this approach boosts statistical power while reducing variability and interference. We guide the audience through the challenges of marketplace experimentation and implementing this approach, period length optimization and switch frequency using a case study from the airline industry.
In this talk, we introduce switchback design, a method that addresses key challenges in marketplace experimentation faced by sectors like airlines and ride-sharing. It addresses traditional A/B testing's limitations on small samples and subtle effects, highlighted by successes in companies like Uber, Lyft, and Doordash. This approach, crucial for accurate effect measurement, aims to boost decision-making and operational optimization, supported by a practical case study.
Talk outline:
- Introduction (5 mins): Quick intro to our expertise and the topic's relevance.
- Challenges (5 mins): Overview of marketplace experimentation challenges.
- Switchback Design (5 mins): Basics of switchback design vs. traditional A/B testing.
- Case Study (10 mins): Real-world application in the airline industry, showcasing benefits and drawbacks.
- Key Takeaways (5 mins): Summary of our talk.
- Q&A (10 mins): Open discussion for clarifications and further insights.
The intended audience: Targeted at data scientists, data analysts, product managers, and anyone interested in data-driven decision-making. Ideal for those curious about experimentation and causal inference in sectors like tech and e-commerce.
No background needed: Open to all levels, no prior knowledge needed. Concepts will be explained simply, focusing on practical insights without complex math.
The takeaway for the audience: The audience will understand the effectiveness of switchback design in overcoming marketplace experimentation hurdles, such as dealing with small samples and subtle effects. They'll learn from the success stories of top firms, understand practical steps for executing switchback experiments, and recognize its advantages over conventional A/B testing for more informed decisions and improved results.