How Etsy Handles “Peeking” in A/B Testing

3:45pm - 4:10pm on Friday, October 5 in PennTop South

Kelly Shen

Audience Level:

All

Watch:

https://youtu.be/hI7Mb8oBIG4

Overview

Etsy relies heavily on experimentation to improve our decision-making process. In this talk, I will present how Etsy handles data peeking and how we use Python to help us investigate and assess the problem on our platform.

Description

At Etsy, we leverage our internal A/B testing tool when we launch new campaigns, polish the look and feel of our site, or even make changes to our search and recommendation algorithms. As our experimentation platform scales and the velocity of experimentation increases rapidly across the company, we also face a number of challenges. In this talk, I will talk about how we utilize Python to investigate and evaluate one of the problems, “peeking” at results early in order to detect maximum significance with minimum sample size.

We assessed the overall problem and how it affected experiments at Etsy. We then evaluated a few solutions that have been proposed and applied in industry and academia, keeping in mind the unique challenges we face as a fast-paced e-commerce company. After going through the analysis and evaluation, I will discuss the approach we at Etsy took to tackle the peeking problem.