Background

The top data driven companies in the tech industry have implemented a culture of using experimentation to learn about their users and refine their products. But tracking the impact of A/B tests on your metrics is often time consuming and painful. Some companies, such as Airbnb have even gone as far as to build internal tools to assist in the process of reporting on experiment results. This is a clear signal that such a tool can be productized. Internally, we’d been mulling over this problem for a while, but the final push was when it became part of a must-win strategic deal. Uber specifically called out the need for a tool to help them analyze the impact of experiments on their metrics. Now was the time to bring the experiment reporting framework to life!

The Problem

How can we provide a quick and easy way for customers to understand how their A/B tests are doing all in one place, so they know which experiments are worth the effort?

Project Details

  • Role: Product Designer

  • Status: Beta release Jan 2020

  • Collaborators: 1 Product Manager, 1 Technical Lead, 2 Engineers

  • Tools Used: Sketch, Abstract, Framer

  • Methods Used: Wireframing, Prototyping

Project Goals

  1. Make it easy to track experiment results.

  2. Make it easy to understand the results of an experiment on the business. (Experiments can be a feature, marketing, initiative/team or other activity.)

  3. Enable companies to run hundreds of experiments concurrently.

  4. Be compatible with (and promote) rigorous experimentation practices to earn trust of sophisticated users.

Design Scope

Design a reporting UI that lets a customer quickly and understandably see the results of any experiment for:

  • MVP

  • Long-term Direction

What do users want to learn from Experiments?

  1. Significance: Is there a statistically significant difference between the control & variant?

  2. Confidence score: What’s the probability that any difference is not due to chance?

  3. Confidence over time: How has confidence changed over the chosen time period?

  4. Winner: Did the control or the variant “win” (have the higher rate of events per user per day)? By how much (overall?)

  5. Delta over time: How did the delta (difference between the 2 variants) change over time?

Success Metrics & Results

  1. Delivery of MVP within 1 week turn-around  ✅

  2. Outcome of Uber deal: We WON! ✅

    • Winning the deal was a huge morale boost for the company and had a significant impact on the bottom line.

  3. Beta Acquisition: 70% of all report views are experiments (averaged over 30 days)

  4. Overall Retention: 13% Viewed Experiments weekly addiction

The Final Design

The Experimentation report analyzes how AB test variants impact metrics on any dashboard. By leveraging everything Mixpanel is already tracking, it removes all the heavy lifting. Running an experiment is as easy as selecting an experiment, dashboard, and the experiment’s variants in the query builder. MP has a messaging and mobile AB testing tool, so variants can be any multi-variant message, mobile AB test, or experiment that exists in the customer’s MP implementation. Experiments can be super flexible, so you can even create a custom experiment (meaning the variants can be any cohort, user property, or event property).

Future Improvements

Mixpanel is in the process of onboarding all Uber employees and will be gathering more feedback to determine future improvements beyond the Beta release.