A/B Testing Simulation

Learn to effectively run and analyze randomized experiments to inform product and market strategies

Instructor-led
What's Included?
  • Practice Mode: Learners engage in a single player on demand experience to master the basics
  • Tournament Mode: A competitive team-based experience
Skills you'll learn
  • Using A/B testing tools
  • Running rigorous business experiments to inform go to market decisions
  • Gain familiarity with the statistical reasoning behind A/B testing
Time commitments
Experience duration:
2-4 hours
Full details below

Gain mastery in the latest A/B Testing Techniques

Before launching a new product, most organizations run A/B tests so they can make data-driven choices about their go-to-market strategy. Decisions such as which marketing message works best with consumers, what is the best way to describe the product’s benefits and features in the marketplace, and what price is the consumer willing to pay? Those who aren’t taking advantage of these fast and simple experiments are not only operating in the dark, but are missing out on potential sales and revenue.

The world of A/B testing has rapidly evolved over the past decade, with new streams of data, new methods for analysis, and new techniques for customer targeting. However, students are learning about these developments in ways that are too theoretical, meaning that they aren’t developing an intuition about how to use A/B tests for real-world decision making. This simulation is designed to expose students to these concepts, with a key focus on highlighting the importance of the explore-exploit tradeoff—a fundamental tension between acting quickly on uncertain results and gathering more data to make more precise estimates.

A/B Testing Simulation

The simulation encourages students to think about A/B testing in ways that reflect practical tradeoffs in hypothesis testing, and can also be used to introduce multi-armed bandits (MABs) and Reinforcement Learning. Without requiring advanced statistics or a mathematics degrees, Wharton Interactive’s A/B Testing Simulation helps students learn about these core concepts in a deeply engaging and interactive simulated environment.

Playing the role of the eCommerce director of Nano, a company that has just launched its next generation smartphone, students must increase customer conversion rates to maximize profits. To evaluate what components of the Nanophone website are key for driving customer behavior, students have access to a realistic eCommerce test suite where, over the course of 12 time periods, they run a series of experiments testing multiple variations of each component. The in-game eCommerce dashboard is modelled after real-world testing interfaces, where students set up their experiments, gather data about customer conversion rates, evaluate that data, and make decisions about the go to market strategy for the Nano product based on what they learned.

There are two stages to the experience, allowing students to run through a single-player Practice Mode ahead of the team-based Tournament Mode, where they’ll compete against other teams in the class. In the practice mode, students are able to familiarize themselves with the test suite. The tournament mode is a different base market from the practice mode, meaning that consumer preferences and sensitivities will be different, thus challenging students to take the strategies they learned from the practice mode and adapt them to the new market in the tournament mode.

ABTesting-InterfaceKanBan.png

Learners must run tests, analyze the results and lock in their final choices

ABTesting-InterfaceAssessResults.png

Detailed analysis is provided on the performance of the different tests

An instructor debrief builds on the experience and ties the lessons of the simulation together, giving students critical tactical and strategic insights. In addition to comparing students’ performances to each other and comparing strategies, the simulation also allows users to see how modern reinforcement learning algorithms approach the A/B testing problem. Students should come away from the experience with solid intuition about the role of A/B testing in business decision making, a better understanding of how machine learning solutions handle marketing optimization problems, and appreciation for how and why managers should promote a culture of experimentation and testing within their organizations.

Our Partners

Wharton AI for Business

How our experiences work

Access teaching materials, support and notifications every step of the way

Setup

Configure for your learning objectives, set up classes in the experience

Players Prepare

Invite learners to enroll and set up their groups

Play

Run the experience and access support and notifications as you go

Debrief

Summarize the experience for your learners and the outcomes

Signup to unlock teaching notes and request a free demo *

Unlock extensive teaching notes, helper videos and free trial!

Create a ‘My Interactive’ account *This will unlock these across all of our experiences

Lessons of A/B Testing Simulation include:

  • How to use A/B testing tools to inform go-to-market product strategies
  • Running rigorous business experiments to make decisions that positively impact the bottom line
  • Gain an intuition about how much data to gather to make an effective decision
  • Gain familiarity with the statistical reasoning behind A/B testing
  • Understand the relationship between effect sizes and sample sizes
  • Develop an intuition for the exploration-exploitation tradeoff
  • Evaluate experimentation outcomes and effect sizes to make data informed decisions
  • Work with customer data and statistics in the context of e-commerce and web analytics
  • Practice gathering and evaluating data in a simulated environment
  • Gain exposure to the types of data and analyses that are used to make e-commerce decisions
  • Analytical thinking: the ability to analyze and frame problems
  • Learning orientation; develop a proactive learning mindset and a habit of questioning your assumptions through rigorous testing
  • Innovative thinking; develop the ability to improvise and address business problems and translate data into practice

Learning Objectives

  • How to use A/B testing tools to inform go-to-market product strategies
  • Running rigorous business experiments to make decisions that positively impact the bottom line
  • Gain an intuition about how much data to gather to make an effective decision
  • Gain familiarity with the statistical reasoning behind A/B testing
  • Understand the relationship between effect sizes and sample sizes
  • Develop an intuition for the exploration-exploitation tradeoff

Practice objectives

  • Evaluate experimentation outcomes and effect sizes to make data informed decisions
  • Work with customer data and statistics in the context of e-commerce and web analytics
  • Practice gathering and evaluating data in a simulated environment
  • Gain exposure to the types of data and analyses that are used to make e-commerce decisions

Thinking objectives

  • Analytical thinking: the ability to analyze and frame problems
  • Learning orientation; develop a proactive learning mindset and a habit of questioning your assumptions through rigorous testing
  • Innovative thinking; develop the ability to improvise and address business problems and translate data into practice

Authors

KartikHosanagar.jpg

Kartik Hosanagar

Professor of Operations, Information and Decisions

Read Kartik's Bio