A/B Testing Simulation

For instructors
Learning time
2-4 hours
Authored by:

Learn to effectively run and analyze randomized experiments to inform product and market strategies

What you'll teach

  • Practice Mode: Learners engage in a single player on demand experience to master the basics

    Tournament Mode: A competitive team-based experience

  • Using A/B testing tools

  • Running rigorous business experiments to inform go to market decisions

  • Gain familiarity with the statistical reasoning behind A/B testing

AB Testing
Ready to teach?

You need to become a verified instructor to teach our experiences.

Become a verified instructor*
*Unlock teaching materials, helper videos and prices per learner

About A/B Testing Simulation

Gain mastery in the latest A/B Testing Techniques

Before launching a new product, most organizations run A/B tests so they can make data-driven choices about their go-to-market strategy. Decisions such as which marketing message works best with consumers, what is the best way to describe the product’s benefits and features in the marketplace, and what price is the consumer willing to pay? Those who aren’t taking advantage of these fast and simple experiments are not only operating in the dark, but are missing out on potential sales and revenue.

The world of A/B testing has rapidly evolved over the past decade, with new streams of data, new methods for analysis, and new techniques for customer targeting. However, students are learning about these developments in ways that are too theoretical, meaning that they aren’t developing an intuition about how to use A/B tests for real-world decision making. This simulation is designed to expose students to these concepts, with a key focus on highlighting the importance of the explore-exploit tradeoff—a fundamental tension between acting quickly on uncertain results and gathering more data to make more precise estimates.

Rather than teaching notions of significance testing that are not ideal for many business problems , students are encouraged to think about A/B testing in ways that better reflect practical tradeoffs in hypothesis testing. Without requiring advanced statistics or mathematics degrees, Wharton Interactive’s Digital Marketing: A/B Testing Simulation helps students learn about these core concepts in an engaging and interactive simulated environment.

Playing the role of the eCommerce director of Nano, a company that has just launched its next generation smartphone, students must increase customer conversion rates to maximize profits. To evaluate what components of the Nanophone website are key for driving customer behavior, students have access to a realistic eCommerce test suite where, over the course of 12 time periods, they run a series of experiments testing multiple variations of each component. The in-game eCommerce dashboard is modelled after real-world testing interfaces, where students set up their experiments, gather data about customer conversion rates, evaluate that data, and make decisions about the go to market strategy for the Nano product based on what they learned.

There are two stages to the experience, allowing students to run through a single-player Practice Mode ahead of the team-based Tournament Mode, where they’ll compete against other teams in the class. In the practice mode, students are able to familiarize themselves with the test suite. The tournament mode is a different base market from the practice mode, meaning that consumer preferences and sensitivities will be different, thus challenging students to take the strategies they learned from the practice mode and adapt them to the new market in the tournament mode.

IMPORTANT: Remember to create a class for each of your sections, and then add this game to each class

AB Testing LinkedIn.jpg ABTesting-InterfaceKanBan.png ABTesting-InterfaceAssessResults.png

Course authors

KartikHosanagar.jpg
Kartik Hosanagar Professor of Operations, Information and Decisions

What you'll teach

  • How to use A/B testing tools to inform go-to-market product strategies
  • Running rigorous business experiments to make decisions that positively impact the bottom line
  • Gain an intuition about how much data to gather to make an effective decision
  • Gain familiarity with the statistical reasoning behind A/B testing
  • Understand the relationship between effect sizes and sample sizes
  • Develop an intuition for the exploration-exploitation tradeoff
  • Evaluate experimentation outcomes and effect sizes to make data informed decisions
  • Work with customer data and statistics in the context of e-commerce and web analytics
  • Practice gathering and evaluating data in a simulated environment
  • Gain exposure to the types of data and analyses that are used to make e-commerce decisions
  • Analytical thinking: the ability to analyze and frame problems
  • Learning orientation; develop a proactive learning mindset and a habit of questioning your assumptions through rigorous testing
  • Innovative thinking; develop the ability to improvise and address business problems and translate data into practice
  • How to use A/B testing tools to inform go-to-market product strategies
  • Running rigorous business experiments to make decisions that positively impact the bottom line
  • Gain an intuition about how much data to gather to make an effective decision
  • Gain familiarity with the statistical reasoning behind A/B testing
  • Understand the relationship between effect sizes and sample sizes
  • Develop an intuition for the exploration-exploitation tradeoff
  • Evaluate experimentation outcomes and effect sizes to make data informed decisions
  • Work with customer data and statistics in the context of e-commerce and web analytics
  • Practice gathering and evaluating data in a simulated environment
  • Gain exposure to the types of data and analyses that are used to make e-commerce decisions
  • Analytical thinking: the ability to analyze and frame problems
  • Learning orientation; develop a proactive learning mindset and a habit of questioning your assumptions through rigorous testing
  • Innovative thinking; develop the ability to improvise and address business problems and translate data into practice

Ready to run a new kind of learning experience?

Access all our teaching materials, toolkits and walkthroughs with a free Verified Instructor account.