The Urn of Mystery

At large companies there can be a fear to make decisions with the little data we have available. Statistical significance is always the gold standard, but even without it we can make an informed decision. The Urn of Mystery is a fun thought experiment to illustrate this point. In this post I will explain the problem and also demonstrate a Monte Carlo simulation that shows, even with one tiny piece of data we can make an informed decision that will drive impact for our large company. I first read about the Urn of Mystery in How to Measure Anything by Douglas Hubbard and I strongly recommend it.

Game Rules

The Urn of Mystery is a guessing game. If I can correctly guess, I win money. If I do not, you win money.

Imagine a ware house full of large urns. Each urn is filled with marbles that must either green or red. This would mean, for example, if one of the urns was filled with 15% red marbles, the remaining 85% of marbles would be green. The marbles in the urn are randomly distributed so the odds of getting a red marble in this example is 15%.

Assume that all distributions of marbles are possible and are equally likely amongst all the urns. For example, the chance of picking a random urn with 2% red marbles is the same as the chance of getting an urn with 50% red marbles. Because all marble distributions are equally likely and I have no prior information, my odds of picking the majority marble are 50%.

Now we can play our guessing game! I will pick one urn at random from the warehouse and I will bet on what the majority marble color is in that urn. I will give 2 to 1 odds on this wager and (to keep the math simple) you may wager $10 each time; in other words, I give you $20 if I lose and you pay me $10 if I win. I’ll ask you to play 100 rounds with me and I will pick a new urn at random each round. Should you take the bet?

Yes! Your expected winnings would be around $500 plus or minus $100.

Now let’s modify the rules to make things a little more interesting. Before I guess the majority marble color, I get to draw a single marble from the urn at random. I do so without viewing any other marbles in the urn. Should you take the bet?

Single Sample Majority Rule

It seems like this piece of information is not statistically significant. When I presented this question to a class of data scientist master students, few said it would make a difference. Those who did felt like it would be a 1-5% improvement in win rate from the first example. It turns out that because of something Hubbard calls the single sample majority rule, my chances of winning move to 75%. The single simple majority rule states, “Given maximum uncertainty about a population proportion — such that you believe the proportion could be anything between 0% and 100% with all values being equally likely — there is a 75% chance that single randomly selected sample is from the majority of the population.”

This is a simple outcome of using Bayes’ Theorem plus prior uncertainty. The game rules make it so that we are starting with the absolute most uncertainty possible; a 0% to 100% range on a uniform distribution. We might think the right question is, “What is the chance that the majority of marbles in the urn is green if a randomly drawn marble is green?” But instead we ask a simpler question and use Bayes’ Theorem to break it down.

Before I draw the first marble, we know that the chance of either marble being a majority is 50%. Let’s call the probability of the majority being green P(MG) and the probability that we draw a green marble P(DG). It should be apparent that the probability that the majority is green given that we drew green is the same as the probability of drawing a green marble given that the majority is green. In other words, P(MG) equals P(DG) and they cancel out in Bayes’ Theorem. P(MG|DG) = P(DG|MG) * P(MG) / P(DG) = P(DG|MG). The weighted average then of drawing green is (0.5 + 1.0) / 2 = 75%.

Monte Carlo Simulation

Maybe that last part feels too complicated or too much like a parlor trick. Let’s encode the Urn of Mystery game and see just how much we would win using the modified rules. The source code is available in Colab. We will walk through the source code and show results along the way. Let’s start with the first version of the game I described which would lead to you win about $500 dollars.

The first thing we need to do is show how we would pick one of the urns at random. We do this by creating a random uniform distribution.

import numpy as np

# Pick an urn out of the warehouse
urnGreenDistribution = np.random.uniform()
urnGreenDistribution

Next we demonstrate how we can pick a marble from this urn (i.e., random uniform distribution). This method returns either 1 (green) or 0 (red).

# Pick a marble from the urn
marbleColor = np.random.binomial(1, urnGreenDistribution, 1)[0] # 1 = Green # 0 = Red
marbleColor

Now we will create the code to execute one round of betting. To simplify things, in the Monte Carlo simulation I will simply pick green every time. This method returns your winnings after one round of betting.

def oneRoundOfBetting(): # Replacing odds of DG with MG
  urnGreenDistribution = np.random.uniform()
  marbleColor = np.random.binomial(1, urnGreenDistribution, 1)[0]
  if marbleColor == 1:
    return -10
  return 20

Now we can simulate 100 rounds as was stipulated in the first bet mentioned above. This code sums the results of betting 100 times and returns your total winnings. This form of the game is in your advantage so we expect a big positive number.

rounds = 100
results = [oneRoundOfBetting() for _ in range(rounds)]
sum(results)

When I ran this we got a result of 380 which is a positive result but is pretty unlucky as its outside the 500 plus-or-minus 100 range we expect. One play through of this game is insufficient to call it a Monte Carlo approach. Instead we will execute 1,000 of the 100 round games, resetting the winnings to zero each time to simulate a fresh game. We will plot the distribution of winnings from every simulation using Seaborn.

import pandas as pd
import seaborn as sea

def oneMonteCarloSimulation():
  rounds = 100
  results = [oneRoundOfBetting() for _ in range(rounds)]
  return sum(results)

simulations = 1000
simulationResults = [oneMonteCarloSimulation() for _ in range(simulations)]
dataFrame = pd.DataFrame(simulationResults)

sea.histplot(data=dataFrame)

The distribution of winnings from planning the Urn of Mystery game from 1,000 simulations. The x-axis shows the dollar amount of winnings and the y-axis shows the number of simulations that resulted in that dollar amount. 

Just as expected the most frequent amount is near $500, and many of the winnings are within +/- 100 of that amount. Since there never was a negative amount in our simulations you should feel extremely confident about taking this bet. Note that we can run the same simulation by replacing DG with MG using the code snippet below.

def oneRoundOfBetting(): # Replacing odds of DG with MG
  urnGreenDistribution = np.random.uniform()
  if urnGreenDistribution > 0.5:
    return -10
  return 20

Let’s modify the code to look at the impact of allowing me to peek at just one marble before I bet. My simulation will simply pick to bet on the color of whatever marble I see.

def oneRoundOfImprovedBetting():
  urnGreenDistribution = np.random.uniform()
  marbleColor = np.random.binomial(1, urnGreenDistribution, 1)[0]
  if marbleColor == 1:
    if urnGreenDistribution > 0.5:
      return -10
    return 20
  if urnGreenDistribution < 0.5:
    return -10
  return 20

def oneImprovedMonteCarloSimulation():
  rounds = 100
  results = [oneRoundOfImprovedBetting() for _ in range(rounds)]
  return sum(results)

simulations = 1000
simulationResults = [oneImprovedMonteCarloSimulation() for _ in range(simulations)]
dataFrame = pd.DataFrame(simulationResults)
sea.histplot(data=dataFrame)

The distribution of winnings from planning the Urn of Mystery game from 1,000 simulations where I get to peek at the marble first. The x-axis shows the dollar amount of winnings and the y-axis shows the number of simulations that resulted in that dollar amount. 

This single piece of information has made the game extremely unfavorable for you. In a majority of cases, there is a negative winnings amount. This is a great demonstration of how even tiny pieces of information can allow us to make informed, impactful decisions. This is especially true when the amount of prior knowledge that we have is limited; this is especially true in the Mystery Urn example since we are using a random uniform distribution.

This should encourage us when faced with a decision to use the information that we have to make the best possible one instead of saying that no decision could be made because the data is “not statistically significant.”