1. Project background

Cookie Cats is a hugely popular mobile puzzle game developed by Tactile Entertainment. It's a classic "connect three"-style puzzle game where the player must connect tiles of the same color to clear the board and win the level. It also features singing cats. Check out this short demo:

As players progress through the levels of the game, they will occasionally encounter gates that force them to wait a non-trivial amount of time or make an in-app purchase to progress. In addition to driving in-app purchases, these gates serve the important purpose of giving players an enforced break from playing the game, hopefully resulting in that the player's enjoyment of the game being increased and prolonged.

But where should the gates be placed? Initially the first gate was placed at level 30. In this project, we're going to analyze an AB-test where we moved the first gate in Cookie Cats from level 30 to level 40. In particular, we will look at the impact on player retention.

Data Description from Aurelia Sui's notebook
The data is from 90,189 players that installed the game while the AB-test was running. The variables are:

• userid - a unique number that identifies each player.
• version - whether the player was put in the control group (gate_30 - a gate at level 30) or the test group (gate_40 - a gate at level 40).
• sum_gamerounds - the number of game rounds played by the player during the first week after installation
• retention_1 - did the player come back and play 1 day after installing?
• retention_7 - did the player come back and play 7 days after installing?

When a player installed the game, he or she was randomly assigned to either gate_30 or gate_40.

2. Packages

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

from scipy import stats

3. Configurations

import warnings
warnings.filterwarnings('ignore')

DATA_PATH = 'cookie_cats.csv'

# confident interval
ALPHA = 0.05

4. Prepare the data

df = pd.read_csv(DATA_PATH)
df.head()

	userid	version	sum_gamerounds	retention_1	retention_7
0	116	gate_30	3	False	False
1	337	gate_30	38	True	False
2	377	gate_40	165	True	False
3	483	gate_40	1	False	False
4	488	gate_40	179	True	True

df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 90189 entries, 0 to 90188
Data columns (total 5 columns):
 #   Column          Non-Null Count  Dtype 
---  ------          --------------  ----- 
 0   userid          90189 non-null  int64 
 1   version         90189 non-null  object
 2   sum_gamerounds  90189 non-null  int64 
 3   retention_1     90189 non-null  bool  
 4   retention_7     90189 non-null  bool  
dtypes: bool(2), int64(2), object(1)
memory usage: 2.2+ MB

df.describe()

	userid	sum_gamerounds
count	9.018900e+04	90189.000000
mean	4.998412e+06	51.872457
std	2.883286e+06	195.050858
min	1.160000e+02	0.000000
25%	2.512230e+06	5.000000
50%	4.995815e+06	16.000000
75%	7.496452e+06	51.000000
max	9.999861e+06	49854.000000

# Check for missing values
print('Number of missing values: ', df.isnull().sum().sum())

Number of missing values:  0

# check for duplicates
print('Number of duplicates: ', df.duplicated().sum())

Number of duplicates:  0

# check unique of userid
print('All userid of dataset is unique?', df['userid'].nunique() == len(df))

All userid of dataset is unique? True

5. Analyzing the data

sns.countplot(x='version', data=df)
plt.xlabel('Version')
plt.ylabel('Number of players')
plt.title('Number of players in each version')
for p in plt.gca().patches:
  height = int(p.get_height())
  plt.gca().annotate(f'{height:,}', (p.get_x() + p.get_width() / 2, height), ha='center', va='bottom')
plt.show()

sns.boxplot(data=df, y='sum_gamerounds', x='version')
plt.xlabel('Version')
plt.ylabel('Number of gamerounds')
plt.title('Number of gamerounds by version (before removing outliers)')
plt.show()

df['sum_gamerounds'].value_counts().sort_index()

sum_gamerounds
0        3994
1        5538
2        4606
3        3958
4        3629
         ... 
2294        1
2438        1
2640        1
2961        1
49854       1
Name: count, Length: 942, dtype: int64

49854 / 2961

16.836879432624112

Note: There are one player that played 49,854 rounds of the game in a week. This is a lot more than 16.84 times and second highest player. This player is an outlier and will be removed from the analysis.

df = df[df['sum_gamerounds'] < df['sum_gamerounds'].max()]

sns.boxplot(data=df, y='sum_gamerounds', x='version')
plt.xlabel('Version')
plt.ylabel('Number of gamerounds')
plt.title('Number of gamerounds by version (after removing outliers)')
plt.show()

df.groupby('sum_gamerounds')['userid'].count()[:100].plot()
plt.xlabel('Number of gamerounds')
plt.ylabel('Number of players')
plt.title('The number of players that played 0-100 game rounds during the first week')
plt.show()

Note: There are 3994 players never played the game after installing during the first week

sns.countplot(data=df[df['sum_gamerounds'] == 0], x='version')
plt.xlabel('Version')
plt.ylabel('Number of players')
plt.title('Number of players that never played the game after installing')
for p in plt.gca().patches:
  height = int(p.get_height())
  plt.gca().annotate(f'{height:,}', (p.get_x() + p.get_width() / 2, height), ha='center', va='bottom')
plt.show()

# remove players that never played the game after installing
df = df[df['sum_gamerounds'] > 0]

df.groupby('sum_gamerounds')['userid'].count()[:100].plot()
plt.xlabel('Number of gamerounds')
plt.ylabel('Number of players')
plt.title('The number of players that played 1-100 game rounds during the first week')

quantiles = [0.25, 0.5, 0.75]
quantile_labels = ['25th percentile', '50th percentile (Median)', '75th percentile']
x_values = [df['sum_gamerounds'].quantile(q) for q in quantiles]
for i, percentage in enumerate(quantiles):
  x_value = x_values[i]
  plt.axvline(x_value, color='red', linestyle='dashed')
  plt.text(x_value + 1, plt.ylim()[1] / 4, f'{quantile_labels[i]}', color='red', rotation=90)

x_values = [0] + x_values + [100]
plt.xticks(x_values)
plt.show()

Note: We lost 50% of players after 18 rounds

df.groupby('version')['sum_gamerounds'].agg(['mean', 'median'])

	mean	median
version
gate_30	53.667766	18.0
gate_40	53.728357	18.0

6. AB Testing

sns.barplot(y=df[['retention_1', 'retention_7']].mean().values, x=['1st day', '7th day'])
plt.xlabel('Retention')
plt.ylabel('Retention rate')
plt.title('Retention rate 1 day and 7 days')
for p in plt.gca().patches:
  height = p.get_height()
  plt.gca().annotate(f'{height:.2f}', (p.get_x() + p.get_width() / 2, height), ha='center', va='bottom')
plt.show()

6.1. Retention 1 day

A common metric in the video gaming industry for how fun and engaging a game is 1-day retention: the percentage of players that comes back and plays the game one day after they have installed it. The higher 1-day retention is, the easier it is to retain players and build a large player base.

df.groupby('version')['retention_1'].mean()

version
gate_30    0.467541
gate_40    0.462171
Name: retention_1, dtype: float64

for version in df['version'].unique():
  percentage = df[df['version'] == version]['retention_1'].mean() * 100
  print(f'{percentage:.2f}% of players who assigned to {version} version came back the next day')

46.75% of players who assigned to gate_30 version came back the next day
46.22% of players who assigned to gate_40 version came back the next day

# H0: distribution is normal
# H1: distribution is not normal

ntA = stats.shapiro(df[df['version'] == 'gate_30']['retention_1'])[1] < ALPHA
ntB = stats.shapiro(df[df['version'] == 'gate_40']['retention_1'])[1] < ALPHA

if not ntA and not ntB:
  print('Both distributions are normal')
else:
  print('Both distributions are not normal')

Both distributions are not normal

# H0: retention rate 1 day of version gate_30 is equal to retention rate 1 day of version gate_40
# H1: retention rate 1 day of version gate_30 is greater than retention rate 1 day of version gate_40

_, pvalue = stats.mannwhitneyu(
  df[df['version'] == 'gate_30']['retention_1'],
  df[df['version'] == 'gate_40']['retention_1'],
  alternative='greater'
)
print(f'p-value: {pvalue:.4f}')

if pvalue < ALPHA:
  print('Reject H0')
else:
  print('Fail to reject H0')

p-value: 0.0570
Fail to reject H0

With 95% confidence interval, there is evidence that 1-day retention of gate_30 is equal to gate_40.

6.2. Retention 7 days

There is a high probability that 1-day retention is better when the gate is at level 30. However, since players have only been playing the game for one day, it is likely that most players haven't reached level 30 yet. That is, many players won't have been affected by the gate, even if it's as early as level 30.

But after having played for a week, more players should have reached level 40, and therefore it makes sense to also look at 7-day retention.

df.groupby('version')['retention_7'].mean()

version
gate_30    0.198424
gate_40    0.190321
Name: retention_7, dtype: float64

for version in df['version'].unique():
  percentage = df[df['version'] == version]['retention_7'].mean() * 100
  print(f'{percentage:.2f}% of players who assigned to {version} version came back after 7 days')

19.84% of players who assigned to gate_30 version came back after 7 days
19.03% of players who assigned to gate_40 version came back after 7 days

# H0: distribution is normal
# H1: distribution is not normal

ntA = stats.shapiro(df[df['version'] == 'gate_30']['retention_7'])[1] < ALPHA
ntB = stats.shapiro(df[df['version'] == 'gate_40']['retention_7'])[1] < ALPHA

if not ntA and not ntB:
  print('Both distributions are normal')
else:
  print('Both distributions are not normal')

Both distributions are not normal

# H0: retention rate 7 days of version gate_30 is equal to retention rate 1 day of version gate_40
# H1: retention rate 7 days of version gate_30 is greater than retention rate 1 day of version gate_40

_, pvalue = stats.mannwhitneyu(
  df[df['version'] == 'gate_30']['retention_7'],
  df[df['version'] == 'gate_40']['retention_7'],
  alternative='greater'
)
print(f'p-value: {pvalue:.4f}')

if pvalue < ALPHA:
  print('Reject H0')
else:
  print('Fail to reject H0')

p-value: 0.0013
Reject H0

With 95% confidence interval, there is strong evidence that 7-day retention is greater when the gate is at level 30 than when it is at level 40.

7. Conclusion

After analyzing the data and performing some A/B tests, we can conclude that:

• 3994 players never played the game after installing during the first week (4.43%)
• 50% of players quit after playing 18 rounds in the first week
• 54% of players quit after 1 day
• 71% of players quit after 7 days
• There is evidence that 1-day retention of gate_30 is equal to gate_40
• There is strong evidence that 7-day retention is greater when the gate is at level 30 than when it is at level 40