-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathtoc-ab-testing-vs-multi-armed-bandit.html
95 lines (86 loc) · 4 KB
/
toc-ab-testing-vs-multi-armed-bandit.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8" />
<title>Multi-armed bandit vs. A/B testing for web service optimization</title>
<link rel="stylesheet" href="/theme/css/main.css" />
<!--[if IE]>
<script src="https://html5shiv.googlecode.com/svn/trunk/html5.js"></script>
<![endif]-->
</head>
<body id="index" class="home">
<header id="banner" class="body">
<h1><a href="/">And yet it moves! </a></h1>
<nav><ul>
<li class="active"><a href="/category/data-processing.html">Data processing</a></li>
</ul></nav>
</header><!-- /#banner -->
<section id="content" class="body">
<article>
<header>
<h1 class="entry-title">
<a href="/toc-ab-testing-vs-multi-armed-bandit.html" rel="bookmark"
title="Permalink to Multi-armed bandit vs. A/B testing for web service optimization">Multi-armed bandit vs. A/B testing for web service optimization</a></h1>
</header>
<div class="entry-content">
<footer class="post-info">
<abbr class="published" title="2016-04-28T12:26:00+01:00">
Published: jeu. 28 avril 2016
</abbr>
<address class="vcard author">
By <a class="url fn" href="/author/gael.html">Gaël</a>
</address>
<p>In <a href="/category/data-processing.html">Data processing</a>.</p>
</footer><!-- /.post-info --> <p><em>This post is the first of a series dedicated to comparing the A/B testing
method with the multi-armed bandit strategy for the purpose of optimizing
a website.
The Jupyter notebooks with all the code, discussions and conclusions presented
in this series can be found <a href="">here</a>.</em></p>
<p>I recently stumbled across a <a href="https://redd.it/4dlioz">reddit link</a>
to a (now 3-year-old) blog post about the multi-armed bandit
strategy, a method claimed to maximize the click-through rate
even during the data-gathering stage of the optimization process.</p>
<p>That <a href="http://stevehanov.ca/blog/index.php?id=132">blog post</a> is rather
disappointing. Most of the claims
seemed fishy and given the tone and the lack of demonstrations or proofs,
I thought this would be the opportunity to dig into the subject and
compare the methods.</p>
<p>The main purpose (and TL;DR) of this blog series is to restate
the obvious:</p>
<blockquote>
<ul>
<li>A/B testing is a very efficient method designed to answer one question:
"Is there a difference?". It is also very efficient to quantify this
difference in an unbiased way.</li>
<li>The multi-armed bandit strategy was designed to maximize the reward
(i.e. the number of clicks) throughout the data-gathering process.</li>
</ul>
</blockquote>
<p>A/B testing does so at the expense of some clicks and the multi-armed bandit
does so at the expense of correctness and sensitivity. That is all there is
to know... or, is it?</p>
<p>If you are not interested in code (Python), maths or very long posts, feel
free to read the summary <a href="">here</a>.
For the others, here is the outline of this series:</p>
<h1 id="table-of-content">Table of content</h1>
<ol>
<li>Presentation of the two methods</li>
<li>Construction of the Monte-Carlo models</li>
<li>The pros of the A/B method</li>
<li>The pros of the multi-armed bandit strategy</li>
<li>Conclusion</li>
</ol>
<p>Enjoy!</p>
</div><!-- /.entry-content -->
</article>
</section>
<section id="extras" class="body">
</section><!-- /#extras -->
<footer id="contentinfo" class="body">
<address id="about" class="vcard body">
Proudly powered by <a href="http://getpelican.com/">Pelican</a>, which takes great advantage of <a href="http://python.org">Python</a>.
</address><!-- /#about -->
<p>The theme is by <a href="http://coding.smashingmagazine.com/2009/08/04/designing-a-html-5-layout-from-scratch/">Smashing Magazine</a>, thanks!</p>
</footer><!-- /#contentinfo -->
</body>
</html>