I've been working on optimising revenue on my Countdown website the last few days.

I have had a Countdown solver tool online since about 2009. It is to this day the most popular website I have ever made, it currently gets about 70,000 pageviews per month. The site has been earning revenue from AdSense for years.

Up until last week the site was just 2 static HTML pages: one for the Countdown Solver and one for the Countdown Practice game. That didn't give me much opportunity to run experiments on the site, and I never really had the inclination to try. It was basically a web program.

But now, LLMs to the rescue. I now have a Python Flask application serving the site, and a lot more related information pages for people to read. And serving the site with an actual web application means I can run experiments like A/B tests to see if there are changes I can make to the site that cause people to stick around longer. And therefore look at more ads.

A good alternative to A/B tests is multi-armed bandits.

Instead of splitting your traffic equally between the different variants you want to try out, and then waiting to collect data, and then picking a winner, you have the site automatically determine the winner on a continuous basis, and show the winner 90% of the time (greedy), and a random selection the rest of the time (epsilon).

I am using a multi-armed bandit to decide which "info" pages to suggest at the bottom of each page, and also to decide which Amazon Affiliate links to show. (Yes this is all very grubby, what can you do?). The "winner" is the choice that has the highest click-through rate. So for each choice we need to track how many times we've displayed it, and how many times it's been clicked on. If your reward function is more complicated you might find it more complicated.

Steve Hanov's blog post on multi-armed bandits, linked above, goes over the case where you might worry that a particular variant gets a click early, just by random chance, which gives it an apparent high click-through rate, which then means the site is going to show that variant to everyone. And that's not actually a big deal, because showing the apparently-high-performing variant to 90% of traffic gives it a lot of opportunities to prove that it's not actually that good, and it's click-through rate will come back down.

A much bigger issue, in my opinion, is when you add a new variant.

Let's say you already have 9 variants that all have click-through rates around 1% and have had about 1000 views each. Then you add a new variant. This new variant starts out with 0 clicks.

Now you have 10 variants, 9 of which have a CTR of 1% and your new one has a CTR of 0% (technically a degenerate case with 0 views, but becomes firmly 0% after the first view). And let's say your site expects 1000 views per day.

90% of the time your site is going to be showing one of the old variants, because no matter what happens to their CTRs, they can't go below 0%, so they will forever look better than your new variant. The remaining 10% of the time your site is going to be picking at random amongst all variants. So your new variant is going to get about 1% of your traffic. Or 10 views per day.

If your new variant also has a CTR of about 1%, then you'll expect to get about 1 click per 100 views. If it is only getting 10 views per day then it could easily be 10 days before you get the first click, during which time you're not even gathering much data on it.

So what I'm doing instead is defining the CTR to be (clicks+1)/(views+1). That is, we always optimistically assume that the next view is going to get a click.

That means a new variant starts out with a CTR of 1/1 = 100%. We skew the selection towards those variants that have not had many opportunities to prove themselves yet.

In this case the new variant will get 91% of the traffic until its optimistic-CTR falls below that of the next best variant. That could easily happen within the first day of releasing the new variant, so this "optimistic epsilon-greedy" algorithm broadly behaves exactly the same once the number of views is high enough, but it discovers the true CTR for newly-added variants much more quickly than the standard algorithm.

Even if the new variant actually never generates any clicks, its CTR drops below 1% within about 100 views (1/101 < 1%) so it won't be taking much traffic away from your older variants if it doesn't work very well.