I've been receiving lots of "Undeliverable Mail Returned to Sender" lately for Protohackers signup attempts. Protohackers login is via a "magic link", so attempting to sign up or log in results in sending an email. But none of the email body is user-controlled, so I don't really see the logic in abusing this form to spam people.

One email address has been put in over 100 times over the last 3 days, and I received "Undeliverable Mail Returned to Sender" each time, because it is a GMail address that doesn't exist. What's the logic in this? Most of the email addresses did exist however and presumably the spam either reached them or was filtered by GMail.

Some ideas I can think of:

• someone griefing particular users by bombarding them with signup spam for hundreds of services they don't use
• someone trying to get me specifically banned from GMail by making me send lots of unsolicited emails to GMail addresses
• some grey-hat chaos-monkey type operation trying to nudge all website operators into locking down forms that can cause email sending
• a weird botnet communicates internally by triggering Protohackers signup emails to itself, and the timestamp of the email allows them to reliably communicate about 10 bits at a time??

I think the first one is the best idea but I still don't really see why you would do this. Although it's a better reason than the others, it still doesn't seem like a good enough reason to actually bother.

I did already have a rate limit of 60 emails per recipient per day, and a burst limit of 5 per recipient per minute. I don't really want to stop people from being able to log in as many times as they need to, but getting 60 spam emails per day for a service you don't use is obviously too much. I did tighten the rate limits to 10 per day and 2 per minute, but really we don't want to be sending any spam.

At any rate, I wanted to stop this. The goal is to stop whatever bot is sending these emails, without impacting legitimate users (be they man or machine).

I noticed that all of these signup attempts were originating from the same netblock: 169.58.0.0/17, apparently operated by Contabo.

I'd rather not specifically discriminate against particular netblocks, both because legitimate users could be using the same netblock, and because a bot can easily change its hosting or use proxies.

But the fact that it always used the same netblock makes it easy to identify, and the fact that it comes back every few minutes makes it easy to investigate.

So my first mitigation was to add a tiny JavaScript proof-of-work, on the basis that a simple bot is probably not executing JavaScript. I was surprised to find that this actually didn't help. As an experiment, I kept ramping up the difficulty on the proof-of-work, and the bot was still successfully submitting the form even when it was taking over a minute to calculate the proof-of-work.

So I've left the proof-of-work in place, but back down to a trivial level so as not to inconvenience real users.

The next thing I did was selectively put the Bot Forensics collector script on the page only for clients within 169.58.0.0/17, with the idea that this would quickly reveal identifying features of this particular bot that I might be able to filter on without causing collateral damage.

I was disappointed to learn that the bot never posted off the Bot Forensics beacon. If you were using Bot Forensics as general-purpose bot detection, this would kind of be the ideal case. If you refuse to send emails for any session that has not posted a good beacon, then this misbehaving bot is blocked and you don't really care what the beacon would have contained.

But Bot Forensics is a bit too invasive for me to want to put it on the page for every user, and by this stage I was mainly motivated to learn more about this particular bot. And in any event, I don't actually have a problem with bots using the form in principle, I only have a problem with abuse of the form, whether by bot or by human.

I wondered if the reason the bot wasn't sending the beacon was simply because the proof-of-work blocked the page so it couldn't compute the beacon. So my next experiment was to put a 10-second timeout between completing the proof-of-work and sending off the email. The idea was that the page would then have a good 10 seconds in which to send off the Bot Forensics beacon.

Surprisingly, the 10-second timeout inhibited sending the email! Even though it previously spent over a minute calculating the proof-of-work. The bot must be waiting for inactivity and then closing the page after something less than 10 seconds.

A 10-second delay is still a bit much to be imposing on legitimate users though, so I tried reducing it to 3 seconds, and then the bot was back to successfully sending emails.

Although we weren't getting the full beacon content from Bot Forensics, we could still see:

• the bot is fetching the collector HTML for the iframe
• it's fetching other resources included inline in the HTML
• it's fetching resources requested by the JavaScript code
• it is able to send POST requests for exception logging
• but it is not POSTing the full beacon

(The exception that we log is expected, it's just a failure trying to fetch a resource which doesn't exist.)

I have a list of all of the User-Agent headers seen from the Contabo netblock. I'm not saying all of these are the malicious bot, but I suspect the majority are. ChatGPT points out that this list is probably from UserAgentString.com, good find ChatGPT.

Despite seemingly choosing a User-Agent at random from that list, the sec-ch-ua header always lists "HeadlessChrome", example:

User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1) Gecko/20061031 BonEcho/2.0
sec-ch-ua: "HeadlessChrome";v="143", "Chromium";v="143", "Not A(Brand";v="24"

So this does give us one way to block this bot with extremely low chance of causing collateral damage to legitimate users: we refuse to send email for any request that has "HeadlessChrome" in the sec-ch-ua header but not in the User-Agent header. That way we still don't block legitimate users even if they are using headless Chrome, as long as they're not messing with the User-Agent header.

Let's keep that one in our back pocket, I'd really like to get a bit more of a smoking gun.

I made the Bot Forensics collector send back a much smaller beacon, synchronously, and discovered:

• timezone is set to Europe/Berlin
• screen size is 1280x720
• it doesn't have any custom functions injected into the page, that's disappointing, they're normally my favourite thing to look at
• navigator.platform is "Linux x86_64" but navigator.userAgentData.platform is edited to suit the User-Agent header

So apart from having "HeadlessChrome" in sec-ch-ua but not User-Agent, the mismatch between navigator.platform and navigator.userAgentData.platform is another thing we could filter on. This bot does sometimes use a real headless Chrome User-Agent, and it is its most common one, but the vast majority of requests use the other weird ones.

At this point I noticed one other bizarre behaviour from this bot: shortly after sending the signup email, it tries to load the user profile page, even though that page is not linked from the signup page. What's the angle there? Maybe this is some kind of automated vulnerability scanner that thinks it might be able to access random people's accounts simply by sending the email and speculatively browsing to the profile page? I literally don't understand why you would even check this. Even if it worked, which it doesn't, even if they click on the link, because that only authenticates the session that clicked the link and not the one that sent the email... but even if it worked, what benefit do you get from hacking someone's Protohackers account?

Anyway, I'm out of time and stopping for now. So changes in response to this bot are:

• email sending now requires a (tiny) proof-of-work, and the JavaScript code includes a 1-second sleep; this doesn't stop this bot but might stop others
• rate limits reduced from 60/day and 5/minute to 10/day and 2/minute
• backend now refuses to send email for clients who have inconsistent "HeadlessChrome" and "Linux x86_64"; this blocks almost all emails from this particular bot

And the Bot Forensics collector is now removed, even for clients from Contabo.

If you find you now have trouble logging in to Protohackers, I'm sorry, please let me know. Also if you can work out what this bot is actually trying to achieve I'd be really interested to know.

If we have to do any more on this, I think I might try a proof-of-work system that starts out easy but drastically ramps up in difficulty based on how many emails have been sent to that recipient, or from that client netblock, in the past day.