Time for yet another blog status update.
Block (AI bot) party
Recently (at least for me), loading my site’s been running slow, and I suspect part of the reason’s AI bot traffic. As such, I thought I’d try blocking those as much as possible.
I last wrote on how to block AI several years ago, but a lot’s changed since then. A more up-to-date article I’ve found is by Perishable Press (from a few months ago) on how to block a list of common AI bots. It’s appropriately titled “the Ultimate Block List to Stop AI Bots,” and offers a list of AI bots (including all the major ones) to use in robots.txt or .htaccess files. As the previous sentence’s bit of tech jargon shows, one downside is you’ll need some technical skills to make full use of this list. A few WordPress plugins, however, exist that perform a similar function.
Of course, said blocking means isn’t perfect: not all AI bots care about respecting robots.txt files; new AI bots come along; and blocking tools don’t do much about ChatGPT, etc. already likely having scraped sites. Still, for those with the skills to do so, it might be worth a try, as it’s better than nothing.
I installed the block list into my site’s robots.txt and .htaccess files. While I still get a few hits from ChatGPT, it seems to have reduced the amount of traffic from China, which I suspect is just bots. That said, I’ve also tried installing IP Location Block, a WordPress plugin that allows blocking by country, to block China period. (I doubt I have many readers from there to make a difference, versus the bots.) As I only installed all of these a few days ago, I’m not sure how much has improved so far, but my site feels more responsive. (Only applicable for my end: I also recently swapped out my house’s router, as my broadband connection has had issues.) That said, I hope this AI bubble bursts soon, and that the open internet’s not killed by it in the process.
Image by StockSnap from Pixabay
Last night I had to block a dozen /16 networks beating up my own web server. Mostly Hong Kong, but also Iraq, Cambodia, and Bangladesh. Definitely not legit traffic, and masquerading as common browsers.
It _could_ be AI bots, but that’s only because AI bot traffic tends to be indistinguishable from denial of service attacks and vulnerability probing. The people doing AI scraping tend to be as grossly incompetent as common skript kiddies.