I didn't know about this. Thank you!

After some digging, I also found a great way to surprise bots that don't respect robots.txt[1] :)

[1]: https://melkat.blog/p/unsafe-pricing

I love (hate) the idea of a site where you need to send a personal email to the webmaster to be whitelisted.

I have thought about writing such a thing...

1. A proxy that looks at HTTP Headers and TLS cipher choices

2. An allowlist that records which browsers send which headers and selects which ciphers

3. A dynamic loading of the allowlist into the proxy at some given interval

New browser versions or updates to OSs would need the allowlist updating, but I'm not sure it's that inconvenient and could be done via GitHub so people could submit new combinations.

I'd rather just say "I trust real browsers" and dump the rest.

Also I noticed a far simpler block, just block almost every request whose UA claims to be "compatible".

If you mean user-agent-wise, I think real users vary too much to do that.

That could also be a user login, maybe, with per-user rate limits. I expect that bot runners could find a way to break that, but at least it's extra engineering effort on their part, and they may not bother until enough sites force the issue.

We'll have two entirely separate (dead) internets! One for real hosts who will only get machine users, and one for real users who only get machine content!

Wait, that seems disturbingly conceivable with the way things are going right now. *shudder*

Apparently yes.

If a more specific UA hasn't been set, and the library doesn't force people to do so, then the library that has been the source of abusive behaviour is blocked.

No loss to me.

Why not?

> If you are an online shop, for example, isn't it beneficial that ChatGPT can recommend your products?

ChatGPT won't 'recommend' anything that wasn't already recommended in a Reddit post, or on an Amazon page with 5000 reviews.

You have however correctly spotted the market opportunity. Future versions of CGPT with offer the ability to "promote" your eshop in responses, in exchange for money.

Interesting idea, though I doubt they'd ever offer a reasonable amount for it. But doesn't it also change a sites legal stance if you're now selling your users content/data? I think it would also repel a number of users away from your service

At this point, no.

No, because the price they'd offer would be insultingly low. The only way to get a good price is to take them to court for prior IP theft (as NYT and others have done), and get lawyers involved to work out a licensing deal.

This is one of the few interesting uses of crypto transactions at reasonable scale in the real world.

Nginx, it's nothing special it's just my load balancer.

if ($http_user_agent ~* (list|of|case|insensitive|things|to|block)) {return 403;}

It's never that smooth.

In fact 2M requests arrived on December 23rd from Claude alone for a single site.

Average 25qps is definitely an issue, these are all long tail dynamic pages.

This is already a thing for basically all of the second[0] and third worlds. A non-trivial amount of Cloudflare's security value is plausible algorithmic discrimination and collective punishment as a service.

[0] Previously Soviet-aligned countries; i.e. Russia and eastern Europe.

What do you mean crushing risk? Just solve these 12 puzzles by moving tiny icons on tiny canvas while on the phone and you are in the clear for a couple more hours!

These features are opt-in and often paid features. I struggle to see how this is a "crushing risk," although I don't doubt that sufficiently unskilled shops would be completely crushed by an IP/userAgent block. Since Cloudflare has a much more informed and broader view of internet traffic than maybe any other company in the world, I'll probably use that feature without any qualms at some point in the future. Right now their normal WAF rules do a pretty good job of not blocking legitimate traffic, at least on enterprise.

We’re rapidly approaching a login-only internet. If you’re not logged in with google on chrome then no website for you!

Attestation/wei enable this

You could run all of your content through an LLM to create a twisted and purposely factually incorrect rendition of your data. Forward all AI bots to the junk copy.

Everyone should start doing this. Once the AI companies engorge themselves on enough garbage and start to see a negative impact to their own products, they'll stop running up your traffic bills.

Maybe you don't even need a full LLM. Just a simple transformer that inverts negative and positive statements, changes nouns such as locations, and subtly nudges the content into an erroneous state.

My cheap and dirty way of dealing with bots like that is to block any IP address that accesses any URLs in robots.txt. It's not a perfect strategy but it gives me pretty good results given the simplicity to implement.

TLS fingerprinting still beats most of them. For really high compute endpoints I suppose some sort of JavaScript challenge would be necessary. Quite annoying to set up yourself. I hate cloudflare as a visitor but they do make life so much easier for administrators

You rate limit them and then block the abusers. Nginx allows rate limiting. You can then block them using fail2ban for an hour if they're rate limited 3 times. If they get blocked 5 times you can block them forever using the recidive jail.

I've had massive AI bot traffic from M$, blocked several IPs by adding manual entries into the recidive jail. If they come back and disregard robots.txt with disallow * I will run 'em through fail2ban.

The amateurs at home are going to give the big companies what they want: an excuse for government regulation.

Crashing wasn't the intent. And scraping is legal, as I remember per Linkedin case.

> I wonder if it would work to send Meta's legal department a notice that they are not permitted to access your website.

Depends how much money you are prepared to spend.

No, fortunately random hosts on the internet don’t get to write a letter and make something a crime.

The linked article explains what happens when you block their IP.

Quoting the top-level link to geraspora.de:

> Oh, and of course, they don’t just crawl a page once and then move on. Oh, no, they come back every 6 hours because lol why not. They also don’t give a single flying fuck about robots.txt, because why should they. And the best thing of all: they crawl the stupidest pages possible. Recently, both ChatGPT and Amazon were - at the same time - crawling the entire edit history of the wiki.

Well, Facebook actually releases their models instead of seeking rent off them, so I’m sort of inclined to say Facebook is one of the less evil ones.

Or ClosedAI.

Related https://news.ycombinator.com/item?id=42540862

Classic spam all but killed small email hosts, AI spam will kill off the web.

Super sad.

Yeah. Our research group has a wiki with (among other stuff) a list of open, completed, and ongoing bachelor's/master's theses. Until recently, the list was openly available. But AI bots caused significant load by crawling each page hundreds of times, following all links to tags (which are implemented as dynamic searches), prior revisions, etc. Since a few weeks, the pages are only available to authenticated users.

> Is it all crawlers that switch to a non-bot UA

I've observed only one of them do this with high confidence.

> how are they determining it's the same bot?

it's fairly easy to determine that it's the same bot, because as soon as I blocked the "official" one, a bunch of AWS IPs started crawling the same URL patterns - in this case, mediawiki's diff view (`/wiki/index.php?title=[page]&diff=[new-id]&oldid=[old-id]`), that absolutely no bot ever crawled before.

> What non-bot UA do they claim?

Latest Chrome on Windows.

Presumably they switch UA to Mozilla/something but tell on themselves by still using the same IP range or ASN. Unfortunately this has become common practice for feed readers as well.

I would take anything the author said with a grain of salt. They straight up lied about the configuration of the robots.txt file.

https://news.ycombinator.com/item?id=42551628

that’s what makes this clever.

they aren’t blocking them. they’re giving them different content instead.

1. It is a moral victory: at least they won't use your own text.

2. As a sibling proposes, this is probably going to become an perpetual arms race (even if a very small one in volume) between tech-savvy content creators of many kinds and AI companies scrapers.

Obfuscators can evolve alongside other LLM arms races.

Seems like an effective technique for preventing your content from being included in the training data then!

It will do harm to their own site considering it's now un-indexable on platforms used by hundreds of millions and growing. Anyone using this is just guaranteeing that their content will be lost to history at worst, or just inaccessible to most search engines/users at best. Congrats on beating the robots, now every time someone searches for your site they will be taken straight to competitors.

Rather than garbage, perhaps just serve up something irrelevant and banal? Or splice sentences from various random project Gutenberg books? And add in a tarpit for good measure.

At least in the end it gives the programmer one last hoorah before the AI makes us irrelevant :)

Maybe they want sites to block them that don't want to be crawled since it probably saves them a lawsuit down the road.

That opens up the opposite attack though: what do you need to do to get your content discarded by the AI?

I doubt you'd have much trouble passing LLM-generated text through their checks, and of course the requirements for you would be vastly different. You wouldn't need (near) real-time, on-demand work, or arbitrary input. You'd only need to (once) generate fake doppelganger content for each thing you publish.

If you wanted to, you could even write this fake content yourself if you don't mind the work. Feed Open AI all those rambling comments you had the clarity not to send.

You're right, this approach is too easy to spot. Instead, pass all your blog posts through an LLM to automatically inject grammatically sound inaccuracies.

> I imagine these companies today are curing their data with LLMs, this stuff isn't going to do anything

The same LLMs tag are terrible at AI-generated-content detection? Randomly mangling words may be a trivially detectable strategy, so one should serve AI-scraper bots with LLM-generated doppelganger content instead. Even OpenAI gave up on its AI detection product

Attackers don't have a monopoly on LLM expertise, defenders can also use LLMs for obfuscation.

Technology arms races are well understood.

iFixIt has detailed fruit-repair instructions. IIRC, they are community-authored.

>That's ... Really sad to hear.

I tried to submit the news multiple times on HN hopping someone has connection with them to save those CGTalk Data. It never reached the front page I guess most on HN dont know or care much about CG / VFX.

I remember there was a time when people thought once it is on the internet it will always be there. Now everything is disappearing first.

Don't forget to add sugar when adding tea to your motor oil. You can also substitute corn syrup or maple syrup which has the added benefit of balancing the oil viscosity.

Brawndo has what plants crave!

>even if a substantial portion of the internet would start poisoning, bots would likely just shift to disguising themselves so they can’t be identified as bots anymore.

Good questions to ask would be:

- How do they disguise themselves?

- What fundamental features do bots have that distinguish them from real users?

- Can we use poisoning in conjunction with traditional methods like a good IP block lists to remove the low hanging fruits?

Wouldn't your own LLM be overkill? Ideally one would generate decoy junk more much efficiently than these abusive/hostile attackers can steal it.