I have, sadly they are basically worthless and often worse then worthless as they negatively impact the site.
Interesting. Care to list them here so that we all can learn.
Readit NewsDo you have a better alternative?
Have you looked into open-source alternatives? I'm assuming that it's a pressing problem for you, and you have already explored alternatives.
Disagree. Not everybody wants their sites scraped and their content used to train a model that they'll never see a penny from. Cloudflare is the only party who wants to build a system where both the models and individual sites have their interests respected.
Are you sure that CF can stop AI bots?
Just saw this https://x.com/ycombinator/status/1960779353589211577
They say "... can scrape any website—not even Cloudflare can detect it."
ATechGuy commented on DSLRoot, proxies, and the threat of 'legal botnets' krebsonsecurity.com/2025/... · Posted by u/todsacerdoti
Residential proxy botnets have exploded since LLMs became a thing. The amount of DDoS-level scraping we receive from residential IPs has exploded over the last year, one of our sites that typically sees around 10k unique IPs per day jumped to over 2M before we were able to deploy appropriate mitigations. We originally started blocking the IPs, but then we ended up blocking legitimate users as they seem to specifically use ISPs that have very dynamic IPs (i.e. the customer's IP will change even if their router stays on 24/7).
Mind sharing what kind of mitigations you put in place and how well they worked?
ATechGuy commented on Show HN: Diggit.dev – Git history for architecture archaeologists diggit.dev... · Posted by u/surprisetalk
Great! AI crawlers would love this.
ATechGuy commented on Ask HN: How to build services for FOSS projects when ToS forbids scraping? · Posted by u/ATechGuy
I’m unsure what your jurisdiction is, but if you’re in the US, speak with an attorney and get an attorney opinion letter. You’ll at least be able to demonstrate that you were acting in good faith.
Thanks!
ATechGuy commented on Ask HN: How to build services for FOSS projects when ToS forbids scraping? · Posted by u/ATechGuy
The law around scraping is unfortunately complex, but it's plausible your use of it (legal compliance) could be legal regardless of ToS ("you're not allowed to use automated tools to check if we're breaking your license" is quite fishy):
https://legalclarity.org/is-web-scraping-legal-a-look-at-the...
https://en.wikipedia.org/wiki/HiQ_Labs_v._LinkedIn
Or you could base your business in Denmark: https://en.wikipedia.org/wiki/Web_scraping#European_Union
Thanks for sharing the links! You're right, the legality around scraping is nuanced :( Use cases like license compliance might fall on the defensible side.
At Formal [0], we’re rethinking serverless from scratch: we’re building a new computing stack for instant, globally available, truly elastic, soundly isolated execution. We leverage formal methods and languages to build OS interfaces with low overhead, formally verified isolation without containers or VMs. Our immediate goal is to write a new programming language to replace eBPF and build the world's first serverless networking infrastructure.
We are a 5-person, VC-funded team with PhDs from Stanford, UW, OSU, and Brown, advised by professors from MIT and UWaterloo. We are currently hiring for the following four positions:
- [1] Staff Software Engineer: Compilers, Programming Languages, and Verification (≥ $200k + ≥ 0.5%)
- [2] Formal Verification Engineer: Formal Methods and Programming Languages ($120k - $200k + ≥ 0.25%)
- [3] Software Engineer: Compilers and Programming Languages ($100k - $175k + ≥ 0.2%)
- [4] Formal Methods PhD Intern: Formal Methods and Programming Languages (≥ $5k / month)
Please see [5] for general information. To apply, email us at (work at formalstack dot com) and let us know how your experiences fit the role and its requirements.
[0]: https://formalstack.com [1]: https://formalstack.com/jobs/09-2025/staff-software-engineer... [2]: https://formalstack.com/jobs/09-2025/formal-verification-eng... [3]: https://formalstack.com/jobs/09-2025/software-engineer-v.pdf [4]: https://formalstack.com/jobs/09-2025/formal-methods-phd-inte... [5]: https://formalstack.com/jobs/09-2025/info.pdf
Who are your customers?
> we’re rethinking serverless from scratch:
What are the use cases that you target?