Why not just block the User Agent?
Because it's the single most falsifiable piece of information you would find on ANY "how to scrape for dummies" article out there. They all start with changing your UA.
Readit Newsarewethereyeta commented on Ban me at the IP level if you don't like me boston.conman.org/2025/08... · Posted by u/classichasclass
arewethereyeta commented on Ban me at the IP level if you don't like me boston.conman.org/2025/08... · Posted by u/classichasclass
i think there is an opportunity to train an neural network on browser user agent s(they are catalogued but vary and change a lot). then u can block everything not matching.
it will work better than regex. a lot of these companies rely on 'but we are clearly recognizable' via fornexample these user agents, as excuse to put burden on sysadmins to maintains blocklists instead of otherway round (keep list of scrapables..)
maybe someone mathy can unburden them ?
you could also look who ask for nonexisting resources, and block anyone who asks for more than X (large enough not to let config issue or so kill regular clients). block might be just a minute so u dont have too many risk when an FP occurs. it will be enough likely to make the scraper turn away.
there are many things to do depending on context, app complexity, load etc. , problem is there's no really easy way to do these things.
ML should be able to help a lot in such a space??
What exactly do you want to train on a falsifiable piece of info? We do something like this at https://visitorquery.com in order to detect HTTP proxies and VPNs but the UA is very unreliable. I guess you could detect based on multiple pieces with UA being one of them where one UA must have x, y, z or where x cannot be found on one UA. Most of the info is generated tho.
there are plenty of other languages. I dont get this love-hate type of speech like golang itself owes you an apology.
arewethereyeta commented on Supabase MCP can leak your entire SQL database generalanalysis.com/blog/... · Posted by u/rexpository
meanwhile people are crying for simple features like the ability to create a transaction (for queries) for years but let's push AI.
arewethereyeta commented on Implementing fast TCP fingerprinting with eBPF halb.it/posts/ebpf-finger... · Posted by u/halb
More useless and harmful anti-bot nonsense, probably with many false detections, when a simple and neutral rate-limiting 429 does the job.
There are MANY cases for such an implementation. My service [1] implements such a thing, eBPF too, and my users do it for many valid reasons such as:
- shopping cart fraud
- geo-restricted content (think distributing laws)
- preventing abuse (think ticket scalpers)
- preventing cheating and multi-accounting (think gaming)
- preventing account takeovers (think 2FA trigger if fingerprint suddenly changed)
There is much more but yeah, this tech has its place. We cannot just assume everyone has a static website with a free for all content.
Charlie Hebdo
Shameless plug but I have a client that got rid of almost 90% by blocking residential proxies or HTTP proxies in general using our service [1]. I tend to think people try some measures that are very hard to maintain going for behavioural data and other indicators where all fraud sits on this L1 being a proxy or a vpn.
arewethereyeta commented on Ask HN: Startup getting spammed with PayPal disputes, what should we do? · Posted by u/june3739
https://visitorquery.com - my startup. I'm curious if they use proxies or not. Datacenter or residential, my service can detect them. You have a free plan which should allow you to have a better understanding of your traffic, at least from this perspective. Shenanigans with payment gateways usually involve proxies so I'm almost certain you can use it to detect > block the abusers before they reach the checkout page.
arewethereyeta commented on Show HN: SVG Animation Software expressive.app/expressive... · Posted by u/msarca
Maybe that's why people don't bother targeting Linux. Look at your comment. Why are people so easily offended these days by things that are out here, doing no harm to anyone. Like, you could just hit the back button and go on with your day. Why do you need to be so offended? If this is the sort of stuff that triggers you I can't imagine how your day looks like.
I feel like these AI companies are in a gold rush while somebody else is selling the shovels. I've never jumped ship for the same service, from a vendor to another... so often. Looks like a race to the bottom where the snake eats itself.