Well into the second day of Cloudflare’s blockade of my home internet connection, Google Search also began blocking requests. It required me to resolve a CAPTCHA challenge for every other search. This luckily only lasted a day.
Cloudflare shares IP reputation data with partners like Google, coordinated through a program called the Bandwidth Alliance. So, my original offense might not even have been against Cloudflare. It might have received the reputation data from a partner, and it just propagated through the Bandwidth Alliance network.
That's not what Bandwidth Alliance is at all. It's about reducing or eliminating egress fees between a cloud provider and Cloudflare. Not sure where the idea that it's about sharing IP reputation data comes from.
FYI, this guy is far from alone, your "protection" has given me a lot of grief over the past few years, particularly on highly NATed mobile networks.
I've been gradually removing cloudflare based CDNs from services I develop and control because I don't want my users being arbitrarily discriminated against.
There was a good article posted on HN recently titled "The ideal level of fraud is non-zero" which I think is highly relevant here... In essence any mechanism employed to prevent illegitimate use comes with a negative cost to legitimate users, if that cost is too high it defeats the purpose. i.e what's the point in a website that is completely immune to a botnet and also cannot be accessed by anyone else? unplugging the ethernet cable also effectively protects against botnets. More subtly the cost of outright rejecting some legitimate users is usually not worth the savings of rejecting 100% of illegitimate ones. I think Cloudflare's service has it the wrong way around: it currently accept blocking legitimate users far too easily, that is not an acceptable cost; whereas you should be letting a higher level of bots through to avoid pissing off legitimate users - if it's not obviously a DDoS, it's probably worth the bandwidth cost.
Consider the bigger picture, if you save a slither of a penny by blocking a bot, but also end up blocking or seriously inconveniencing 10 real users... is it worth it.
Cloudflare just isn't worth the tradeoffs: the risks associated with their centralization, how they made Tor basically unusable on non-onion sites, the lack of transparency when content-moderating the internet, etc.
The space is in need of solid competitors to break the stranglehold they have on the internet. Whether it's the right combination of services, documentation, etc.
Just 10 minutes ago, I got the following email from a housemate (I'm not home at the moment):
> The past few weeks I've been getting tons of redirects to verify my humanity before being allowed to view a webpage. Usually I just have to click the box that says human, not find all the ladders in a photo. SoFi is doing it every single time I log in. Petco, too, along with others who are more sporadic. This is happening with and without uBlock on. Same browser I've always used. ...
SoFi and Petco both use Cloudflare. I do exactly zero web crawling / scraping / abusive anything from my home connection.
I'm noticing a recent increase in volume of complaints about Cloudflare's human verification filter. I'm starting to wonder if they touched a dial.
I had already started pulling some infra back from Cloudflare after their last appearance in the tech news cycle. Now I've got an additional reason to continue doing that.
You're looking at it all wrong. From Cloudflare's point of view, this kind of blocking is a feature. Anyone doing legitimate web crawling, or offering alternative web services such as Starlink, now needs Cloudflare's permission.
Essentially, for a broad class of web-based businesses, they have made themselves gatekeepers. I'm sure they'll find a profitable use for this position. Charging outright would look bad, but investing in businesses that just happen to not run into Cloudflare-based trouble, but whose competitors do...
Isn’t there an config option to dial down the anti-bot stuff so that you still get the benefit of Cloudglare’s caching but with much less chance of dropping legit traffic from schools, VPNs, etc? I think their lowest setting only really kicks in if they think an ip is participating in a DDOS attack.
My dude, it isn't about money. At least not directly.
I encourage those of you attempting to block Cloudflare to try and host your own website for a bit. Make sure you don't do it on a metered/paid connection. I know one eCommerce site with 1,300 employees that went bankrupt overnight thanks to the AWS bill (and lack of options to get back online, this was prior to companies such as CF). Bankruptcy as in the company filed for bankruptcy and no longer exists. They were profitable for a decade prior. One DDoS attack...
Also make sure you don't have a democratic opinion if you are in the US, like a 50 person manufacturing company. They were shut down completely thanks to saying a single wrong thing about Republicans. CF existed there, but they weren't aware thanks to not having IT folks. They were a non profit.
CF may be evil to some, but there is a reason they exist. I use CF. I don't like throwing money at them every month, however, many of my websites have also been attacked, usually via competitors. We can either deanonymize the internet or allow companies like CF to exist. There is really no other way.
> That's not what Bandwidth Alliance is at all. It's about reducing or eliminating egress fees between a cloud provider and Cloudflare. Not sure where the idea that it's about sharing IP reputation data comes from.
I need to look into that. Thanks for pointing it out. I had totally forgotten about that post.
Edit: team tells me this idea never got off the ground. Did talk with some potential partners (which did NOT include Google) but didn’t happen. So if Google was throwing CAPTCHAs it wasn’t because of our IP reputation.
You block this guy from the internet for a week —- for no apparent reason —- and then you come in here with a nitpick about how another related system works?
The point is that Cloudflare does not beam IP reputation data to Google. If Google and CF are blocking this IP separately, what's the chance there's some malicious device or hacked IoT device on the network, participating in DDOS attacks or unauthorized vulnerability scanning of random websites?
The tone of this reply is a bit shit from a PR perspective.
How about _also_ pointing to a knowledge base article for how an end user could go about working out what network activity from their IP might be flagging Cloudflare’s systems?
>"Not sure where the idea that it's about sharing IP reputation data comes from."
One source of that would be a blog post on your company's website that was actually authored by you! Point 2 below:
>"Once enabled, when we detect a bad bot, we will do three things: (1) we’re going to disincentivize the bot maker economically by tarpitting them, including requiring them to solve a computationally intensive challenge that will require more of their bot’s CPU; (2) for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline; and (3) we’re going to plant trees to make up for the bot’s carbon cost. [1]
That person should start with the assumption they haven't been misclassified and eliminate the possibility that a device on their network is compromised.
But that's beside the main point. You guys are essentially the "single point of failure" for half the internet. [1] Being competent and smart doesn't really help too much, as demonstrated by how you guys had to give in to the pressure to censor recently.
What happened to PrivacyPass? It seems to have stopped working completely when connecting over Tor several months back. I say this from having spent several hours trying to get it work on multiple devices with different OS/client software (chromium/FF), both with the store versions and bundling the extension from source.
We did have it working mostly fine for some time back in 2021 but haven't been able to since.
There are multiple open issues reporting this on the GH repo with no real follow-up from maintainers apart from maybe a "should be fixed, open again if still an issue".
If you'd like to experience this treatment first-hand, try surfing the web using the Tor Browser.
Spoiler alert: many websites simply refuse to load at all (e.g. any google service, and lots of websites "protected" by CF). Captchas are everywhere: in many cases, you can't even complete simple GETs of blogs without donating free labor to CF.
And the most infuriating part, you get CF marketing messages right in your face while your browser is calculating hashcash (I guess?)... At this point I can recognize every single one of them: something about bots making up 40% of all internet traffic, something about their web scraper protection racket, something about small businesses (???), etc etc...
To be fair, Tor exit nodes have an awful reputation for sure. Nevertheless, I have a hard time forgiving how CF makes browsing the Internet hell for those who actually need Tor.
> And the most infuriating part, you get CF marketing messages right in your face while your browser is calculating hashcash (I guess?)... At this point I can recognize every single one of them: something about bots making up 40% of all internet traffic,
Yeah, there's something amazingly aggravating about CF telling you how much traffic is bots while showing that they can't distinguish you from a bot.
CloudFlare are creating a new devision for advertising to bots. They have projected that in the near future, bots will be 90% of spending, so the bot demographic is the most important to target, marketingwise.
The fact that humans are seeing the traffic meant for bots is an unfortunate side-effect.
I personally welcome our future bot overlords (not only because being unwelcome might be unhealthy for me — why would I publicly disagree with an overlord or not want to be their friend?).
Cloudflare has mixed up the definitions of "bot" and "abuse". Tor users may or may not be bots, but as long as they don't abuse (spamming or DoS), they ought to be treated the same.
You don't even need TOR. Try a public wifi that is not in the "preferred geographical location" (i.e. US or Europe). The gaming cafes in SEA are probably responsible for 90% of all AI training datasets lol
I routinely use Youtube with Tor. I will occasionally get kicked off with a "suspicious traffic" message, but it isn't my experience that it "refuses to load at all".
Harsh blocking/limiting/challenging is way too valuable to sites that are actually trying to make money online. It's not going away short of legislation banning it. Losing 1/10,000 legitimate customers to cut fraud attempts, spam, exploit attempts, and so on, by 90% or more, is just too good a trade-off.
I have bad news about the most-likely fix for it, longer term, so we can lay off the IP-based reputation stuff and the geo-blocking: it's tying some form of personal ID to your browsing activity, so that bears the reputation instead of the address.
An alternative that preserves some privacy also doesn't seem that hard to imagine... though it probably has its own can of worms*.
Basically, the core problem is digital identities (accounts, IPs, phone #s etc.) are cheap to create (even considering captchas and all) so fraud is easy. The solution could be just to make it "costly" to create new digital identities. For example, you could get a "verified but anonymous" identity issued by locking some assets (could be real world money, or maybe something intangible like community reputation) as collateral with a trusted party (or, for the crypto people, the blockchain). If you misbehave, you lose your reputation on that identity (and essentially your collateral) and have to start over. This lets anyone bootstrap a "minimal" level of trust at the beginning before they can use time to prove themselves trustworthy.
Note: This model might remind some of things like staking in crypto. However the idea is really not anything new... Putting money on the line is really how most low-trust bootstrapping happens.
*: To name a few:(1) this can result in participation being gated by wealth, which can be unfair. (2) it makes accounts more valuable to hack so people need better security practices [re: twitter checkmark]. (3) one would need some authority to decide how accounts lose their collateral or maybe the collateral is just burned to create that initial credibility...
> Basically, the core problem is digital identities [...] are cheap to create [...] so fraud is easy. The solution could be just to make it "costly" to create new digital identities.
We already use this model in practice. It's why so many services require a phone number verification now - they are hard enough to get en-masse, especially if you block things like Google Voice. They even have a big advantage in that they are comparatively hard to hack, as the SIM card is effectively a weak form of physical security key.
I think the big problems this causes is discussed on HN quite often.
Your idea is comes from a good place, but identity theft is already a thing in the real world. Digital identities would also be very stealable. This malware more harmful in the long term. Imagine if your Twitter gets hacked and your digital identity makes it so your Gmail gets blocked.
Similar, the internet is already very difficult for the people with limited means. This would make it even harder.
> Basically, the core problem is digital identities (accounts, IPs, phone #s etc.) are cheap to create (even considering captchas and all) so fraud is easy. The solution could be just to make it "costly" to create new digital identities. For example, you could get a "verified but anonymous" identity issued by locking some assets (could be real world money, or maybe something intangible like community reputation) as collateral with a trusted party (or, for the crypto people, the blockchain). If you misbehave, you lose your reputation on that identity (and essentially your collateral) and have to start over. This lets anyone bootstrap a "minimal" level of trust at the beginning before they can use time to prove themselves trustworthy.
I've always thought that client certs would be an interesting solution to this problem. Any given certificate can carry signatures from multiple signing authorities, right? So we could imagine a world where there are many different certificate authorities, each of whom have their own criteria for signing a particular certificate and each of whom offer different varieties of assurance regarding the signature-holder's identity.
From here, the question of "should I allow the user identified by this client cert to use my service" simply becomes a question of 1.) checking the validity of the signatures of the client cert and 2.) deciding if the CA's criteria for signing certs aligns with my desired userbase.
For example, a particular CA might insist that their users go through some real-world process to renew their certification every few years, but when they sign a cert it means that the bearer has been strongly vetted as a real person.
An interesting side effect of this auth model is that a service provider accepting certs from a particular CA has someone to complain to if a user bearing their signature acts improperly on their platform. You could imagine a CA which has a code of conduct expected of the users whose certs they sign, and would perhaps revoke a user's certification if too many websites complain.
I think this is true. It also reminds me of one possible purpose of regulation and government, given the majority will usually be happy to throw any sort of minority under the bus for the "greater good."
This also reminds me of the anxiety of Google deciding to just ban my account for some reason. They can't be bothered to commit resources to making sure mistakes can be resolved. They don't care to lose a fleetingly small percentage of customers.
> Harsh blocking/limiting/challenging is way too valuable to sites that are actually trying to make money online.
I'm not understanding the generalized sentiment here. How would, for example, a retailer benefit from this strategy? How does it protect their bottom line?
I can see how a particular kind of "facilitated user economy," such as games, gambling and promotional companies could benefit, but it doesn't seem that broadly applicable to what most people would consider a "mainstream" business.
> so we can lay off the IP-based reputation stuff and the geo-blocking: it's tying some form of personal ID to your browsing activity
And a new market for identity theft is born.
Also, as someone who serves content and geo blocks it, that's not up to me, that's up to the owner of the content or whoever happens to be licensing it for them. So, even if you sent me a picture of your government ID, it changes nothing.
> I'm not understanding the generalized sentiment here. How would, for example, a retailer benefit from this strategy? How does it protect their bottom line?
The amount of automated and apparently-manual attempted credit card fraud (and exploit attempts, for that matter) any halfway-prominent site with a CC form is subjected to is hard to appreciate if you've never seen it. It's a whole lot. They aren't even necessarily trying to buy what you have, but to validate that their stolen cards work. And they're quite busy. If too much of that gets through—really, any more than a very tiny amount of it gets through—you're gonna have an extremely bad time.
Various CC service providers like Stripe do provide tools to try to block those attempts, but defense in depth is usually a very good idea, including fairly aggressive firewall-level blocking.
> a retailer benefit from this strategy? How does it protect their bottom line?
A couple of examples I can think of is blocking bots from scraping their site for pricing and details and from resellers from buying up all of the stock (see sneakers, electronics, etc). The last example doesn't directly impact their bottom line, but it will make customers go elsewhere.
That's not a solution, it would be way worse. Companies would then make automated decisions and associate them with your personal ID, and spammers/DDOSers would be spending serious effort to hack their way to using the IDs of innocents. So rather than just your home network or whatever getting a sh*t reputation with no recourse, you would.
Spammers typically implement bots to carry out tasks. I mean, technically at some point a spammer is a real person, but when you're automating tasks and using bots, it's not at the same scale.
They are already testing out digital IDs. Now link that to the social score... and make the browsers and the sites exchange these data on the background, and make frontend services providers refuse connections from non-supporting browsers as "bots"...
The other not-so-great approach is to act like a normal user. This stuff doesn't tend to happen to the average Joe who browses the WWW. It's when you're doing unusual (albeit harmless) things.
Cloudflare is a regular problem for Starlink users. We're on CGNAT so users share IPv4 addresses. I see CAPTCHAs when using Starlink ten times as often as on my other ISP. I don't think it actually breaks things the way this article describes, it seems like a gentler behavior, but it's annoying.
A few months ago I got on Akamai's naughty list (with my other ISP) for some very light automated website downloading. That was a straight block with HTTP errors and I had to use a proxy to access the Web. It cleared up after a few days.
The lack of any user feedback or support for this situation is really annoying. Reminds you how much power the CDNs have. It'd be really bad if loading websites got as difficult as sending email through all the layers of spam filtering.
I feel like Starlink could at least partially mitigate this by supporting IPv6. T-mobile US supports IPv6, and I hardly notice this as an issue on my phone. Or the time my work ran the business over a 4G mobile while waiting for ISP install.
A genuine question from an ignoramus: how on earth did Starlink launch a brand new ISP in 2020 which doesn't support IPv6? Is IPv6 really so difficult? Does actually nobody care about IPv6 still, after all these years?
> Cloudflare is a regular problem for Starlink users. We're on CGNAT so users share IPv4 addresses. I see CAPTCHAs when using Starlink ten times as often as on my other ISP. I don't think it actually breaks things the way this article describes, it seems like a gentler behavior, but it's annoying.
I've been noticing this too, and it's why Starlink remains my secondary ISP/bulk transfer connection. If I had to drop one connection, I'd drop Starlink for this reason alone.
There are some sites that I simply can't browse, and it's not Cloudflare errors, either. Lowes, in particular, simply returns error pages for anything but the main landing page on a regular enough basis. Of course, my observed public IP changes so it's not consistent, but it's genuinely annoying.
> I've been noticing this too, and it's why Starlink remains my secondary ISP/bulk transfer connection. If I had to drop one connection, I'd drop Starlink for this reason alone.
Could cloudflare legally charge them a bribe to captcha their users less? It isnt good to have a company in this position of power if so.
A combination of shotscraper and metascraper; really more web previews than archives. And in a single thread, to different hostnames, maybe one every 10 seconds? Honestly surprised Akamai or anything even noticed. I fake my user agent now, lesson learned.
But any automated tool won't work. I have a similar problem with my self hosted feed reader, my vps hosting ip doesn't have 100% reputation with Cloudflare and I can't download some feeds
I'm really hoping cloudflare gets busted for having backroom deals with big ISPs or something. (For instance, if the cgnat had a cloudflare CDN cache endpoint behind / accociated with it, I suspect the IP would be white listed.)
They're working on double dipping by providing both the problem and the solution. Somehow this is not a recurrent issue for every other CDN / ddos shield. They're not even mentioning any other hosting company collaborating on this open solution that requires hardware from a specific company they totally don't have a deal with...
If you surf on desktop sites from Philippines on a mobile phone plan (which is often the best Internet connection in that country) you also get Cloudflare's captchas everywhere.
I told it before and tell it now again: Cloudflare is dividing the World between first and second/third World countries with their captchas. I call it discrimination of second/third World countries! If you are from US and Europe you will never notice it but if you travel a little bit more you see these blocking captchas everywhere.
I’ve had a similar experience in India with wired internet from a local ISP: CGNAT is used so there are who knows how many customers on the same IPv4 address, https://iknowwhatyoudownload.com/ shows at least forty hours of movies being downloaded every day, the IP address is on half the blacklists out there because someone is part of an email-sending botnet, and yeah, Cloudflare hates you.
I am from Europe and I notice if I use some non residential ip. The captchas are extremely annoying especially when trying to access a site I have already been logged into with 2fa. Who is protected in this case.
I get it browsing from a major ISP in the US. I have the gall to browse in private mode and to block trackers and ads because of all the malware they contain. (And I don't use a browser that requires me to login just to browse the web - gasp!) And apparently, that means I'm worthy of this sort of punishment as well.
The other side of this story is that PLDT stands out from other residential networks as a persistent source of web form spam. I’d love to learn what’s going on differently there.
My assumption is also that something on his network is compromised, and getting his IP into reputation issues.
Tarpitting (serving content slowly from the edge, in order to slow down bots) is necessarily one of the most expensive tools in a WAF/CDN's toolbox.
It's much more likely that something on his network is sending sketchy traffic to CF-fronted/Google sites, and the slow loading he's experiencing elsewhere is because his upstream is being saturated by whatever is happening on his network.
(Author here.) My router isn’t a domestic router. It’s a MikroTik running RouterOS, completely unsupported by the ISP. Outgoing connections and DNS is logged. UPnP is only allowed for the Xbox, PS4, and off-most-of-the-time gaming PC. Nothing out of the ordinary in the logs.
Disabling UPnP doesn't break much. I've used enterprise firewalls at home for years, none of them have UPnP, I've never noticed a problem arising from that lack. I don't have a problem with video games or collaboration tools
UPnP allows devices inside your network to open ports to the outside world without your knowledge. I think everyone should avoid it if they can get by without it
To be frank, that's exactly the problem with NAT-PMP et al. assuming that there's no router bugs: the ability to forward ports has been abused to set up bot relays on hacked IoT devices. This is why I predict that even in IPv6 era we would still have to rely on a TURN-equivalent.
So this gets me thinking. We know Cloudflare will boot a site if they really don't like them. Now, what happens if Cloudflare doesn't like you? I mean, really really doesn't like. Maybe, you said something wrong online or participated in a wrong group activity, or something like that. Is it the case that they have the power to essentially deny you (provided you have a static IP and don't use VPN, say) access to a major part of the Internet? And you can do absolutely nothing about it?
I know they haven't done anything like that yet. But the technical capability is there, and we all know how short is the distance between technical capability and doing it, when the appropriate pressure is applied. So I wonder, how long before activists start demanding for CF to boot people from the internet, and how long before CF caves in to that...
> and we all know how short is the distance between technical capability and doing it
Fact-less conspiranoia.
The CIA has the operators, equipment, and info to be able to kill almost any US citizen in a couple of hours for arbitrary reasons. How many times have they done it?
You are overweighing how much technical capability factors in and very much underweighing the costs of doing something like that. Opportunity costs, collateral damage, unintended consequences, reputation costs, brand harm.
Hell even ethics and morals of those involved. Who do you know would want to work for a company that did that? Who do you know would program that feature and not say anything about it? Why do you believe that CloudFlare would have so many of those kinds of people working there, but you know so few?
Why not make the same complaint about your ISP, your hardware manufacturer, your OS manufacturer? You have exactly the same amount of evidence they are doing this or could do this.
Remember that US criminal system attributes 3 elements to a crime: {means, motive, and opportunity} and even then we use evidence and an assumption of innocence. You just threw out every part except “means”.
I’m not defending CloudFlare here so much as tired of conspiracy theories and paranoia and social panics. We have enough of those things right now.
> Who do you know would want to work for a company that did that?
Pretty much anyone who works for Twitter, Facebook, Google, Paypal, Venmo, Amazon, Microsoft, Gofundme, Mailchimp, Tiktok, Reddit, Nextdoor, and many other tech companies routinely engaging in censorship and unpersoning. The idea that people in tech are some kind of high morals freedom lovers that would never work for a company that censors doesn't suffer even minimal scrutiny. If anything, they'd refuse to work for a company that doesn't censor enough - Twitter workers were in utter screaming panic when they thought Musk could but Twitter and relax the censorship a bit. So if anything you just disproven your own argument - maybe what will force CF to censor is not external pressure but the internal one. I don't see why Cloudflare workers would be any better than Twitter ones.
> The CIA has the operators, equipment, and info to be able to kill almost any US citizen in a couple of hours for arbitrary reasons. How many times have they done it?
I love how people reflectively answer with cries of "no evidence!" to something that presents the evidence about exactly the thing they are claiming has no evidence. I get a distinct impression that the only person they're trying to convince is themselves, by self-hypnotically denying the reality in public.
There's a fact of CF booting sites, there's a fact of CF having IP blacklist, there's a fact of getting into IP blacklist being a very frustrating experience, there's a fact of various activists itching to make the lives of their political enemies a very unpleasant experience and launching successful pressure campaigns to do exactly that.
Did that happen with CF and IP blocking? No, I explicitly said it didn't, at least - I don't know any cases of it. But there's a lot of facts confirming there's a capability and motivation to do so. You may not believe it would happen, and you have a right to believe so, but when you are denying known facts, I don't think your beliefs are based on anything but wishful thinking. Your argument would be strong if you showed that, despite the known facts, it still couldn't happen. But instead to claim it couldn't happen you have to deny the facts.
> How many times have they done it?
Probably more than I know, but it's too big to bother with me, so I'm not too concerned about it right now. Maybe if I was in the same business as Assange, I'd be worried more.
> very much underweighing the costs of doing something like that.
Like what costs? You mean to say, no major provider would dare to boot the person from the Internet? Like Facebook, Twitter, Paypal, Venmo, Gofundme, Google, Amazon, Microsoft, Mailchimp, Tiktok, etc. would not dare to block people for political dissent and expressing unpopular opinions? Because, you know, opportunity costs, collateral damage, unintended consequences, reputation costs, brand harm. That' just couldn't happen. All that is fact-less conspiranoia.
> Why not make the same complaint about your ISP, your hardware manufacturer, your OS manufacturer
I can buy different hardware. I can install different OS. With some effort, but I can connect to a different ISP. Any of that won't help if Cloundflare would refuse to talk to me.
> Remember that US criminal system attributes 3 elements to a crime
Oh, but that's not a crime. That's the beauty of it - remember, it's a private action of a free enterprise, and you have no rights there. And even if the government would hold weekly meetings with Cloudflare suggesting them who exactly needs to be banned, it's still free enterprise, right? I mean, excluding the fact that the government would never do something like that, because reputation costs, brand harm, etc. That's another instance of fact-less conspiranoia, of course.
> I’m not defending CloudFlare here so much as tired of conspiracy theories and paranoia and social panics.
That's nothing. Imagine how tired you'd be when it turns out everything you thought is "paranoia" is actually happening. Of course, it would never happen to you - you'd never disagree with the government, or any people in power, or voice any unpopular opinions in public, would you now?
There’s a real lack of education I’ve seen in developers for small projects who go directly to cloudflare for anything and everything. They don’t understand that they are immediately losing a large chunk of their user base who is either from the third world or is privacy literate. Devs working on projects that are targeting those groups need to understand the tradeoffs from using cloudflare.
Obviously they don't. Clourdflare's markets itself as a super easy set-it-and-forget-it solution. The problem is that it isn't. The defaults are broken and it requires careful configuration and monitoring. Of course this isn't good marketing so the only way a user can know is posts like these or to accidently block their users and hear the reports. (Obviously Cloudflare's UI will only tell you how evil bots it blocked were.)
Readit News
Cloudflare shares IP reputation data with partners like Google, coordinated through a program called the Bandwidth Alliance. So, my original offense might not even have been against Cloudflare. It might have received the reputation data from a partner, and it just propagated through the Bandwidth Alliance network.
That's not what Bandwidth Alliance is at all. It's about reducing or eliminating egress fees between a cloud provider and Cloudflare. Not sure where the idea that it's about sharing IP reputation data comes from.
https://www.cloudflare.com/bandwidth-alliance/
So, if Google Search started showing a CAPTCHA that's not Cloudflare.
I've been gradually removing cloudflare based CDNs from services I develop and control because I don't want my users being arbitrarily discriminated against.
There was a good article posted on HN recently titled "The ideal level of fraud is non-zero" which I think is highly relevant here... In essence any mechanism employed to prevent illegitimate use comes with a negative cost to legitimate users, if that cost is too high it defeats the purpose. i.e what's the point in a website that is completely immune to a botnet and also cannot be accessed by anyone else? unplugging the ethernet cable also effectively protects against botnets. More subtly the cost of outright rejecting some legitimate users is usually not worth the savings of rejecting 100% of illegitimate ones. I think Cloudflare's service has it the wrong way around: it currently accept blocking legitimate users far too easily, that is not an acceptable cost; whereas you should be letting a higher level of bots through to avoid pissing off legitimate users - if it's not obviously a DDoS, it's probably worth the bandwidth cost.
Consider the bigger picture, if you save a slither of a penny by blocking a bot, but also end up blocking or seriously inconveniencing 10 real users... is it worth it.
The space is in need of solid competitors to break the stranglehold they have on the internet. Whether it's the right combination of services, documentation, etc.
> The past few weeks I've been getting tons of redirects to verify my humanity before being allowed to view a webpage. Usually I just have to click the box that says human, not find all the ladders in a photo. SoFi is doing it every single time I log in. Petco, too, along with others who are more sporadic. This is happening with and without uBlock on. Same browser I've always used. ...
SoFi and Petco both use Cloudflare. I do exactly zero web crawling / scraping / abusive anything from my home connection.
I'm noticing a recent increase in volume of complaints about Cloudflare's human verification filter. I'm starting to wonder if they touched a dial.
I had already started pulling some infra back from Cloudflare after their last appearance in the tech news cycle. Now I've got an additional reason to continue doing that.
Essentially, for a broad class of web-based businesses, they have made themselves gatekeepers. I'm sure they'll find a profitable use for this position. Charging outright would look bad, but investing in businesses that just happen to not run into Cloudflare-based trouble, but whose competitors do...
I encourage those of you attempting to block Cloudflare to try and host your own website for a bit. Make sure you don't do it on a metered/paid connection. I know one eCommerce site with 1,300 employees that went bankrupt overnight thanks to the AWS bill (and lack of options to get back online, this was prior to companies such as CF). Bankruptcy as in the company filed for bankruptcy and no longer exists. They were profitable for a decade prior. One DDoS attack...
Also make sure you don't have a democratic opinion if you are in the US, like a 50 person manufacturing company. They were shut down completely thanks to saying a single wrong thing about Republicans. CF existed there, but they weren't aware thanks to not having IT folks. They were a non profit.
CF may be evil to some, but there is a reason they exist. I use CF. I don't like throwing money at them every month, however, many of my websites have also been attacked, usually via competitors. We can either deanonymize the internet or allow companies like CF to exist. There is really no other way.
It comes from the Cloudflare blog. https://blog.cloudflare.com/cleaning-up-bad-bots/
There’s a support page about it too. https://developers.cloudflare.com/bots/get-started/free/
Edit: team tells me this idea never got off the ground. Did talk with some potential partners (which did NOT include Google) but didn’t happen. So if Google was throwing CAPTCHAs it wasn’t because of our IP reputation.
Really?
Dead Comment
How about _also_ pointing to a knowledge base article for how an end user could go about working out what network activity from their IP might be flagging Cloudflare’s systems?
One source of that would be a blog post on your company's website that was actually authored by you! Point 2 below:
>"Once enabled, when we detect a bad bot, we will do three things: (1) we’re going to disincentivize the bot maker economically by tarpitting them, including requiring them to solve a computationally intensive challenge that will require more of their bot’s CPU; (2) for Bandwidth Alliance partners, we’re going to hand the IP of the bot to the partner and get the bot kicked offline; and (3) we’re going to plant trees to make up for the bot’s carbon cost. [1]
So it's not such a far-fetched notion is it?
[1] https://blog.cloudflare.com/cleaning-up-bad-bots/
https://developers.cloudflare.com/firewall/recipes/block-ip-...
I was surprised to learn Cloudflare was born out of Project Honeypot, so I am guessing Cloudflare does share data with them:
https://www.projecthoneypot.org/cloudflare_beta.html
Deleted Comment
That person should start with the assumption they haven't been misclassified and eliminate the possibility that a device on their network is compromised.
[1]: https://easydns.com/blog/2020/07/20/turns-out-half-the-inter...
We did have it working mostly fine for some time back in 2021 but haven't been able to since.
There are multiple open issues reporting this on the GH repo with no real follow-up from maintainers apart from maybe a "should be fixed, open again if still an issue".
ie https://github.com/privacypass/challenge-bypass-extension/is...
Probably from scam called mail blacklists
[0] https://www.cloudflare.com/bandwidth-alliance/
Deleted Comment
If two independent sites believe you are a bot, you or something at your address just might be.
Spoiler alert: many websites simply refuse to load at all (e.g. any google service, and lots of websites "protected" by CF). Captchas are everywhere: in many cases, you can't even complete simple GETs of blogs without donating free labor to CF.
And the most infuriating part, you get CF marketing messages right in your face while your browser is calculating hashcash (I guess?)... At this point I can recognize every single one of them: something about bots making up 40% of all internet traffic, something about their web scraper protection racket, something about small businesses (???), etc etc...
To be fair, Tor exit nodes have an awful reputation for sure. Nevertheless, I have a hard time forgiving how CF makes browsing the Internet hell for those who actually need Tor.
Yeah, there's something amazingly aggravating about CF telling you how much traffic is bots while showing that they can't distinguish you from a bot.
The fact that humans are seeing the traffic meant for bots is an unfortunate side-effect.
I personally welcome our future bot overlords (not only because being unwelcome might be unhealthy for me — why would I publicly disagree with an overlord or not want to be their friend?).
I have bad news about the most-likely fix for it, longer term, so we can lay off the IP-based reputation stuff and the geo-blocking: it's tying some form of personal ID to your browsing activity, so that bears the reputation instead of the address.
Sorry. Said it was bad news.
Basically, the core problem is digital identities (accounts, IPs, phone #s etc.) are cheap to create (even considering captchas and all) so fraud is easy. The solution could be just to make it "costly" to create new digital identities. For example, you could get a "verified but anonymous" identity issued by locking some assets (could be real world money, or maybe something intangible like community reputation) as collateral with a trusted party (or, for the crypto people, the blockchain). If you misbehave, you lose your reputation on that identity (and essentially your collateral) and have to start over. This lets anyone bootstrap a "minimal" level of trust at the beginning before they can use time to prove themselves trustworthy.
Note: This model might remind some of things like staking in crypto. However the idea is really not anything new... Putting money on the line is really how most low-trust bootstrapping happens.
*: To name a few:(1) this can result in participation being gated by wealth, which can be unfair. (2) it makes accounts more valuable to hack so people need better security practices [re: twitter checkmark]. (3) one would need some authority to decide how accounts lose their collateral or maybe the collateral is just burned to create that initial credibility...
We already use this model in practice. It's why so many services require a phone number verification now - they are hard enough to get en-masse, especially if you block things like Google Voice. They even have a big advantage in that they are comparatively hard to hack, as the SIM card is effectively a weak form of physical security key.
I think the big problems this causes is discussed on HN quite often.
Similar, the internet is already very difficult for the people with limited means. This would make it even harder.
I've always thought that client certs would be an interesting solution to this problem. Any given certificate can carry signatures from multiple signing authorities, right? So we could imagine a world where there are many different certificate authorities, each of whom have their own criteria for signing a particular certificate and each of whom offer different varieties of assurance regarding the signature-holder's identity.
From here, the question of "should I allow the user identified by this client cert to use my service" simply becomes a question of 1.) checking the validity of the signatures of the client cert and 2.) deciding if the CA's criteria for signing certs aligns with my desired userbase.
For example, a particular CA might insist that their users go through some real-world process to renew their certification every few years, but when they sign a cert it means that the bearer has been strongly vetted as a real person.
An interesting side effect of this auth model is that a service provider accepting certs from a particular CA has someone to complain to if a user bearing their signature acts improperly on their platform. You could imagine a CA which has a code of conduct expected of the users whose certs they sign, and would perhaps revoke a user's certification if too many websites complain.
This also reminds me of the anxiety of Google deciding to just ban my account for some reason. They can't be bothered to commit resources to making sure mistakes can be resolved. They don't care to lose a fleetingly small percentage of customers.
Not sure I have an answer. Just a thought.
I'm not understanding the generalized sentiment here. How would, for example, a retailer benefit from this strategy? How does it protect their bottom line?
I can see how a particular kind of "facilitated user economy," such as games, gambling and promotional companies could benefit, but it doesn't seem that broadly applicable to what most people would consider a "mainstream" business.
> so we can lay off the IP-based reputation stuff and the geo-blocking: it's tying some form of personal ID to your browsing activity
And a new market for identity theft is born.
Also, as someone who serves content and geo blocks it, that's not up to me, that's up to the owner of the content or whoever happens to be licensing it for them. So, even if you sent me a picture of your government ID, it changes nothing.
The amount of automated and apparently-manual attempted credit card fraud (and exploit attempts, for that matter) any halfway-prominent site with a CC form is subjected to is hard to appreciate if you've never seen it. It's a whole lot. They aren't even necessarily trying to buy what you have, but to validate that their stolen cards work. And they're quite busy. If too much of that gets through—really, any more than a very tiny amount of it gets through—you're gonna have an extremely bad time.
Various CC service providers like Stripe do provide tools to try to block those attempts, but defense in depth is usually a very good idea, including fairly aggressive firewall-level blocking.
A couple of examples I can think of is blocking bots from scraping their site for pricing and details and from resellers from buying up all of the stock (see sneakers, electronics, etc). The last example doesn't directly impact their bottom line, but it will make customers go elsewhere.
That wouldn't just be bad news, it would be disastrous news. It would immediately render the entirety of the web worthless to me.
And once your spammer has been identified then that's them banned/removed, unable to sign up again.
Deleted Comment
A few months ago I got on Akamai's naughty list (with my other ISP) for some very light automated website downloading. That was a straight block with HTTP errors and I had to use a proxy to access the Web. It cleared up after a few days.
The lack of any user feedback or support for this situation is really annoying. Reminds you how much power the CDNs have. It'd be really bad if loading websites got as difficult as sending email through all the layers of spam filtering.
I've been noticing this too, and it's why Starlink remains my secondary ISP/bulk transfer connection. If I had to drop one connection, I'd drop Starlink for this reason alone.
There are some sites that I simply can't browse, and it's not Cloudflare errors, either. Lowes, in particular, simply returns error pages for anything but the main landing page on a regular enough basis. Of course, my observed public IP changes so it's not consistent, but it's genuinely annoying.
Could cloudflare legally charge them a bribe to captcha their users less? It isnt good to have a company in this position of power if so.
Why are you using Starlink at all if you have other options?
Edit: spelling
Starlink is at least 10 better (fewer captchas).
I'm really hoping cloudflare gets busted for having backroom deals with big ISPs or something. (For instance, if the cgnat had a cloudflare CDN cache endpoint behind / accociated with it, I suspect the IP would be white listed.)
I told it before and tell it now again: Cloudflare is dividing the World between first and second/third World countries with their captchas. I call it discrimination of second/third World countries! If you are from US and Europe you will never notice it but if you travel a little bit more you see these blocking captchas everywhere.
I do these things as well. It’s been months since I’ve seen a CloudFlare challenge page.
You would be surprised to see how easy it is to hack domestic routers.
1. Find and disinfect the devices, including the router. If you don’t have enough technical knowledge, then buy a new router.
2. Use 30 character long random password on the router.
3. Disable UPnP.
4. Anything with WI-FI and weak password can be hacked within minutes, so check your other devices as well, especially IOT ones.
Tarpitting (serving content slowly from the edge, in order to slow down bots) is necessarily one of the most expensive tools in a WAF/CDN's toolbox.
It's much more likely that something on his network is sending sketchy traffic to CF-fronted/Google sites, and the slow loading he's experiencing elsewhere is because his upstream is being saturated by whatever is happening on his network.
https://google.com/search?q=mikrotik+botnet
These things are the absolute scourge of the internet.
It's almost certainly compromised.
UPnP allows devices inside your network to open ports to the outside world without your knowledge. I think everyone should avoid it if they can get by without it
Deleted Comment
I know they haven't done anything like that yet. But the technical capability is there, and we all know how short is the distance between technical capability and doing it, when the appropriate pressure is applied. So I wonder, how long before activists start demanding for CF to boot people from the internet, and how long before CF caves in to that...
Fact-less conspiranoia.
The CIA has the operators, equipment, and info to be able to kill almost any US citizen in a couple of hours for arbitrary reasons. How many times have they done it?
You are overweighing how much technical capability factors in and very much underweighing the costs of doing something like that. Opportunity costs, collateral damage, unintended consequences, reputation costs, brand harm.
Hell even ethics and morals of those involved. Who do you know would want to work for a company that did that? Who do you know would program that feature and not say anything about it? Why do you believe that CloudFlare would have so many of those kinds of people working there, but you know so few?
Why not make the same complaint about your ISP, your hardware manufacturer, your OS manufacturer? You have exactly the same amount of evidence they are doing this or could do this.
Remember that US criminal system attributes 3 elements to a crime: {means, motive, and opportunity} and even then we use evidence and an assumption of innocence. You just threw out every part except “means”.
I’m not defending CloudFlare here so much as tired of conspiracy theories and paranoia and social panics. We have enough of those things right now.
Pretty much anyone who works for Twitter, Facebook, Google, Paypal, Venmo, Amazon, Microsoft, Gofundme, Mailchimp, Tiktok, Reddit, Nextdoor, and many other tech companies routinely engaging in censorship and unpersoning. The idea that people in tech are some kind of high morals freedom lovers that would never work for a company that censors doesn't suffer even minimal scrutiny. If anything, they'd refuse to work for a company that doesn't censor enough - Twitter workers were in utter screaming panic when they thought Musk could but Twitter and relax the censorship a bit. So if anything you just disproven your own argument - maybe what will force CF to censor is not external pressure but the internal one. I don't see why Cloudflare workers would be any better than Twitter ones.
How would we know?
Deleted Comment
I love how people reflectively answer with cries of "no evidence!" to something that presents the evidence about exactly the thing they are claiming has no evidence. I get a distinct impression that the only person they're trying to convince is themselves, by self-hypnotically denying the reality in public.
There's a fact of CF booting sites, there's a fact of CF having IP blacklist, there's a fact of getting into IP blacklist being a very frustrating experience, there's a fact of various activists itching to make the lives of their political enemies a very unpleasant experience and launching successful pressure campaigns to do exactly that.
Did that happen with CF and IP blocking? No, I explicitly said it didn't, at least - I don't know any cases of it. But there's a lot of facts confirming there's a capability and motivation to do so. You may not believe it would happen, and you have a right to believe so, but when you are denying known facts, I don't think your beliefs are based on anything but wishful thinking. Your argument would be strong if you showed that, despite the known facts, it still couldn't happen. But instead to claim it couldn't happen you have to deny the facts.
> How many times have they done it?
Probably more than I know, but it's too big to bother with me, so I'm not too concerned about it right now. Maybe if I was in the same business as Assange, I'd be worried more.
> very much underweighing the costs of doing something like that.
Like what costs? You mean to say, no major provider would dare to boot the person from the Internet? Like Facebook, Twitter, Paypal, Venmo, Gofundme, Google, Amazon, Microsoft, Mailchimp, Tiktok, etc. would not dare to block people for political dissent and expressing unpopular opinions? Because, you know, opportunity costs, collateral damage, unintended consequences, reputation costs, brand harm. That' just couldn't happen. All that is fact-less conspiranoia.
> Why not make the same complaint about your ISP, your hardware manufacturer, your OS manufacturer
I can buy different hardware. I can install different OS. With some effort, but I can connect to a different ISP. Any of that won't help if Cloundflare would refuse to talk to me.
> Remember that US criminal system attributes 3 elements to a crime
Oh, but that's not a crime. That's the beauty of it - remember, it's a private action of a free enterprise, and you have no rights there. And even if the government would hold weekly meetings with Cloudflare suggesting them who exactly needs to be banned, it's still free enterprise, right? I mean, excluding the fact that the government would never do something like that, because reputation costs, brand harm, etc. That's another instance of fact-less conspiranoia, of course.
> I’m not defending CloudFlare here so much as tired of conspiracy theories and paranoia and social panics.
That's nothing. Imagine how tired you'd be when it turns out everything you thought is "paranoia" is actually happening. Of course, it would never happen to you - you'd never disagree with the government, or any people in power, or voice any unpopular opinions in public, would you now?
Deleted Comment