5/31/2026 at 2:55:09 PM
Cloudflare is known to use fingerprinting to detect scrapers For example, they use JA3 fingerprints and match them against the UA to block stuff like cURL while allowing OkHttp (Android clients) - but this can be easily be spoofed with packages such as CycleTLS [1].I don't want to defend them, because they gate away a good chunk of the internet with their "bot protection", but unless you do PoW (which is also ecologically a nightmare), probably fingerprinting is the way to go - completely destroying the privacy of everyone involved.
Cromite, a privacy conscious fork of Chromium for Android, has constantly issues with CloudFlare Turnstile [2] because they (Cloudflare) try to fingerprint it in multiple ways in order to pass the challenge. The only way to get it to work would be to join the CloudFlare Browser Developer program - which requires signing an NDA. Rightfully so, the project maintainer didn't want to do it.
If you want to see the extent of what CloudFlare does to fingerprint the browsers, just have a look in the issue [2] and see which flags need to be disabled in order to allow CloudFlare to pass the challenge.
I understand both sides, but at least CloudFlare could be flexible enough to fall back to PoW instead of just blocking people from sending forms or accessing websites...
by denysvitali
5/31/2026 at 4:40:00 PM
> I don't want to defend them, because they gate away a good chunk of the internet with their "bot protection"They also gate away a good many people with their "bot protection". I am extremely worried about how so many seem to have outsourced the control over who can access their websites to a company, with no second thoughts whatsoever.
by jwr
5/31/2026 at 6:55:28 PM
The problem is what is the alternative? I'm (not) defending them or this practice by any measure, but we all know what happens if you just open your site up without these, especially with AI bots which hammer servers and are in effect a legalized DDoS system. I've hated CAPTCHAs ever since I first encountered them and I can't wait for them to just finally die a permanent death, but I also don't know how we solve the "how do you identify a human and a bot" in a way which doesn't require server admins to have extremely beefy servers or similar setups to handle the extra load. I'm not going to do the "there HAS to be a way thing" either because, for all I know, this could just be one of those impossible-to-solve problems.by ethin
5/31/2026 at 7:08:10 PM
> we all know what happens if you just open your site up without these, especially with AI bots which hammer servers and are in effect a legalized DDoS systemNo, we don't know. I honestly do not understand the problem. I run websites, both static and non-static. Granted, my sites aren't exactly the most popular internet go-to destinations, but I should be seeing this DDoS too, right?
I do see lots of requests. Nothing that any modern system can't handle. Computers are stupid fast these days. Unless you are doing something unreasonable, it's really hard to even notice this "extra load".
I understand there are sites for whom this causes problems, but I think these are rare and could be optimized not to do unreasonable things.
I think too many people are annoyed by AI companies (arguably understandable position), look at their logs and speak of "hammering", "DDoS" and "extra load", while in reality it doesn't matter much.
by jwr
5/31/2026 at 8:53:12 PM
I second this. My website exposes a cgit and 99% of the traffic now is AI scraping the sources, but the load is nowhere near DoS territory. And this is running on the cheapest VPS I could find.Not saying I'm not annoyed by the scraping; I am looking to block them, but I'm also not going to put the site behind the gatekeeper. If anything, Cloudflare must love AI scraping now for the same reason AV companies love malware.
Now, if you are running a PHP stack...yeah, maybe that's the problem right there.
by canyp
5/31/2026 at 8:35:50 PM
It might depend on the tech stack. I run a small niche website but it has PHP and a database (MediaWiki/PHPBB) and without Cloudflare I'd estimate I'd need to spend several hundred dollars a month to handle the traffic. Traffic used to be tens of thousands of requests a day. AI has increased that to between 400k and 3M requests per day but it's not a smooth distribution. This is with bot fight mode on that greatly reduces traffic.I adopted Cloudflare because it was getting DDoSed by the AI crawlers. I'm pretty sure all of them are vibe coding their crawlers and don't bother adding rate limiting as a requirement.
by matt_heimer
5/31/2026 at 7:34:35 PM
Has anyone pointed an AI scraper at your server at all? Unless your website appears in search engine listings I don't think the AI scrapers will slam it. My server has never been hit by them but my server is also practically unknown. All of this said, I'm not going to claim that server loads can handle it because many sysadmins have claimed otherwise, and I would like to think that their claims are reliable.by ethin
5/31/2026 at 7:42:32 PM
As soon as you get your TLS certificate you get bombarded with scraping. You don't need someone to "point a scraper at you".What matters most is usually how much there is to scrape. If you have like 5 pages that's nothing. For forum like websites where each thread, each user profile, etc. gets scraped that's when traffic increases. I just let them have at it with no issues though, computers are fast.
by redox99
5/31/2026 at 7:42:37 PM
Also, how do we even know they're really "AI scrapers", or just a deliberate DDoS to push sites into using CF or other "anti-bot" providers?by userbinator
5/31/2026 at 8:13:45 PM
A small, single EU country focused non-static e-commerce, with proper robots.txt instructions that worked perfectly well in the search & co bots -only "era" with rate limiting for nginx/php-fpm setup - is kinda struggling without CF to handle 15000 requests per 15 minutes, coming from Chrome "users" from IPv6. Best so far was an avg. server load in htop = 40 on an 8-core server x_xby dr_um
5/31/2026 at 7:39:18 PM
You get downvoted for these opinions but I agree. Most people that complain that their servers get hammered by AI bots are those that run very unoptimized servers that can only handle like 100 rps. I've never had any issues with any of my moderately optimized websites. A $10 VPS can handle sooo much traffic.by redox99
5/31/2026 at 8:29:19 PM
The most plausible near-term path is probably micropayments embedded invisibly in AI agents. Your agent that has learned what you value and can make a reasonable decision to allow a micropayment for certain content pays on your behalf without requiring a conscious decision each time, eliminating the mental transaction cost problem entirely. It's the mental transaction cost that arguably led to the failure of the micro payment model back in the early 2000s.Although the cynical part of me says that this will result in malicious actors trying to trick agents into giving out a bunch of micro payments. There are counter defenses that can help detect and compensate for that, but perhaps the best we will be able to do is prompt user with the default agent recommendation.
by steelframe
5/31/2026 at 7:45:57 PM
I don't think it's just privacy, it also increasingly turns the web itself into a walled garden. The end result is that websites can only ever be accessed by "approved" clients - the latest Chrome, Edge, Safari and if you're lucky Firefox - and nothing else.by xg15
5/31/2026 at 5:55:05 PM
I can no longer access any website that's "protected" by Cloudflare. As soon a website enables that stuff… "Shoot, another one bites the dust." I wonder if the website owners realise at all how many actual users they lose by this sort of "protection."by binaryturtle
5/31/2026 at 6:18:07 PM
Cloudflare will just tell them that 70% traffic drop is because 70% of their traffic was bots, and everything is working fine, and hey, don't you want to upgrade to a paid plan to block 50% of the remainder? Think about how many bots will be blocked with that upgrade!by tardedmeme
5/31/2026 at 6:46:25 PM
I'm one of those who have enabled cloudflare on all of the sites I maintain. Additionally, Added turnstile on every form.I know some actual users get blocked. But the amount of spam we get without it, the amount of bot traffic simply overwhelming the server... It is just too much.
Recently I also hard blocked all IPs from china Singapore India Pakistan Russia and whole of africa. Do I want to do it? No. But the amount of bot traffic and corresponding spam is a bigger problem :(
by CrimsonRain
5/31/2026 at 8:45:52 PM
> I know some actual users get blocked. But the amount of spam we get without it, the amount of bot traffic simply overwhelming the server... It is just too much.
One of the core tenets of system design is Availability. If your service is not available - if your forms are blocking legitimate users - then why are you pretending to have a form submission feature at all? Just to frustrate users?
by dotancohen
5/31/2026 at 8:34:28 PM
I took the time to write to one on LinkedIn and they didn't replyby dwedge
5/31/2026 at 6:36:48 PM
>I wonder if the website owners realise at all how many actual users they lose by this sort of "protection."How many people do you think are browsing with a weird enough config (eg. custom browser like OP, or some weird config like firefox with fingerprinting protection on a raspeberry pi) to trip cloudflare's protection?
by gruez
5/31/2026 at 8:04:32 PM
I got locked out of some websites by Cloudflare Turnstile on some very standard configurations, like an iPhone on Safari, or a Windows 11 desktop with Firefox or Edge, neither with a VPN on. I never found out why.by benhurmarcel
5/31/2026 at 6:47:53 PM
Well… I know plenty people in my circle affected by this. Just have a slightly outdated system you simply can't afford to update: it's way to easy to get cut off like this. IMHO, a rather systematic discrimination of poorer people.by binaryturtle
5/31/2026 at 8:15:08 PM
There are dozens of us :)In my experience what really makes it loop every single time though is JShelter. CF doesn't like having your fingerprintable data bits messed with.
There are legitimate uses for non-instrusive, ethical and legal scraping, but some of us have had to resort to extreme measures:
by ranger_danger
5/31/2026 at 8:39:21 PM
Do you by chance have that installed? I don't use Cloudflare but I am curious if that code can scrape my silly blog? [1] Trying to pick the appropriate article... I'm guessing it can. I dont do the fancy javascript or TLS fingerprint inspections, just some janky hill-billy protections and silly redirects.[1] - https://blawg.nochan.net/b/Internet-Crap/20260522-Maybe-AI-B...
by Bender
5/31/2026 at 4:58:34 PM
They sometimes have to comply with legal requests (which I understand), but at the same time they have a huge market share - which means that the internet is becoming less and less decentralized and more in their control. We've seen the effects of that in previous outages...by denysvitali
5/31/2026 at 5:22:00 PM
>I am extremely worried about how so many seem to have outsourced the control over who can access their websites to a company, with no second thoughts whatsoever.I think the Web is on its last legs, anyway. Generative AI and LLM-instead-of-search has destroyed what little value remained.
by stackghost
5/31/2026 at 6:17:10 PM
It's just one more facet of the enshittoscene, the era where actual product quality is completely irrelevant. Put it in the same bucket as websites that lag when you scroll, apps that refuse to show you video without a huge play/pause button overlaid in the middle of it that never goes away, and the movie Melania. My hypothesis is that billion-dollar businesses no longer exist to sell things to customers, but only to impress other billionaires to get their investment money.by tardedmeme
5/31/2026 at 5:29:26 PM
> I don't want to defend them, because they gate away a good chunk of the internet with their "bot protection", but unless you do PoW (which is also ecologically a nightmare), probably fingerprinting is the way to go - completely destroying the privacy of everyone involved.Bot protection with fingerprinting is just an illusion. Any signals like this which is on client side can be spoofed by an above average person. Fingerprinting is just way to consolidate the market for advertising business. Assigning Reputation to residential IP addresses and commercial blocks is is another approach to achieve the desired result. Providers would be a lot more careful to allow their IP addresses for misuses, however turns out that it would bring down the DDOS business on both sides, attackers and protectors.
Ironically, more than often its the same companies that invest in building their own bots and finding ways to stop bots from other companies.
by sandeepkd
5/31/2026 at 6:09:09 PM
> Bot protection with fingerprinting is just an illusion. Any signals like this which is on client side can be spoofed by an above average person.At the upper bound, fraud can always be committed by paying real people with real accounts to perform the desired action in a way that is 100% truly indistinguishable from organic. There's fundamentally actual prevention technique at the limit.
So the entire game is only "increasing the costs until it's not viable ROI", not "holistically prevent", which is why fingerprinting is a relevant technique here.
by esrauch
5/31/2026 at 8:06:28 PM
> entire game is only "increasing the costs until it's not viable ROI", not "holistically prevent", which is why fingerprinting is a relevant technique here.As per cloudlare's own report, about 78% of the DDOS attacks are at the network layer where the fingerprinting technique is not useful.
DDOS is done against targets for certain reasons, most businesses are not even viable targets for everyone.
However letting everyone being fingerprinted on the pretext of solving the DDOS is where the privacy gets compromised (not much of it is left though). Some search engines did it indirectly by letting people use tag managers for free in their website and then utilize the data for their advertising business.
Relatively the end game is same, its just how these companies are approaching it.
by sandeepkd
5/31/2026 at 7:34:00 PM
Fingerprinting for "bot protection" is indistinguishable from fingerprinting for mass surveillance.by leonidasrup
5/31/2026 at 3:47:49 PM
it's all for nothing, because Cloudflare's scraping protection works about as well as a $5 padlock - good enough to dissuade bored teens, not good enough to dissuade even an amateur burglar. if someone wants to scrap your publicly visible data, they will. there's nothing you can do.by b65e8bee43c2ed0
5/31/2026 at 3:49:06 PM
At the same time: it sure works well enough to annoy anyone with a "bad ASN" IP with 80 captchas a day.by ACCount37
5/31/2026 at 4:18:40 PM
exactly that's what I was thinking... like the day they provided a solution to the issue they posedby shideneyu
5/31/2026 at 5:28:30 PM
Exactly. I’m constantly amazed at how little you actually need to bypass CF, Amazon, Azure WAFs and so on (Incapsula springs to mind too). When you look at the code you’ve come up with, it’s actually quite small and compact.More to the point, these systems actually help scraping because proof of work unlocks essentially unlimited scraping, in my experience.
That said - from my experience on the other side, sure you can’t stop people like me or you, but you can stop 99% of the others. That’s more than worth it operationally.
by mootothemax
5/31/2026 at 8:34:24 PM
Brave has aggressive fingerprinting protection, I have Auto-Shred (formerly Forgetful Browsing) turned on, I use VPN and yet I rarely get gated out.by jorvi
5/31/2026 at 8:40:27 PM
A testament to how well Brave protects you from being identified by [Cloudflare in this example]by high_priest
5/31/2026 at 4:25:52 PM
> but unless you do PoW (which is also ecologically a nightmare)Can you expand? I don't see a problem with some napkin math. 5W load for 2 seconds is 0.002Wh (we have to let smartphones pass and not by doing PoW for 10s of seconds). 8 billion checks a day for a year = 8GWh.
by petu
5/31/2026 at 4:55:52 PM
I stand corrected. It's not a nightmare scenario (as for Bitcoins) - but I'm still of the idea that "useless" computations should be avoided (as we should avoid having 10MB websites).In any case, according to some napkin math done by Kimi 2.6 (which by itself is probably already consuming more than all of my PoW challenges for the upcoming 5 years) - the situation looks incredibly in favor of PoW: https://www.kimi.com/share/19e7ef40-a432-8912-8000-0000b4a71...
Which makes me wonder why CloudFlare isn't switching to this already
by denysvitali
5/31/2026 at 5:03:36 PM
Because it doesn’t solve the problem of residential botnets.by dcrazy
5/31/2026 at 8:51:23 PM
Neither does fingerprinting.by dotancohen
5/31/2026 at 6:34:45 PM
They're also anti free speech.by gadders
5/31/2026 at 3:20:43 PM
This is why I have two separate browsers. If you want to do official stuff like paying for things you need to get through cloudflare.by PearlRiver
5/31/2026 at 4:16:07 PM
You can use Firefox with different profiles and configure it to launch particular profile directly, without launching default profile and using about:profiles.Firefox with a non-default profile can be created like that:
./firefox -CreateProfile "profile-name /home/user/.mozilla/firefox/profile-dir/"
# For, say, cloudflare that would be:
./firefox -CreateProfile "cloudflare /home/user/.mozilla/firefox/cloudflare/"
./firefox -profile "/home/user/.mozilla/firefox/profile-dir/"
# For cloudflare that would be:
./firefox -profile "/home/user/.mozilla/firefox/cloudflare/"
- create a copy of it, say, /usr/bin/firefox-cloudflare
- adjust the relevant line, adding the -profile argument
Of course, "./firefox" from examples above should be replaced with the actual path to executable. For default installation of Firefox the path would be in /usr/bin/firefox script.
So, you can have a separate profiles for something sensitive/invasive (linkedin, cloudflare, shops, banks, etc.) and then you can have a separate profile for everything else.
And each profile can have its own set of extensions.
by notafox
5/31/2026 at 6:19:07 PM
They're blocking Firefox quite often. Stripe does something that makes Firefox hang. I use Chrome for those sites and then go back to Firefox...by tardedmeme
5/31/2026 at 5:29:41 PM
You do now do this from `Profiles` menu too, without going down to CLI path. It's extremely simple now.by t_mahmood
5/31/2026 at 7:23:18 PM
If that works for you - that's fine.I'd argue, that for some, CLI path is actually cleaner.
You see, the way described above creates entirely separate points of entry, and you don't have to go to the central menu to launch specific profile.
It eliminates one step (Profile Manager, about:profiles or whatever) allowing you to get faster to the desired profile - same way you'd launch a default profile.
It's logical separation too. It's like separate browsers from UX standpoint (they do use the same distribution though ...unless they aren't - you can configure different distributions for different profiles - nothing stops you from that).
by notafox
5/31/2026 at 8:39:17 PM
We are not in any kind of disagreement :)I'm just leaving the information about the gui option to other who may not be aware that it can be done from the gui too, and think its difficult to do in Firefox.
by t_mahmood
5/31/2026 at 4:46:48 PM
Except that fingerprinting means that both profiles are actually tied together by cloudflare (and other tech companies)by ferfumarma
5/31/2026 at 5:55:00 PM
I think the idea is that they have the functionality that cloudflare is using to generate the fingerprint (like webGL in this case) disabled in their non-cloudflare profile and only use the cloudflare profile to do things they have to that are behind cloudflareby VoidWhisperer
5/31/2026 at 3:40:22 PM
Firefox added profile switching recently. Works good.(That said, I still keep separate machines. One for doing "official" things, the other for everything else)
by helterskelter
5/31/2026 at 4:18:19 PM
> Firefox added profile switching recently.I think this was as recent as 25 years ago?
Recently they added some new UI. There was and still is (I think) classic Profile Manager UI, which you can launch with
./firefox -ProfileManager
But you don't have to use any of those anyway - see my comment above (a response to parent).
by notafox
5/31/2026 at 5:01:20 PM
They actually have at least 3 kinds of profile: 1. containers - As they say its somekind of sandbox, technically a profile 2. profiles that are accesible through about:proflies, which they had for years, and probably the one you are talking about... 3. New profiles that comes with a pop-up much like how chromium browsers shows itby opem
5/31/2026 at 5:35:30 PM
The old UI was pretty difficult to use, and hard to discover unless you knew where to look though.by thayne
5/31/2026 at 3:57:06 PM
Odd - they've had that for years, but only on the command line. Wonder if it's different under the hood? They also have firefox containers which also never quite became a first-class feature (you have to install a plugin).by ajb
5/31/2026 at 3:53:18 PM
>Works good.does it? same binary, same machine, same display, same 781 other heuristics.
by b65e8bee43c2ed0
