How to get past Cloudflare when scraping public data
By Mechelle Henderson · Published: 23 July 2026 · 7 min read
TL;DR: Cloudflare does not run one bot check, it runs several - IP/ASN reputation, TLS and HTTP/2 fingerprinting, and interactive JS challenges (Managed Challenge / Turnstile). You get past it by matching each layer: a residential proxy for the IP, a browser-accurate TLS fingerprint (curl_cffi, or a real browser), and a real browser that runs JavaScript for the challenge pages. Which combination you need depends on which layer is actually stopping you. Scrape public data only, and rate-limit politely.
Cloudflare is not one wall, it's three
People talk about "bypassing Cloudflare" as if it were a single lock. It is not. Cloudflare's bot management runs several independent checks, and a request has to satisfy all of them. When you are blocked, the useful question is not "how do I bypass Cloudflare" but "which of its checks am I failing?" - because each one has a different fix.
| Layer | What it checks | How you fail it | The fix |
|---|---|---|---|
| IP reputation | The exit IP's ASN | Requesting from a datacenter range | Residential proxy |
| TLS / HTTP2 fingerprint | Your ClientHello (JA3/JA4) | A Python/library fingerprint | curl_cffi or a real browser |
| JS challenge | Whether JavaScript runs | No JS engine (raw HTTP) | A real browser that executes JS |
Layer 1: IP reputation - the one you can't fake from code
Before Cloudflare looks at anything your client sends, it scores the IP you are coming from. Datacenter ASNs (AWS, Hetzner, OVH) carry a standing penalty because real visitors almost never browse from them. No header, no fingerprint trick, and no browser can change the ASN of the IP you exit from - only routing through a different network can.
A residential proxy gives Cloudflare a real home-broadband IP to score. On sites that rely mostly on IP reputation, this single change is often the whole fix. On harder sites it is necessary but not sufficient - you still have layers 2 and 3 to clear.
Layer 2: TLS fingerprint - the silent block
During the TLS handshake, Cloudflare hashes your ClientHello into a JA3/JA4 fingerprint. Python's requests and httpx produce a fingerprint no real browser emits, so Cloudflare can block you silently, before a single byte of the page is sent - no visible challenge, just a 403 or a stall.
If you are scripting with HTTP requests rather than a browser, restore a browser-accurate fingerprint with curl_cffi. That clears layer 2 — but as our measurements below show, clearing layers 1 and 2 is necessary and not sufficient on sites running Cloudflare's aggressive bot mode:
from curl_cffi import requests
proxies = {"https": "http://USER:PASS@gw.roamproxy.com:41080"}
r = requests.get("https://target.example", impersonate="chrome124", proxies=proxies)
print(r.status_code)What we measured (July 2026)
Most advice about Cloudflare is asserted, not measured. We ran the two HTTP-layer
defences against each other on real targets, from our own gateway, so the numbers below
are reproducible rather than folklore. Two variables, four combinations: exit IP
(a Psychz datacenter address vs a Frontier Communications residential line, both US) and
TLS stack (Python requests vs curl_cffi impersonating Chrome).
Seven Cloudflare-fronted sites, two passes each.
| Combination | Passed | Result |
|---|---|---|
| Datacenter IP + default Python TLS | 0 / 7 | 403, several with a JS challenge page |
| Datacenter IP + Chrome-impersonated TLS | 0 / 7 | identical to the above |
| Residential IP + default Python TLS | 0 / 7 | identical to the above |
| Residential IP + Chrome-impersonated TLS | 0 / 7 | identical to the above |
Two things in that table are worth sitting with, because both contradict advice you will read elsewhere.
The residential IP bought us nothing on these targets. Not a single site in the set treated the Frontier residential exit differently from the Psychz datacenter one. That does not mean IP reputation is a myth — it means these sites never got as far as judging our IP, because a request that fails the bot-management check is refused on other grounds first. IP quality decides outcomes on the middle of the difficulty curve; at the hard end it is not the binding constraint, and buying better IPs will not move it.
Chrome TLS impersonation alone did not flip a single site either. These targets require executing the JavaScript challenge, which no HTTP client does. If you are stuck at 403 on a site like this, adding a proxy or a TLS library is not the fix — see layer 3.
For calibration, the same script sailed through Cloudflare-fronted sites that are not running aggressive bot mode, on every combination including plain Python from a bare datacenter host. "Protected by Cloudflare" spans an enormous range of strictness, and it is worth establishing which end of it your target sits at before you spend money on tooling.
Stop chasing a specific JA3 hash
While measuring, we checked what fingerprint our own requests were actually presenting.
Three consecutive curl_cffi requests with the same impersonate="chrome"
setting, from the same host, produced three different JA3 hashes
(a0052cf3…, 0899dce7…, d2de58db…). That is not a bug:
real Chrome shuffles TLS extensions and injects GREASE values, so its JA3 changes per
connection, and a good impersonation library reproduces that. Any guide telling you to match
one specific JA3 string is describing a browser that stopped existing years ago.
What did stay identical across all nine of our requests — direct, via residential
proxy, and via datacenter proxy — was the HTTP/2 fingerprint
(52d84b11…), derived from the SETTINGS frame, header table size, window size
and pseudo-header order. It is far more stable than JA3, which is precisely why it is worth
more to the people fingerprinting you. Check both when you debug a silent block; we used
tls.browserleaks.com/json.
Method: 28 requests across 7 Cloudflare-fronted sites, two passes per combination, 27 July 2026, US exits. A pass required HTTP 200 and a body without challenge markers — a 200 that returns "Just a moment…" is a block wearing a success code.
Layer 3: the JS challenge - when you need a real browser
Some Cloudflare configurations serve an interactive challenge - the "Checking your browser…" interstitial or a Turnstile widget. These work by handing the client JavaScript that a genuine browser executes to prove it is one. A fingerprint fix does not help here, because there is no JavaScript engine in an HTTP library to run the challenge.
The reliable answer is to use a real browser that runs the JS - headless Chromium via Playwright, or an LLM-driven agent on top of it. Point that browser through the same residential proxy and it clears all three layers at once. See using residential proxies with browser-use for the browser-plus-proxy setup; the same proxy dict works for plain Playwright.
Match the tool to the layer
- Blocked by IP only (works locally, fails from a server) - a residential proxy is usually the whole fix.
- Silent 403 with correct IP and headers - a TLS fingerprint block; add curl_cffi.
- Visible "Checking your browser" / Turnstile - a JS challenge; use a real browser through the proxy.
Do not reach for a full headless browser when a residential IP plus curl_cffi would do - the browser is slower and heavier. Escalate only as far as the target's actual defense requires. For the broader checklist of why scrapers get blocked, see how to avoid getting blocked while web scraping.
Collect public data responsibly. These techniques are for scraping publicly accessible pages at a reasonable rate. Honor robots.txt and a site's terms where they apply, do not access data behind an authorization you do not have, and rate-limit so you are never degrading the site for its real users.
FAQ
Why does the same request work in my browser but not in my script?
Because your browser passes all three of Cloudflare's checks and your script passes none of them. Your browser exits from your home residential IP, sends a real Chrome TLS fingerprint, and runs the JavaScript challenge automatically. A plain requests script from a cloud server does the opposite on every count. The fix is not one trick - it is restoring each of those three properties.
Is a residential proxy alone enough to get past Cloudflare?
It depends which layer the site leans on. A residential IP clears the reputation check, which is often enough on sites that only score IPs. But if the site also fingerprints TLS or shows a JS challenge, the clean IP alone will not help - you will still be stopped by the layer you did not address. On a hard target you typically need the IP fix and a fingerprint or real-browser fix together.
What is the difference between the TLS check and the JS challenge?
The TLS check happens silently during the handshake - Cloudflare reads your JA3/JA4 fingerprint before any page loads and can block on it with no visible challenge. The JS challenge is the visible 'Checking your browser' / Turnstile page: Cloudflare serves JavaScript that a real browser executes to prove it is a browser. A fingerprint fix (curl_cffi) beats the first; only a real browser that runs the JS beats the second.
Is scraping a Cloudflare-protected site legal?
Scraping publicly available data is broadly permissible in many jurisdictions, but it is not a blanket right. Stay on the safe side: only collect public data, honor the site's robots.txt and terms where they apply, never scrape data behind a login you are not authorized to access, and rate-limit so you do not degrade the service for real users. The techniques here are for legitimate data collection, not for overwhelming a site.
The IP layer is the one you cannot fake from code, and it is the one Cloudflare checks first. Roam residential IPs - rotating at $2/GB, static at $4/IP per month, over HTTP and SOCKS5 - give Cloudflare a real home-broadband ASN to score instead of a datacenter range. Create an account and get 300MB of free trial traffic to test against your target.