How to get past Cloudflare when scraping public data
Published: 23 July 2026 · 7 min read
TL;DR: Cloudflare does not run one bot check, it runs several - IP/ASN reputation, TLS and HTTP/2 fingerprinting, and interactive JS challenges (Managed Challenge / Turnstile). You get past it by matching each layer: a residential proxy for the IP, a browser-accurate TLS fingerprint (curl_cffi, or a real browser), and a real browser that runs JavaScript for the challenge pages. Which combination you need depends on which layer is actually stopping you. Scrape public data only, and rate-limit politely.
Cloudflare is not one wall, it's three
People talk about "bypassing Cloudflare" as if it were a single lock. It is not. Cloudflare's bot management runs several independent checks, and a request has to satisfy all of them. When you are blocked, the useful question is not "how do I bypass Cloudflare" but "which of its checks am I failing?" - because each one has a different fix.
| Layer | What it checks | How you fail it | The fix |
|---|---|---|---|
| IP reputation | The exit IP's ASN | Requesting from a datacenter range | Residential proxy |
| TLS / HTTP2 fingerprint | Your ClientHello (JA3/JA4) | A Python/library fingerprint | curl_cffi or a real browser |
| JS challenge | Whether JavaScript runs | No JS engine (raw HTTP) | A real browser that executes JS |
Layer 1: IP reputation - the one you can't fake from code
Before Cloudflare looks at anything your client sends, it scores the IP you are coming from. Datacenter ASNs (AWS, Hetzner, OVH) carry a standing penalty because real visitors almost never browse from them. No header, no fingerprint trick, and no browser can change the ASN of the IP you exit from - only routing through a different network can.
A residential proxy gives Cloudflare a real home-broadband IP to score. On sites that rely mostly on IP reputation, this single change is often the whole fix. On harder sites it is necessary but not sufficient - you still have layers 2 and 3 to clear.
Layer 2: TLS fingerprint - the silent block
During the TLS handshake, Cloudflare hashes your ClientHello into a JA3/JA4 fingerprint. Python's requests and httpx produce a fingerprint no real browser emits, so Cloudflare can block you silently, before a single byte of the page is sent - no visible challenge, just a 403 or a stall.
If you are scripting with HTTP requests rather than a browser, restore a browser-accurate fingerprint with curl_cffi. Combined with a residential proxy, that clears layers 1 and 2 in one script:
from curl_cffi import requests
proxies = {"https": "http://USER:PASS@gw.roamproxy.com:41080"}
r = requests.get("https://target.example", impersonate="chrome124", proxies=proxies)
print(r.status_code)Layer 3: the JS challenge - when you need a real browser
Some Cloudflare configurations serve an interactive challenge - the "Checking your browser…" interstitial or a Turnstile widget. These work by handing the client JavaScript that a genuine browser executes to prove it is one. A fingerprint fix does not help here, because there is no JavaScript engine in an HTTP library to run the challenge.
The reliable answer is to use a real browser that runs the JS - headless Chromium via Playwright, or an LLM-driven agent on top of it. Point that browser through the same residential proxy and it clears all three layers at once. See using residential proxies with browser-use for the browser-plus-proxy setup; the same proxy dict works for plain Playwright.
Match the tool to the layer
- Blocked by IP only (works locally, fails from a server) - a residential proxy is usually the whole fix.
- Silent 403 with correct IP and headers - a TLS fingerprint block; add curl_cffi.
- Visible "Checking your browser" / Turnstile - a JS challenge; use a real browser through the proxy.
Do not reach for a full headless browser when a residential IP plus curl_cffi would do - the browser is slower and heavier. Escalate only as far as the target's actual defense requires. For the broader checklist of why scrapers get blocked, see how to avoid getting blocked while web scraping.
Collect public data responsibly. These techniques are for scraping publicly accessible pages at a reasonable rate. Honor robots.txt and a site's terms where they apply, do not access data behind an authorization you do not have, and rate-limit so you are never degrading the site for its real users.
FAQ
Why does the same request work in my browser but not in my script?
Because your browser passes all three of Cloudflare's checks and your script passes none of them. Your browser exits from your home residential IP, sends a real Chrome TLS fingerprint, and runs the JavaScript challenge automatically. A plain requests script from a cloud server does the opposite on every count. The fix is not one trick - it is restoring each of those three properties.
Is a residential proxy alone enough to get past Cloudflare?
It depends which layer the site leans on. A residential IP clears the reputation check, which is often enough on sites that only score IPs. But if the site also fingerprints TLS or shows a JS challenge, the clean IP alone will not help - you will still be stopped by the layer you did not address. On a hard target you typically need the IP fix and a fingerprint or real-browser fix together.
What is the difference between the TLS check and the JS challenge?
The TLS check happens silently during the handshake - Cloudflare reads your JA3/JA4 fingerprint before any page loads and can block on it with no visible challenge. The JS challenge is the visible 'Checking your browser' / Turnstile page: Cloudflare serves JavaScript that a real browser executes to prove it is a browser. A fingerprint fix (curl_cffi) beats the first; only a real browser that runs the JS beats the second.
Is scraping a Cloudflare-protected site legal?
Scraping publicly available data is broadly permissible in many jurisdictions, but it is not a blanket right. Stay on the safe side: only collect public data, honor the site's robots.txt and terms where they apply, never scrape data behind a login you are not authorized to access, and rate-limit so you do not degrade the service for real users. The techniques here are for legitimate data collection, not for overwhelming a site.
The IP layer is the one you cannot fake from code, and it is the one Cloudflare checks first. Roam residential IPs - rotating at $2/GB, static at $4/IP per month, over HTTP and SOCKS5 - give Cloudflare a real home-broadband ASN to score instead of a datacenter range. Create an account and get 300MB of free trial traffic to test against your target.