Getting blocked isn't bad luck — it's a signal. Anti-bot systems score every request on IP reputation, request rate, headers and behavior. Fix the signals that give you away and block rates fall dramatically. Here's the practical checklist, roughly in order of impact.
The single biggest signal is your IP's ASN. Datacenter ranges are on every anti-bot blocklist; residential IPs from real ISPs carry ordinary-user reputation. If a site blocks you instantly regardless of how carefully you throttle, your IP type is the problem — switch to residential.
Thousands of requests from one IP is the clearest bot tell. Rotate to a fresh IP per request for wide crawls, or hold a sticky IP for flows that need continuity (login, cart, pagination). With Roam you control both from the username: a random -session-<id> rotates, a fixed one sticks.
Humans don't fire 20 requests a second on a metronome. Add delays, randomize them, and cap concurrency per target. Respect robots.txt and any published rate limits — both to stay polite and to stay unblocked.
A default library User-Agent screams 'bot'. Send a real browser User-Agent, a consistent set of Accept/Accept-Language headers, and keep them consistent with the IP's country. Don't mismatch a Japanese IP with an English-only header set.
Some targets need a real browser (Playwright, Puppeteer) to render JS and pass fingerprint checks. Pair a headless or antidetect browser with residential proxies, and give each browser profile its own sticky IP so the IP and fingerprint stay consistent.
Timing is only one signal. If your IP is a datacenter range, no amount of throttling helps — the block happens at the network layer before behavior matters. Switch to residential IPs first, then tune timing.
With per-GB residential proxies you don't count proxies — you rotate through a large pool automatically. Focus on request rate per target instead. A rotating residential endpoint gives you a fresh IP whenever you change the session token.
Using proxies is legal. Legality depends on what you scrape and how: stick to publicly available data, follow each site's terms, avoid personal data, and keep request rates reasonable.
Sign up in seconds, test your targets on real residential IPs, and only pay for what you use. $2/GB, forever-valid balance.