Using a proxy with Scrapy

By Mechelle Henderson · Published: 7 September 2026 · 6 min read

The one-line way: request.meta

Scrapy already ships HttpProxyMiddleware enabled by default. It reads a proxy from each request's meta and, when the URL carries credentials, extracts them into a Proxy-Authorization header automatically. So the smallest working example needs no settings change at all:

import scrapy

class IpSpider(scrapy.Spider):
    name = "ip"

    def start_requests(self):
        yield scrapy.Request(
            "https://ip.sb",
            meta={"proxy": "http://YOUR_USERNAME:[email protected]:41080"},
            callback=self.parse,
        )

    def parse(self, response):
        self.logger.info("Exit IP: %s", response.text.strip())

Run it and the logged exit IP should be a Roam residential address, not your server's. This form is fine for a quick test or a spider that only ever needs one proxy.

The reusable way: a downloader middleware

Hard-coding credentials in every Request gets messy. Move them into a downloader middleware so every request — including retries and redirects Scrapy generates for you — goes through the proxy, and your spiders stay clean:

# myproject/middlewares.py
class RoamProxyMiddleware:
    GATEWAY = "gw.roamproxy.com:41080"
    USER = "YOUR_USERNAME"
    PASSWORD = "YOUR_PASSWORD"

    def process_request(self, request, spider):
        request.meta["proxy"] = f"http://{self.USER}:{self.PASSWORD}@{self.GATEWAY}"

# myproject/settings.py
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RoamProxyMiddleware": 610,
}

Put the credentials in the proxy URL and let Scrapy handle the header. On modern Scrapy, HttpProxyMiddleware (order 750) reads meta["proxy"], extracts the user:pass into a Proxy-Authorization header, and strips the credentials back out of the stored URL. Don't set that header yourself — since a 2022 security fix, Scrapy rewrites Proxy-Authorization to match the proxy URL on every request, so a hand-set header gets dropped on redirects and retries.

Rotating IPs per request

With a rotating residential proxy you don't maintain a list of proxy IPs — you point every request at one gateway and change the sticky-session token embedded in the username. Each distinct session maps to a different residential exit IP, so a fresh token per request gives you a fresh IP per request from a single credential:

import itertools

class RotatingRoamProxyMiddleware:
    GATEWAY = "gw.roamproxy.com:41080"
    USER = "YOUR_USERNAME"
    PASSWORD = "YOUR_PASSWORD"
    _counter = itertools.count()

    def process_request(self, request, spider):
        session = next(self._counter)
        user = f"{self.USER}_session_{session}"
        request.meta["proxy"] = f"http://{user}:{self.PASSWORD}@{self.GATEWAY}"

Reuse the same session token across several requests when you need them to share an IP (a login flow, a paginated result set); generate a new one when you want a fresh IP. See what are rotating proxies for how sticky sessions work.

When the IP isn't the problem

A clean residential IP fixes reputation-based blocks. It does not fix a TLS fingerprint that screams "Python," or a JavaScript challenge that a plain HTTP client can't run. If a residential proxy still gets you blocked, the target is fingerprinting your handshake or requiring a browser — see bypassing TLS/JA3 fingerprinting with curl_cffi and how to get past Cloudflare. When the page genuinely needs a browser, drive Playwright or Puppeteer through the same proxy instead.

FAQ

How do I set a proxy for a single Scrapy request?

Put it in the request's meta: yield scrapy.Request(url, meta={"proxy": "http://user:[email protected]:41080"}). Scrapy's built-in HttpProxyMiddleware is enabled by default, reads meta["proxy"], and moves the user:pass into a Proxy-Authorization header for you. No settings change is needed for the one-off case.

How do I use one proxy for the whole crawl?

Write a small downloader middleware whose process_request sets request.meta["proxy"] on every request, and enable it in DOWNLOADER_MIDDLEWARES. That keeps credentials out of your spider code and applies the proxy to every request the crawl makes, including retries and redirects.

How do I rotate IPs per request in Scrapy?

With a rotating residential proxy you don't need a list of proxies. Point every request at the same gateway and vary the sticky-session token in the username per request — each distinct session gets a different residential exit IP. A middleware that builds a fresh username_session_N on each process_request gives you a new IP per request from one credential.

Why do I get 407 Proxy Authentication Required in Scrapy?

A 407 means the proxy did not receive valid credentials. The usual cause is a bare meta["proxy"] of http://host:port with no user:pass. Put the credentials in the proxy URL — http://user:pass@host:port — and let HttpProxyMiddleware turn them into the Proxy-Authorization header. Don't hand-set that header: modern Scrapy rewrites it to match the proxy URL, so a manual header is dropped on redirects and retries.

Scrapy expects a plain host:port proxy plus credentials — exactly what Roam gives you. Rotating residential at $2/GB and static residential at $4/IP per month, over HTTP and SOCKS5, with sticky sessions for per-request rotation. Create an account and get 300MB of free trial traffic to point your first spider at a residential exit.