Web Scraping with Residential Proxies, Responsibly

The first job is geo accuracy. Prices, search results, stock availability, and ad placements differ by country and often by city. A pool of 16M+ residential IPs across 195+ countries lets you request the regional version of a page directly: append -cc-US or -ct-Dallas to your proxy username and the request exits from that location. Targeting lives entirely in the username — no separate configuration is needed.

The second job is load distribution. In rotating mode, every request through <proxy-host>:80 (HTTP, HTTPS CONNECT, or SOCKS5) exits from a different IP. Spread across thousands of addresses, a crawl puts a light per-IP footprint on the target instead of a sustained stream from one machine — which matters for the target's health as much as for your crawl's reliability.

Typical compliant workloads — price monitoring, SERP tracking, ad verification — are covered in more depth in market research and SEO monitoring.

Write these into your project plan before you write the crawler.

• Only publicly accessible data. If a page sits behind an account, a paywall, or any authorization you do not hold, it is out of scope. Full stop.
• Respect `robots.txt` and the target's terms of service. Under the platform's AUP, violating a target site's ToS or rate-limit policy is a violation of the platform's terms as well.
• Rate-limit yourself. No dataset justifies degrading someone else's service. Set a conservative requests-per-second budget per target and enforce it in code.
• Personal data changes everything. Names, emails, profiles, and photos trigger GDPR-class obligations: lawful basis, minimization, retention limits, deletion rights. "Publicly visible" does not mean "free to process."
• When in doubt, get legal advice. Scraping law varies by jurisdiction and keeps moving. This article is not legal counsel.

Use rotation for breadth. Rotating is the default mode — a fresh IP per request — and it is the right choice for large lists of independent URLs. See rotating vs. static proxies for when each applies.

Use sticky sessions for depth. When a flow spans multiple pages that must come from the same IP — pagination that depends on server-side state, multi-step forms you are authorized to use — pin the IP with a session ID and a TTL. The TTL defaults to 60 seconds and can be raised to 86400 (24 hours). Sticky sessions covers the parameters in full.

user123-cc-US              # rotating, US exit per request
user123-cc-US-s-crawl01-ttl-600   # same US IP for 10 minutes

• Retry with backoff. On failure, wait, then wait longer — exponential backoff with jitter, capped at a few attempts. A 429 or 403 is a signal to slow down, never to retry harder.
• Cache aggressively. Never fetch the same URL twice inside its freshness window. Store raw responses so parser changes do not force a re-crawl.
• Deduplicate before you fetch. Normalize URLs and drop duplicates in the queue, not after the download.

You pay the published rate per GB, so wasted transfer is a direct line item. The biggest single saving is blocking media in headless browsers: images, video, fonts, and third-party analytics scripts routinely make a rendered page many times the size of its HTML. Estimating real per-page cost is covered in how much bandwidth do you need.

• Prefer plain HTTP requests over a headless browser whenever the data is already in the HTML. A browser is the expensive fallback, not the default.
• Send Accept-Encoding: gzip, br and make sure your client actually decompresses — compressed HTML is a fraction of the raw size.
• Fetch only what you parse. If you need one field, do not download an API response with fifty.

A responsible crawler behaves like a considerate visitor, not an anonymous flood. The habits are simple and cheap.

• Crawl off-peak for the target's timezone. Overnight traffic competes with fewer real users.
• Honor `Retry-After`. When a 429 or 503 includes the header, wait exactly that long — it is the site telling you its limit.
• Identify yourself where appropriate. For research or archival crawls, a descriptive User-Agent with a contact address lets an operator reach you instead of blocking you.
• Watch your error rates. A rising share of 5xx responses means the target is struggling. Slow down or pause; the data will still be there tomorrow.

These map directly to prohibited conduct in the AUP. None of them are gray areas.

• No credential stuffing or logging into accounts you do not own. That is unauthorized access, whatever the tooling.
• No paywall or authentication circumvention. If content requires payment or login, obtain it the way the site intends or not at all.
• No DoS-scale request rates. A crawl that degrades a site's service is an attack regardless of intent, and it is treated as one.
• No working around a target's security controls or rate limits. If a site is telling you no, the answer is no.
• No harvesting personal data for spam, harassment, or resale.

Proxies distribute your traffic; they do not launder it. Responsibility for what your crawler does — legally and contractually — stays with you. If you are unsure whether a planned workload is acceptable, ask via the support tickets page in your dashboard before you run it.