Most anti-scraping advice was written for a world where a scraper had one IP address and announced itself in the User-Agent string. That world is gone. A residential proxy pool costs a few dollars per gigabyte, and each request arrives from a different home connection with a plausible browser signature attached.
What follows is what still works once the attacker can buy their way past the address.
Scraping is three problems wearing one word
Decide which one you have before you build anything, because the right response differs for each.
- Competitive scraping. Someone lifts your prices, listings or inventory on a schedule. Low volume, high patience, and it comes back tomorrow whatever you do today.
- Bulk extraction. Someone wants the whole dataset once. High volume, short window. Stopping it is mostly a matter of making the crawl slower than their deadline.
- Authenticated abuse. The scraper is logged in, on a real account, pulling data that account is entitled to see at a rate no human produces.
The third is the expensive one and the only one where network detection is the wrong first instrument. If they are authenticated, the account is the identifier, and the question is velocity per account rather than anything about the address. Handle it there and stop reading about proxies.
User-Agent blocking is theatre
Blocking on User-Agent stops exactly the scrapers that were not trying. A serious one sends a current Chrome string, because setting a header is one line in every HTTP library ever written. The same goes for robots.txt: it is a request, not a control, and it is honoured by the crawlers you probably wanted anyway.
Worse, both mislead you. A quiet dashboard after a User-Agent rule looks like a fix. It usually means the operator changed one string and your logs stopped labelling them.
Per-IP rate limiting fails by design
Rate limiting assumes the address is scarce. Against a residential proxy pool it is not: the same crawl arrives across thousands of home connections, and each individual address makes three requests an hour. Every one of them sits comfortably under any limit you can set without breaking real users on shared carrier NAT.
Keep rate limits. They stop the lazy and they cap your blast radius. Just stop treating them as scraper detection, because a pool defeats them arithmetically and no threshold fixes that.
What actually separates a scraper from a customer
Three layers, in the order they cost the attacker money to defeat.
Infrastructure. The cheapest crawls run in datacenters, and that is still most of them. A hosting range, a commercial VPN exit or a Tor exit is not proof of scraping, but it is a strong prior on a page that no ordinary customer reaches through a tunnel. This is one API call and it removes most of the volume before you build anything clever.
curl -X POST https://maskbreak.com/api/lookup \
-H 'Content-Type: application/json' \
-d '{"ip":"185.220.101.1"}'{
"ip": "185.220.101.1",
"known": true,
"verdict": "block",
"risk_score": 90,
"signals": { "vpn": false, "proxied": false, "tor": true, "dch": false, "anon": true },
"network": { "asn": null, "org": "Tor exit node", "country": null, "city": null }
}Device. When the crawl moves to residential addresses, the network layer goes quiet and the browser has to answer instead. Headless runtimes, automation drivers and patched fingerprints leave marks that cost real engineering to remove, and removing them has to be redone every time the browser ships. That is a maintenance bill the operator pays forever, which is the point.
Behaviour. Scrapers traverse. They walk a category tree in order, request pages no human sequence produces, and never load the image assets that a rendering browser fetches without being asked. Sort your access logs by session and look at the shape of the path rather than the volume of it.
The check
Put the network check in front of the routes that hold the data, not in front of the whole site. The point is to spend the call where the value is.
async function scrapeGuard(req, res, next) {
try {
const r = await fetch('https://maskbreak.com/api/lookup', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ ip: req.ip }),
signal: AbortSignal.timeout(1500)
});
const risk = await r.json();
if (risk.verdict === 'block') {
return res.status(429).set('Retry-After', '3600').json({ error: 'Rate limited.' });
}
if (risk.verdict === 'review') {
req.serveDegraded = true; // fewer results per page, no bulk fields
}
next();
} catch (err) {
next(); // fail open, always
}
}
app.get('/api/listings', scrapeGuard, listingsHandler);Note the failure path. A scraping defence that can take down your product when a third party has a bad afternoon has cost you more than the scraping did. Fail open, set a timeout, and let the check be the thing that breaks.
Respond by cost, not by block
A hard 403 is a free, instant signal that you detected them. The operator changes one thing and tries again, and your only feedback is that the traffic stopped looking familiar. That is a bad trade for you and a cheap iteration loop for them.
Degrade instead. Serve fewer results per page. Drop the fields that make bulk extraction worth doing. Add latency. A crawl that takes eleven days instead of six hours fails the deadline it was built for, and nothing about the response tells the operator which signal fired.
Reserve outright blocking for the cases where the answer is unambiguous and the page is expensive to serve.
Decide about AI crawlers separately
Assistant traffic is not scraping in the sense above, and lumping the two together produces bad policy in both directions. Some of it drives referrals you want. Some of it trains a model on your catalogue and sends nothing back. That is a commercial decision about your own content, and it should be made on purpose rather than inherited from a bot rule written for sneaker resellers.
What this does not solve
Nothing here stops a determined operator on clean residential addresses driving a patched browser at human pace. That attacker exists, and against them the honest answer is that you raise their cost until the data is worth less than the crawl, then you stop. Detection is an economics exercise, not a wall.
What it does solve is the eighty per cent that runs in a datacenter because nobody made it expensive to.
Frequently Asked Questions
Does blocking datacenter IP ranges stop scraping?
It stops the cheap majority and none of the serious minority. Datacenter blocking is worth doing because most crawls never leave a hosting provider, but treat a hosting range as a strong prior rather than proof: your own monitoring, corporate egress and plenty of legitimate integrations arrive from the same ranges. Score it and route on the score.
Why does rate limiting not catch residential proxy scraping?
Because rate limiting assumes the address is scarce, and a residential pool makes it abundant. The same crawl spreads across thousands of home connections at a few requests each per hour, which sits under any threshold you can set without breaking users on shared carrier NAT. Keep the limits for blast radius, not for detection.
Should I block scrapers with a 403?
Usually not. A 403 tells the operator instantly and precisely that something fired, which turns your defence into a fast feedback loop for them. Degrading the response is better: fewer results, missing bulk fields, added latency. The crawl misses its deadline and the operator learns nothing about which signal caught it.
Is scraping illegal, so can I just send a takedown?
It varies by jurisdiction and by what is scraped, and legal action is slow next to a crawl that finishes this week. Send the notice if the operator is identifiable and the data matters, but do not let it replace the technical work. Most scrapers are not identifiable, which is rather the point of the proxy pool.
Check an address before you serve the data
One call returns VPN, proxy, Tor and datacenter signals with a verdict you can route on. Free at 1,000 requests an hour, no card.
Get a free API key