When every site is different, the parser stops being code and becomes an agent
We got a request to build a scraper. Not a trivial one: process 5,000 URLs, confirm whether each site has a careers section, and figure out what's actually inside it.
If there are open positions — which ones? What categories do they fall into? Which cities? And how does the application process work: an internal form, or a link to an external platform like HeadHunter, Avito, or SuperJob?
The hard part wasn't fetching pages. It was the analysis. Every site is unique, and writing one algorithm that correctly extracts job data from all of them is a losing battle.
The moment your parser needs a per-site exception, you're not writing a parser anymore. You're writing 5,000 of them.
So we decided to hand the entire analysis step to AI and see how far it would go.
To find out which model fit, we connected to OpenRouter. It lets you cycle through a large number of models quickly and compare them on three axes:
After a short round of experiments, we landed on Haiku from Anthropic. It was accurate enough for the task and fast enough to run at volume.
Here's the catch. If you pay per request — through OpenRouter or directly through the Anthropic API — checking a single site costs roughly $0.25–0.40. Multiply that by 5,000 and you're looking at $1,250–2,000 for one pass over the list. Well outside a reasonable budget for this project.
The fix was to stop paying per request. We moved to a subscription plan and worked through it directly. Our plan runs $200/month, and in practice it gives us around 1,000 sites checked in about 4 hours, and roughly 15,000 sites over a week.
Same model, same work, completely different economics. On volume projects, the billing model matters as much as the model itself.
That fit both our budget and the project timeline.
All we had from the client was an incoming XLS file with the URLs. Everything else we built around it.
Postgres for storage, Django for control. We stood up Postgres and put Django on top of it — specifically for the admin panel. Django's admin takes about five minutes to deploy and gives you a working interface for managing thousands of records without writing a line of frontend code.
Import in one prompt. Getting the spreadsheet into the database was a single prompt: read the Excel file, map the columns, write the rows. Five minutes of work.
From there, the admin panel became the control room. Each site is a row. Each row has a status.
To start processing, you select sites with a checkbox — one at a time or in bulk — and move them to pending.
As soon as a site lands in pending, a separate Python script picks up the event and launches 10 independent Claude Code instances. When one finishes, another starts immediately. They keep cycling until every site in pending has moved to either completed or error.
That's the whole orchestration layer: a status column and a worker loop. No queue broker, no scheduler. The database is the queue.
The interesting part is inside Claude Code. We wrote the instructions in layers.
CLAUDE.md holds the general project context: how to work with this project, where things live, what the conventions are.
Then two skills:
First, a curl request checks whether the site responds. Anything other than a 200 and the site moves to error status.
There's a wrinkle here. Every site gets analyzed through Chrome in headless mode, and some sites detect headless browsers and block them. Those get flagged as errors incorrectly. We handle it by re-running the failed batch later in normal (non-headless) mode.
Some sites throw a captcha. Some aren't available in our region at all. We measured it — roughly 1% of the list. For this project, that's noise we can safely ignore.
Don't build handling for the 1% before you've shipped the 99%. Measure the failure rate first, then decide if it's worth code.
Once the site is confirmed reachable, Claude looks for the careers page. It starts cheap: pattern-matching common URLs — /jobs, /career, /vacancies, and similar variants.
If that finds nothing, it falls back to the expensive path: manually walking the homepage and several internal pages, reading the content, and identifying the careers page by secondary signals — not the URL or the link label, but the actual text on the page.
Once the page is found, Claude needs to manipulate it — issue requests, simulate clicks, scroll, expand lists. For that we connected the Chrome DevTools MCP.
The first thing Claude does on a careers page is check whether the job data is available through an API. Often it is, and that's by far the cleanest path.
Take a large retail site as an example. The vacancy list renders from an API call, and which vacancies you get depends on the selected city. Claude watches the network traffic, identifies which request returns the city list and which returns the vacancies, pulls the city ID out of the first response, injects it into the second, and gets a clean per-city list of openings.
No DOM parsing, no brittle selectors — just the site's own API, reverse-engineered on the fly.
When there's no API, Claude falls back to reading the DOM directly. It looks at what's on the page and decides for itself what constitutes a vacancy. If listings are paginated, it pages through them. If there's a "Load more" button, it clicks it until everything is loaded. It imitates what a person would do.
When the analysis is done, Claude sends the results back through the Django REST API. They're written to the database through the ORM, and the site's status flips to completed.
Everything shows up in the admin panel immediately, and we can work with the data from there.
A simple site takes about 30 seconds. A complex one runs 2–3 minutes, maybe 5 at the outside.
With 10 instances running in parallel, the whole list moves quickly enough that the wall-clock time stopped being a concern.
Yes — it works, and it can be very effective. But it depends entirely on the shape of your project.
5,000 URLs with 5,000 different page structures? This approach is completely justified. The alternative is writing and maintaining hundreds of site-specific extractors.
50,000 or 100,000 URLs? I'd look at a hybrid approach: handle the bulk with a conventional Python parser and route only the difficult cases to AI. At that scale the per-site cost of AI analysis starts to dominate, and most sites don't need it.
One site with a known structure? Don't bring AI into it at all. If you're pulling product data from a single source and you already know the layout and the selectors, a normal bot does it faster, cheaper, and more predictably.
AI is for the hard cases. If you can describe the structure in advance, you don't need a model to find it for you.
The real question isn't "can AI scrape websites." It's whether the variability in your data sources is high enough to justify paying a model to think about each one.
For us, on this project, it was.
Liked this? Let us know!
Let's talk about your idea. We'll tell you what's possible and how fast.
Get a free consultation