All projects
Project

CanSpider — LLM-Planned Crawling Platform

One generic spider replaces N hand-written ones: an LLM reads a page's DOM digest and emits a crawl plan, which is statically linted, dry-run against the live site, and auto-repaired before anything runs. 64,572 researcher records across 143 institutions in 7 countries, at roughly CAD 0.03 per record.

crawlingllmproductioninfrastructure

Problem

Faculty and funding data lives behind 143 different site structures, none of them stable. One person was collecting five or six university departments a day by hand, and every hand-written spider broke the moment a site changed.

Solution

gpt-5.1 derives a per-site crawl plan from a DOM digest. The plan is then statically linted (selector-type homogeneity, cssselect translatability, regex host scope), dry-run against the live page across sample profiles with per-field coverage reporting, and auto-repaired over up to two rounds before a single real crawl starts. A single generic Scrapy + Playwright spider with five pagination strategies executes it into Kafka, where a 50-worker async consumer runs structured LLM digestion into a 40-field record, followed by RapidFuzz entity resolution and idempotent upsert.

Impact

209,753 pages crawled, 209,162 classifications, 156,748 digests, 3,119 crawl jobs at 97.3% completion, peaking at 12,879 digests in a single day. 32,000 profiles in the first 40 days; 64,572 today across 7 countries. Cost held near CAD 0.03 per record by a prompt-hash response cache and a token-bucket limiter at 75% TPM.

Stack

PythonScrapyPlaywrightKafkaPostgreSQLQdrantTypeScriptNext.js

CanSpider — LLM-Planned Crawling Platform

The bottleneck

Faculty and research-funding data lives behind 143 different site structures, none of them stable, none of them documented. The existing process was one person collecting five or six university departments a day by hand. The obvious fix — write a spider per site — fails on contact with reality: every hand-written spider breaks the first time a department redesigns its people page, and you end up maintaining N brittle scrapers instead of doing the work.

The idea, and the part that matters

The idea is not "use an LLM to scrape." Anyone can do that, and the result is a system that silently produces garbage.

The idea is that the model proposes a crawl plan and the system decides whether to trust it. gpt-5.1 reads a compressed DOM digest of a site and emits a structured plan. Before a single real request is made, that plan goes through three gates:

  1. A static linter — checks selector-type homogeneity, that every selector actually translates through cssselect, that no content-specific string literals leaked into what should be a general rule, and that regex host scoping can't wander off-domain.
  2. A dry run against the live page, fetching a handful of sample profiles and reporting per-field coverage. A plan that extracts names but silently misses affiliations fails here, not in production.
  3. An auto-repair loop, up to two rounds, feeding the failures back to the model.

Only a plan that survives all three gets executed. This is the same instinct behind everything else I build: the model is allowed to be creative, and then something deterministic has to be accountable for what actually happened.

Execution

One generic PlanSpider (Scrapy + scrapy-playwright, five pagination strategies) runs every plan — so there is exactly one crawler to maintain, not 143. Output lands on a Kafka topic, where a digestion consumer running 50 async workers behind a 12-way LLM semaphore turns raw pages into a structured ~40-field record via gpt-5-nano, then resolves entities with RapidFuzz and upserts idempotently into a curated table.

Cost was a first-class design constraint, not an afterthought: a prompt-hash response cache in Qdrant (161K digestion + 189K classification cached records) and a token-bucket limiter pinned at 75% of TPM and 95% of RPM held the marginal cost near CAD 0.03 per researcher record.

Results

  • 64,572 curated researcher records across 143 institutions in 7 countries
  • 209,753 pages crawled · 209,162 classifications · 156,748 digests
  • 3,119 crawl jobs at 97.3% completion, peaking at 12,879 digests in one day
  • 167K interest and expertise labels, clustered into 152 subfields under 15 domains
  • ~CAD 0.03 per record — roughly CAD 2,000 for the full corpus

The honest version of the headline: the first 40 days produced about 32,000 profiles across 53 institutions. Reaching 64K took roughly four months as coverage widened.

What I'd fix

There are no automated tests and no CI on this repo, and the operator console has a 5,000-line page component that should have been decomposed three refactors ago. Both are consequences of building it alone under time pressure, and both are the first things I'd address with a second engineer.