site-crawl

site-crawl is the multi-page companion to scrape-render. Single-page scraping (scrape-render, scrape-clean) is fine when you know the URL; site-crawl is what you reach for when you want to ingest *a whole section of a site* for RAG, archival, or competitive research.

**Algorithm:** • BFS from `startUrl`. Same browser, sequential nav (no per-page launch overhead). • Same-origin filtering by default (`allowExternal: true` to follow outbound links). • Strip fragments (`#section`) and skip non-HTML extensions (.pdf, .png, .mp4, .zip, …). • Per-host politeness delay between pages (default 400 ms) — don't hammer the target. • Each page returns `{ url, finalUrl, status, title, html, text, linkCount, depth, ok, error }`. A single page failing doesn't fail the whole call.

**Limits (sync, MVP):** • `maxPages`: default 5, hard cap 15. Bigger crawls need an async variant. • `maxDepth`: default 2, hard cap 5. • HTML truncated at 2 MB per page; text at 80k chars. • 15s navigation timeout per page.

**Use it for:** RAG ingest of a docs site, link audit, content-snapshot for compliance, competitor-site enumeration, broken-link surveys.

**Don't use it for:** entire-domain crawls (we cap at 15 pages — use a real crawler like Crawlee for that), images / file downloads (we strip those from the queue), authenticated pages (no cookie injection yet).