Starts a site crawl (same host, depth and page limits from the job), discovers internal links, and records HTTP status per URL. This is a lightweight crawler—not a full Screaming Frog replacement: advanced SEO checks (JS rendering, duplicate packs, custom extractions, sitemap builder) would be follow-on work on top of this pipeline.
No job yet
Start crawl
Status
No status loaded.
Phase 2 — Compare crawls
Baseline a is usually the earlier crawl; b the later. URLs are matched by crawl URL hash. Rows: new_in_b, removed_in_a, changed (status, title, canonical, meta, word count, H1, content-type, robots meta, meta refresh, body hash, X-Robots-Tag, HTML lang, response time). With two jobs selected, this page updates the URL with compareA / compareB so you can bookmark or share the pair. With the diff table visible, filters and sort are also written to the URL (for example csk=status_delta to sort by HTTP status change between A and B).
No crawl jobs returned. Start a crawl above, or check the database connection.
Past crawl jobs
Remove individual jobs from the database (queue, audits, fetches for that job). Use Filter jobs in Phase 2 to narrow this list and the compare dropdowns. View loads that job into Status, reports, and Discovered URLs above.
No jobs loaded. Use Reload list or start a crawl.
Phase 1 Reports
126 downloads (CSV exports + sitemap).
One ZIP: csv/ (125 files), xml/sitemap.xml, and json/summary.json. May take a minute on large jobs.