community-automations/autonomous-crawler

Research & Intel

PublicClaude subagent

Autonomous Crawler

Crawling a whole site by hand means missing pages, hammering the server, or losing track of what you already fetched. This crawler manages the whole sweep.

sonnet4 daysPlaywrightPythonRedisPostgres
ClaudeClaude
ROI for
README.md

Why this subagent

Crawling a whole site by hand means missing pages, hammering the server, or losing track of what you already fetched. This crawler manages the whole sweep.

From a seed URL it discovers links, stays inside the domain, and paces requests so the site stays happy. Failures retry with backoff, visited URLs are tracked to avoid loops, and every page is stored with its metadata. When the queue empties, you get a coverage report.

How it runs

    • Read

      Used at step 01 to kick off the pipeline.

    • Write

      Used at step 01 to kick off the pipeline.

    • WebFetch

      Used at step 01 to kick off the pipeline.

    • WebSearch

      Used at step 01 to kick off the pipeline.

Sample output

json
// Sample output
// (generated when the pipeline finishes)

Crawl a single domain from a seed URL with rate limiting, retries, and dedup, storing each page and reporting coverage.

Unlock the rest

The full agent definition, install snippet, and starter task are gated for community members.

Members get the full `.md` agent file, the npm / pnpm install one-liners, a starter prompt that we've tuned against real runs, and the open-source repo when this automation ships there. One email, magic link, done.