community-automations/autonomous-crawler

Research og indsigt

PublicClaude-subagent

Autonom crawler

At crawle et helt site i hånden betyder manglende sider, en overbelastet server eller at miste overblikket over, hvad du allerede har hentet. Denne crawler styrer hele gennemkørslen.

sonnet4 dagePlaywrightPythonRedisPostgres
ClaudeClaude
ROI for
README.md

Hvorfor denne subagent

At crawle et helt site i hånden betyder manglende sider, en overbelastet server eller at miste overblikket over, hvad du allerede har hentet. Denne crawler styrer hele gennemkørslen.

Ud fra en seed-URL finder den links, holder sig inden for domænet og spreder forespørgslerne, så sitet forbliver tilfreds. Fejl forsøges igen med backoff, besøgte URL'er spores for at undgå loops, og hver side gemmes med sine metadata. Når køen er tom, får du en dækningsrapport.

Sådan kører den

    • Read

      Used at step 01 to kick off the pipeline.

    • Write

      Used at step 01 to kick off the pipeline.

    • WebFetch

      Used at step 01 to kick off the pipeline.

    • WebSearch

      Used at step 01 to kick off the pipeline.

Eksempel på output

json
// Sample output
// (generated when the pipeline finishes)

Crawl a single domain from a seed URL with rate limiting, retries, and dedup, storing each page and reporting coverage.

Unlock the rest

The full agent definition, install snippet, and starter task are gated for community members.

Members get the full `.md` agent file, the npm / pnpm install one-liners, a starter prompt that we've tuned against real runs, and the open-source repo when this automation ships there. One email, magic link, done.