community-automations/autonomous-crawler

研究與情報

PublicClaude 子代理

自動爬蟲

手動爬整個網站,不是漏掉頁面、就是把伺服器打爆,再不然就是搞不清哪些已經抓過了。這支爬蟲能統籌整個掃描過程。

sonnet4 天PlaywrightPythonRedisPostgres
ClaudeClaude
ROI for
README.md

為什麼選擇此子代理

手動爬整個網站,不是漏掉頁面、就是把伺服器打爆,再不然就是搞不清哪些已經抓過了。這支爬蟲能統籌整個掃描過程。

它從一個種子網址出發探索連結、守在網域範圍內,並控制請求節奏,避免對網站造成負擔。失敗時以退避機制重試,記錄造訪過的網址以避免重複迴圈,每一頁都連同其詮釋資料一起儲存。當佇列清空,你會收到一份涵蓋率報告。

運作方式

    • Read

      Used at step 01 to kick off the pipeline.

    • Write

      Used at step 01 to kick off the pipeline.

    • WebFetch

      Used at step 01 to kick off the pipeline.

    • WebSearch

      Used at step 01 to kick off the pipeline.

輸出範例

json
// Sample output
// (generated when the pipeline finishes)

Crawl a single domain from a seed URL with rate limiting, retries, and dedup, storing each page and reporting coverage.

Unlock the rest

The full agent definition, install snippet, and starter task are gated for community members.

Members get the full `.md` agent file, the npm / pnpm install one-liners, a starter prompt that we've tuned against real runs, and the open-source repo when this automation ships there. One email, magic link, done.