community-automations/web-scraper

研究與情報

PublicClaude 子代理

網頁爬蟲代理程式

現在的網站都用 JavaScript 渲染,光靠一般的 HTTP 請求只會拿到一個空殼。這支爬蟲會驅動真正的瀏覽器,抓下實際載入的資料。

sonnet1 週Browser-UsePlaywrightPostgresAmazon S3
ClaudeClaude
ROI for
README.md

為什麼選擇此子代理

現在的網站都用 JavaScript 渲染,光靠一般的 HTTP 請求只會拿到一個空殼。這支爬蟲會驅動真正的瀏覽器,抓下實際載入的資料。

它會等待單頁式應用完成渲染、將欄位擷取為帶型別的 JSON,並在儲存前驗證每一筆紀錄。乾淨的資料列進到 Postgres 供查詢,原始 JSON 則備份到 S3,這樣版面一旦變動就能被察覺,而不是悄悄弄髒你的資料集。

運作方式

    • Read

      Used at step 01 to kick off the pipeline.

    • Write

      Used at step 01 to kick off the pipeline.

    • WebFetch

      Used at step 01 to kick off the pipeline.

    • WebSearch

      Used at step 01 to kick off the pipeline.

輸出範例

json
// Sample output
// (generated when the pipeline finishes)

Render a single-page app, extract fields into validated JSON, and write to S3 and Postgres, flagging malformed records.

Unlock the rest

The full agent definition, install snippet, and starter task are gated for community members.

Members get the full `.md` agent file, the npm / pnpm install one-liners, a starter prompt that we've tuned against real runs, and the open-source repo when this automation ships there. One email, magic link, done.