community-automations/web-scraper

Research & Intel

PublicClaude subagent

Web Scraper Agent

Modern sites render with JavaScript, so a plain HTTP request returns an empty shell. This scraper drives a real browser to get the data that actually loads.

sonnet1 weekBrowser-UsePlaywrightPostgresAmazon S3
ClaudeClaude
ROI for
README.md

Why this subagent

Modern sites render with JavaScript, so a plain HTTP request returns an empty shell. This scraper drives a real browser to get the data that actually loads.

It waits for the single-page app to render, extracts fields into typed JSON, and validates each record before it's stored. Clean rows go to Postgres for querying and the raw JSON lands in S3 as a backup, so a layout change is caught instead of silently corrupting your dataset.

How it runs

    • Read

      Used at step 01 to kick off the pipeline.

    • Write

      Used at step 01 to kick off the pipeline.

    • WebFetch

      Used at step 01 to kick off the pipeline.

    • WebSearch

      Used at step 01 to kick off the pipeline.

Sample output

json
// Sample output
// (generated when the pipeline finishes)

Render a single-page app, extract fields into validated JSON, and write to S3 and Postgres, flagging malformed records.

Unlock the rest

The full agent definition, install snippet, and starter task are gated for community members.

Members get the full `.md` agent file, the npm / pnpm install one-liners, a starter prompt that we've tuned against real runs, and the open-source repo when this automation ships there. One email, magic link, done.