community-automations/etl-pipeline

DevOps & Code

PublicClaude subagent

ETL Pipeline Runner

Text-heavy ETL jobs fail quietly: a few malformed rows, a flaky source, and you find out days later when a report looks wrong. This runner makes every batch traceable.

sonnet1 weekPythonPostgresS3Slack
ClaudeClaude
ROI for
README.md

Why this subagent

Text-heavy ETL jobs fail quietly: a few malformed rows, a flaky source, and you find out days later when a report looks wrong. This runner makes every batch traceable.

It validates against the schema, cleans and enriches the text, retries transient errors with backoff, and quarantines records that keep failing instead of dropping them. Every run ends with an audit log that counts what went in, what was rejected, and what landed, so you can trust the numbers.

How it runs

    • Read

      Used at step 01 to kick off the pipeline.

    • Write

      Used at step 01 to kick off the pipeline.

    • WebFetch

      Used at step 01 to kick off the pipeline.

    • WebSearch

      Used at step 01 to kick off the pipeline.

Sample output

json
// Sample output
// (generated when the pipeline finishes)

Given a source batch, run extract-transform-load with retries and return a JSON audit log of rows read, transformed, retried, quarantined, and loaded.

Unlock the rest

The full agent definition, install snippet, and starter task are gated for community members.

Members get the full `.md` agent file, the npm / pnpm install one-liners, a starter prompt that we've tuned against real runs, and the open-source repo when this automation ships there. One email, magic link, done.