Screaming Frog Guide 2026: Technical SEO Audits with AI Integration

Screaming Frog Guide 2026: Technical SEO Audits with AI Integration

Screaming Frog SEO Spider is a desktop-based website crawler that identifies 300+ technical SEO issues per crawl, broken links, duplicate titles, redirect chains, missing schema, and more. If you've ever wondered how agencies audit sites with millions of pages, this is the tool they reach for first.

This guide covers everything from your first crawl to advanced AI prompt integration, custom XPath extraction, and the regex patterns that save hours on every audit.

What Is Screaming Frog and Why Do SEOs Swear By It?

Screaming Frog SEO Spider is a desktop application that crawls websites the way Google does, analyzing every URL for technical SEO issues like broken links, missing meta tags, redirect chains, and orphaned pages, all without sending your data to someone else's cloud.

Unlike cloud-based crawlers that throttle your crawl speed and charge per URL, Screaming Frog runs locally on your machine. You get full control over crawl depth, rendering, and speed. That's why it's been the go-to technical SEO audit tool for agencies, in-house teams, and developers since 2010.

The tool covers response codes, page titles, meta descriptions, H1/H2 analysis, image alt text, canonical tags, hreflang implementation, structured data validation, and JavaScript rendering. It integrates with Google Analytics, Google Search Console, PageSpeed Insights, and, as of v21 -- AI providers like OpenAI and Google Gemini.

For a broader look at how Screaming Frog fits alongside other tools, check out our complete SEO tools roundup.

Free vs Paid: What You Actually Get

The free version is genuinely useful for small sites. But once you hit the 500-URL ceiling, you'll need the paid license at £199/year (~$259).

How to Set Up Screaming Frog for Your First Crawl

Getting from download to running crawl takes about five minutes. Screaming Frog runs on Windows, macOS, and Linux, it's a Java application, so it works anywhere a JVM does.

Download and Install

Grab the installer from screamingfrog.co.uk. If you have a paid license, activate it under Licence > Enter Licence after installation. The free version works immediately with no registration.

Configuration Settings That Actually Matter

Before you hit Start, tweak three things:

1. User-Agent, Configuration > User-Agent. Default is "Screaming Frog SEO Spider". Switch to Googlebot if you want to see what Google sees (some sites serve different content based on user-agent).

2. Crawl Speed, Configuration > Speed. The default 5 threads is fine for most sites. For your own site, crank it to 10-20. For client sites, keep it conservative to avoid triggering rate limits.

3. Rendering Mode, Configuration > Spider > Rendering. Set to JavaScript if the site uses React, Angular, Vue, or any SPA framework. This uses Chromium under the hood and slows crawls significantly, but you'll miss dynamically loaded content without it.

4. Memory Allocation, This is the one most people skip. The default 2GB will choke around 30K URLs.

# Increase memory allocation for large sites
# Edit ScreamingFrogSEOSpider.l4j.ini (Windows) or Info.plist (Mac)
-Xmx8g

If you're crawling a site with 50K+ pages, bump memory to 8GB minimum. For 500K+ URL sites, allocate 16GB and consider using database storage mode (Configuration > System > Storage Mode > Database).

Enter your URL in the top bar and click Start. That's it.

Running Your First Technical SEO Audit

A technical SEO audit in Screaming Frog follows five steps: crawl the site, review response codes, check meta tags, audit internal links, and export reports for your team, a process that catches issues invisible to manual review.

According to Semrush's 2025 Website Health benchmark, 72% of websites fail at least one critical technical SEO factor. In our experience, the first crawl of any site reveals at least 5-10 critical issues that are invisible without a crawler.

For a complete understanding of technical SEO fundamentals, see our complete SEO guide.

Understanding Response Codes (200, 301, 302, 404, 500)

Click the Response Codes tab after your crawl completes. Here's what to prioritize:

• 4xx (Client Errors), Broken pages. These need redirects or content. Sort by inlinks to fix the most impactful ones first.
• 3xx (Redirects), Not inherently bad, but redirect chains (301 > 301 > 200) waste crawl budget. Check Reports > Redirect Chains.
• 5xx (Server Errors), Your server is choking on these URLs. Could be timeout issues, missing database records, or misconfigured routes.
• 200 with issues, Pages that load fine but have thin content, duplicate titles, or missing meta descriptions. Filter using the Page Titles or Meta Descriptions tabs.

Meta Tags and Title Audit

The Page Titles tab flags duplicates, missing titles, titles over 60 characters, and titles under 30 characters. Same for Meta Descriptions, filter by "Missing," "Duplicate," or "Over 160 Characters."

Sort by inlinks to prioritize your most-linked pages. A missing meta description on your homepage matters more than one on a pagination page.

Internal Linking Analysis

The Internal tab shows every internal link on your site with anchor text, link depth, and follow/nofollow status. Two things to check immediately:

• Orphan pages, Pages with zero internal links pointing to them. Find these under Reports > Orphan Pages (requires a sitemap or Google Analytics integration).
• Deep pages, Anything beyond crawl depth 3 is probably getting minimal link equity. Filter by Crawl Depth and consider adding internal links to important deep pages.

For content optimization after your technical audit is done, check out our Surfer SEO guide for content optimization.

Finding and Fixing Broken Links, Redirects, and Canonicals

These three issues account for the majority of technical SEO problems on most sites. Screaming Frog surfaces all of them in a single crawl.

Broken Links: Find and Fix in 3 Steps

1. Go to Response Codes > filter by Client Error (4xx)
2. Right-click any broken URL > Inlinks to see which pages link to it
3. Bulk export the list (Export > All) and hand it to your dev team with recommended redirect targets

Pro tip: Sort broken URLs by number of inlinks. A 404 page with 50 internal links pointing to it is bleeding link equity. A 404 with 1 inlink from a footer is low priority.

Redirect Chains: Why They Kill Your Crawl Budget

Redirect chains of 3+ hops aren't just slow, Google may stop following them entirely. Screaming Frog flags these automatically under Reports > Redirect Chains.

The fix is straightforward: update the origin URL to point directly to the final destination. If Page A redirects to Page B, which redirects to Page C, update Page A's redirect to go straight to Page C.

Check Reports > Redirect Loops too. A loop (A > B > A) creates an infinite redirect that returns a browser error. These are critical fixes.

Canonical Tag Audit

Under the Canonicals tab, look for three problems:

• Missing canonicals, Every indexable page should have a self-referencing canonical
• Canonicalized to wrong URL, Pages pointing their canonical to an unrelated page (common after site migrations)
• Non-indexable canonical targets, A canonical pointing to a 404 or noindexed page defeats its purpose

Custom Extraction with XPath and CSS Selectors

Custom extraction lets you pull specific data from every page during a crawl, structured data, tracking codes, Open Graph tags, or any HTML element, turning Screaming Frog from an auditor into a full-site scraper.

This is the feature that separates power users from beginners, and we use custom extraction on every audit to verify tracking codes and schema markup across thousands of pages simultaneously. You can combine Screaming Frog crawl data with keyword data from tools like Semrush for a complete picture.

Set it up under Configuration > Custom > Extraction. Choose XPath or CSS Path, paste your selector, and run the crawl. Results appear in their own tab.

Here are five extractions I use on nearly every audit:

// Extract schema type from JSON-LD
//script[@type='application/ld+json']

// Extract all H2 headings
//h2/text()

/* Extract canonical URL */
link[rel='canonical']::attr(href)

// Extract Open Graph title
//meta[@property='og:title']/@content

// Check for Google Analytics 4 tracking
//script[contains(@src,'gtag')]/@src

The JSON-LD extraction is particularly powerful. Run it across an entire site and you'll instantly see which pages have structured data and which don't, no manual page-by-page checking.

You can add up to 100 custom extractions per crawl (paid version only). Combine them to build a complete audit of your site's metadata, tracking, schema, and third-party scripts in one pass.

How Does Screaming Frog's AI Integration Work with OpenAI, Gemini, and Anthropic?

Screaming Frog's AI integration (v21+) runs custom prompts against every page's content during a crawl, letting you classify content quality, detect missing schema, score meta descriptions, and find internal link opportunities at scale using OpenAI, Google Gemini, Anthropic Claude, or local Ollama models.

This is the feature no other Screaming Frog guide covers, and it's the biggest reason to upgrade to v23. After testing all four AI providers in Screaming Frog, OpenAI consistently returns the most structured responses for SEO classification tasks, though Gemini 2.5 Flash is faster and cheaper for simple yes/no checks.

Workshop Digital documented how they used AI integration to audit thousands of pages in a fraction of the time manual review would take.

Setting Up AI Providers (OpenAI, Gemini, Anthropic, Ollama)

Configuration > AI > select your provider. Enter your API key and choose your model. In v23, the default models are:

• OpenAI: gpt-5-mini (fast, cheap, good for classification)
• Google Gemini: gemini-2.5-flash (fastest option, great for bulk analysis)
• Anthropic: claude-sonnet-4-5 (best for nuanced content analysis)
• Ollama: Any local model (free, private, no API costs, but slower)

You can add up to 100 custom screaming frog AI prompts per crawl. Results appear in the AI tab with one column per prompt.

5 Ready-to-Use AI Prompt Templates

Copy these directly into Configuration > AI > Add Prompt:

Prompt 1: Content Quality Classifier
"Analyze this page's content. Classify as: Thin (<300 words of unique content), Adequate (300-800 words), Comprehensive (800+ words). Return only the classification and word count."

Prompt 2: Missing Schema Detector
"Review this page's HTML. List any Schema.org types that SHOULD be present based on the content but are MISSING. If the page is a product, check for Product schema. If it's an article, check for Article schema. Return 'None missing' if all appropriate schemas are present."

Prompt 3: Meta Description Quality Scorer
"Score this page's meta description from 1-10 based on: includes primary topic keyword (3pts), has a clear CTA or benefit statement (3pts), is between 145-160 characters (2pts), is unique and not template-generated (2pts). Return score and one-line improvement suggestion."

Prompt 4: Internal Link Opportunity Finder
"Analyze this page's content and identify 3 topics mentioned that could link to other pages on the same site. Return each topic and a suggested anchor text phrase."

Prompt 5: Accessibility Quick Check
"Check this page for: missing alt text on images, missing form labels, low contrast text indicators, missing ARIA landmarks. Return a brief accessibility score (Good/Fair/Poor) with the top issue to fix."

The Content Quality Classifier alone saves hours. Run it across a 10,000-page site and you'll have a thin content report in minutes instead of days.

Copy-Paste Regex Patterns for Common SEO Tasks

Screaming Frog supports regex in Include/Exclude URL filters and Custom Search, making it possible to isolate specific page types, find tracking implementations, or audit hreflang tags without manually checking each page.

Most SEO guides skip regex entirely. Here's a library of patterns I keep in a text file and paste into every audit:

To use these: for URL filters, go to Configuration > Include/Exclude. For Custom Search patterns, go to Configuration > Custom > Search. Select "Contains" and set the search mode to Regex.

The taxonomy page filter (tag|category|author) is particularly useful for WordPress audits. These pages often generate thin content and duplicate title issues at scale.

What's New in Screaming Frog Version 23 (2026)?

Screaming Frog v23 "Rush Hour," released October 2025, updated AI model defaults to gpt-5-mini, gemini-2.5-flash, and claude-sonnet-4-5, and added Ahrefs v3 API integration alongside semantic similarity embeddings for smarter redirect mapping.

Here's what changed across the v23.x releases, per the official release history:

• v23.0 (October 2025), Updated AI model defaults across all four providers. Ahrefs v3 API integration replaces the legacy v2 connector. Semantic similarity embeddings for redirect mapping, the tool now suggests redirects based on content similarity, not just URL patterns.

• v23.1-v23.3 (November 2025 - February 2026), Bug fixes for JavaScript rendering edge cases, improved memory handling for 1M+ URL crawls, and auto-deleting crawls feature (set crawls to auto-delete after X days to manage disk space).

The semantic similarity feature for redirect mapping is the standout. During site migrations, you can now upload a list of old URLs and Screaming Frog will suggest the best matching new URL based on actual page content, not just URL structure. According to the v23 release blog, this uses local embeddings, no data leaves your machine.

Is Screaming Frog Better Than Sitebulb, Lumar, or Ahrefs Site Audit?

Screaming Frog is the best choice for power users who want full control, AI integration, and custom extraction, but Sitebulb wins on visualization, Lumar suits enterprise teams, and Ahrefs Site Audit works best if you already pay for Ahrefs.

We've used both Screaming Frog and Sitebulb on the same sites, here's what we found:

Verdict: Screaming Frog wins for power users who want full control and AI integration. Sitebulb is better if you want prettier visualizations without the learning curve. Lumar makes sense only at enterprise scale with dedicated SEO teams. And if you're already paying for Ahrefs, their Site Audit is good enough for basic website crawl analysis.

For a deeper look at Ahrefs' capabilities beyond site auditing, see our Ahrefs deep-dive. For a full comparison of all SEO tools, check our complete SEO tools roundup.

FAQ

What is Screaming Frog used for?

Screaming Frog SEO Spider is used for technical SEO auditing. It crawls your website and identifies broken links, missing or duplicate meta tags, redirect chains, canonical tag issues, structured data errors, and JavaScript rendering problems. The v21+ versions also support AI-powered content analysis using OpenAI, Gemini, Anthropic, or Ollama.

Is Screaming Frog free or paid?

The free version crawls up to 500 URLs per session with basic features. The paid license costs £199/year (~$259) and unlocks unlimited crawling, JavaScript rendering, custom extraction, AI integration, scheduled crawls, crawl comparison, and the ability to save and reopen crawl files. Most professionals need the paid version.

How do I do a site audit with Screaming Frog?

Enter your URL in the top bar and click Start. Once the crawl finishes, review the Response Codes tab for broken pages and redirects, check Page Titles and Meta Descriptions for duplicates or missing tags, and examine the Directives tab for noindex issues. Export reports under File > Export for your team.

How do I find broken links with Screaming Frog?

After crawling, go to the Response Codes tab and filter by "Client Error (4xx)." This shows every broken page on your site. Right-click any broken URL and select "Inlinks" to see which pages link to it. Export the list and set up 301 redirects to the most relevant live pages.

What is the difference between Screaming Frog free and paid?

The free version limits you to 500 URLs per crawl and disables saving crawls, JavaScript rendering, custom extraction, AI integration, API connections, scheduled crawls, and crawl comparison. The paid version (£199/year) removes all limits and unlocks every feature. If your site has more than 500 pages, you need paid.

Can Screaming Frog crawl JavaScript websites?

Yes, with a paid license. Go to Configuration > Spider > Rendering and select JavaScript. Screaming Frog uses Chromium under the hood to render pages exactly like a browser would. Expect crawl times to increase 3-5x compared to HTML-only mode. Make sure to allocate extra memory for JS-heavy sites.

How do I use Screaming Frog's AI integration with OpenAI?

Go to Configuration > AI > OpenAI. Enter your API key, select gpt-5-mini (the default in v23), write your prompt in the prompt field, and run the crawl. Results appear in the AI tab with one column per prompt. You can run up to 100 screaming frog AI prompts per crawl for bulk content analysis.

How many URLs can Screaming Frog crawl?

The free version caps at 500 URLs per crawl. The paid version has no limit, teams have successfully crawled sites with millions of URLs. For large sites (50K+), increase memory allocation to 8-16GB in the config file and switch to database storage mode under Configuration > System > Storage Mode.

Is Screaming Frog better than Sitebulb?

Screaming Frog offers more power and flexibility, AI integration with four providers, deep custom extraction via XPath and CSS selectors, CLI automation for scheduled crawls, and broader API integrations. Sitebulb offers better visualization with auto-generated audit reports and is easier for beginners. Most experienced SEOs prefer Screaming Frog.

How do I schedule automated crawls in Screaming Frog?

This is a paid feature. Use Screaming Frog's command-line interface with Windows Task Scheduler or cron on Mac/Linux. Set the command to run the spider with a specific config file and auto-export results. Example: ScreamingFrogSEOSpiderCli --crawl https://example.com --output-folder /reports --export-tabs "Response Codes,Page Titles".