Screaming Frog SEO Spider is the tool that technical SEO professionals reach for first when auditing a site. It’s been the industry standard for over a decade because it does one thing better than everything else: it crawls a website the way Google’s bot does and reports everything it finds in a format that makes issues easy to identify and prioritize.
The free version crawls up to 500 URLs. The paid version (£259/year as of 2026) crawls unlimited URLs and adds integrations with Google Analytics, Search Console, and PageSpeed Insights. This tutorial covers the workflows that apply to both versions.
Setting Up Your First Crawl
Open Screaming Frog, enter your domain in the search bar, and click Start. That’s the basic crawl. But a few configuration steps before you start make the data much more useful.
Configuration > Spider Settings
Check “Crawl External Links” if you want to verify that your external links aren’t broken. Leave unchecked for faster crawls focused only on your own site.
Check “Follow Internal Nofollow” if you want to see all pages that nofollow internal links point to — useful for understanding your internal link architecture completely.
Under “Extraction,” enable “Extract Custom” if you want to extract specific data like Heading 2s, canonical tags, or any other on-page element beyond Screaming Frog’s defaults.
Configuration > User Agent
The default user agent is Screaming Frog’s own bot. For technical audits, switch to “Googlebot” to see your site as Google’s crawler sees it. This catches cases where sites serve different content to different user agents — a technique called “cloaking” that is a Google policy violation, but which also sometimes occurs accidentally.
Configuration > Crawl Depth
Set a crawl depth limit for large sites to prevent runaway crawls on sites with thousands of URLs. Start with depth 5 and increase if needed.
The Key Reports and What to Do With Them
Response Codes
The first report to check after any crawl. In the top filter bar, select “Response Codes” to see the breakdown of all HTTP status codes in your crawl.
- 301 Redirect: Expected for legacy URLs that’ve been redirected. Investigate if important content pages return 301s — they should return 200.
- 302 Redirect: Temporary redirects that should almost always be changed to 301. Temporary redirects don’t pass full PageRank.
- 404 Not Found: The most common and impactful issue. Every 404 is a wasted link opportunity and a poor user experience. Export these and fix them: redirect to the closest relevant live page.
- 500 Server Error: Indicates server-side problems. These need immediate attention — Googlebot stops crawling URLs that consistently return 5xx errors.
Page Titles
Filter > “Page Titles” shows: Missing (no title tag), Duplicate (same title on multiple pages), Too Long (over 60 chars), Too Short (under 30 chars), Same as H1. Duplicate and missing titles are the priority fixes — they cause significant ranking fragmentation.
Meta Descriptions
Same categories as page titles. Missing meta descriptions are auto-generated by Google from page content — which often produces ugly or unhelpful snippets. Writing custom meta descriptions for your top-traffic pages is worth the time investment.
H1 Tags
Filter > “H1” shows: Missing H1, Duplicate H1, Multiple H1 (more than one H1 on a page). Every page should have exactly one unique H1. Missing H1s on important pages are quick fixes with meaningful on-page SEO impact.
Images
Filter > “Images” > “Alt Text” > “Missing” shows all images without alt text. Alt text serves both accessibility (screen readers) and SEO (helps Google understand what images depict). Export this list and add descriptive alt text to all meaningful images.
Redirect Chains
Reports > Redirects > Redirect Chains identifies sequences of redirects (A → B → C → D) rather than direct redirects (A → D). Each hop in a redirect chain loses some link equity and adds latency. Collapse chains to single-hop redirects wherever possible.
Internal
The “Internal” tab shows all internal links Screaming Frog found during the crawl. Sort by “Inlinks” column to see which pages have the most internal links pointing to them — your de facto authority hierarchy. Sort by “Outlinks” to see which pages link to the most others.
Under Reports > Bulk Export > All Internal URLs, you can export a complete internal link map for further analysis in spreadsheet tools.
Connecting Screaming Frog to Google Analytics and Search Console (Paid Feature)
The paid version allows API connections to GA4 and Search Console. This combination is powerful: you can segment your crawl data by traffic, organic sessions, and conversion data to prioritize fixes by business impact rather than just issue count.
A page with 200 broken links is less urgent to fix than a page with 3 broken links that drives 50,000 monthly organic sessions. The API integration makes this prioritization possible without manual cross-referencing.
Exporting and Prioritizing Your Findings
At the end of every audit, export your findings into three categories:
- Critical (fix this week): 404 errors on pages with inbound links, server errors, noindex tags on important content, missing title tags on top-traffic pages
- High priority (fix this month): Duplicate titles, broken redirect chains, missing H1s, redirect chains
- Optimization (ongoing): Missing alt text, long titles, missing meta descriptions, pages with few internal links
Screaming Frog is most valuable run on a regular schedule — monthly for actively evolving sites, quarterly for more stable ones. Technical issues accumulate quietly and compound over time. Regular crawls catch them before they affect rankings.
Pair Screaming Frog data with Google Search Console for the most complete technical SEO picture, and use the 47-point technical audit checklist to ensure you’re covering everything Screaming Frog doesn’t catch automatically.
