Screaming Frog SEO Spider: The Complete Tutorial for Technical SEO Audits

Screaming Frog SEO Spider is the tool technical SEO professionals reach for first when auditing a site. It has been the industry standard for over a decade because it does one thing better than almost any alternative: it crawls websites the way search engine bots do, then organises the data into structured, filterable reports.

This tutorial covers the complete workflow — from initial setup through every major report tab — with specific instructions for the issues each tab is designed to surface, and guidance on how to prioritise what to fix first.

Setup and Configuration Before You Crawl

Before starting a crawl, configure Screaming Frog for the type of audit you are running. The default settings work for most crawls, but a few adjustments make the data more useful.

User Agent

Set the user agent to Googlebot Desktop (Configuration → User-Agent). This tells servers to respond as they would to Google’s crawler, which can help surface differences between what users see and what the crawler sees — such as server-side user-agent blocking, redirect rules that apply only to bots, or content variations.

Note: using a Googlebot user agent simulates crawler behaviour but does not guarantee exact replication of how Google processes, renders, or evaluates each page. Use crawl data as a diagnostic starting point, not a definitive map of Google’s index.

If your audience is primarily mobile, consider running an additional crawl using the Googlebot Smartphone user agent (Configuration → User-Agent → Custom → select Googlebot Smartphone). Comparing desktop and smartphone crawls can reveal mobile-specific rendering issues, redirect differences, or content variations that would not surface in a desktop-only crawl. (Google: How Googlebot crawls)

JavaScript Rendering

For JavaScript-heavy sites, enable JavaScript rendering (Configuration → Spider → Rendering → JavaScript). This makes Screaming Frog render pages as a browser would before crawling, catching content that only appears after JavaScript execution. JS rendering significantly increases crawl time and resource usage — use it selectively for sites where JavaScript content is important for SEO.

Connect Google Analytics and Search Console

Under Configuration → API Access, connect your Google Analytics and Google Search Console accounts. This adds sessions, organic traffic, and Search Console clicks and impressions data to each URL in the crawl — transforming the audit from a technical list into a prioritised list based on actual traffic value. This integration requires the paid licence.

Understanding the Crawl Results: Major Tabs

Which Issues to Fix First

Technical SEOs do not fix everything. They fix what matters most to business performance first. Not every Screaming Frog warning deserves the same urgency — a single noindex tag on your highest-converting page is usually more important than hundreds of missing image alt attributes across low-value archive pages.

Screaming Frog Audit Workflow

A structured crawl workflow prevents spending time on low-value issues while critical problems remain unaddressed. Follow this sequence:

Response Codes: Start Here

The Response Codes tab is the first stop in any technical audit. Filter to each status code category to find specific issues:

4XX Errors (Client Errors)

Filter to 4XX to see all 404 (not found) and other client errors. These represent broken URLs — either pages that have been deleted without a redirect, or internal links pointing to non-existent URLs.

For each 404, check the Inlinks tab to see which pages link to the broken URL. If the broken URL has internal links or external backlinks pointing to it, create a 301 redirect to the most relevant live equivalent. Broken pages with no inlinks and no external backlinks can generally return a 410 (Gone) or simply be left as 404 without a redirect — forcing a redirect to an unrelated page creates its own problems. (Google Search Central: HTTP status codes)

3XX Redirects

Filter to 3XX to see all redirects. Two issues to address:

• Redirect chains: A redirects to B, which redirects to C. Each additional redirect increases latency and adds crawl overhead. Where practical, update redirects so they point directly to the final destination rather than passing through multiple intermediate URLs.
• 302 (temporary) redirects for permanent moves: Permanent URL changes are generally better handled with 301 redirects. Although Google can sometimes treat long-standing 302 redirects similarly to 301s, using the correct status code provides clearer intent for both search engines and future site maintainers. (Google Search Central: Redirects and Google Search)

Page Titles and Meta Descriptions

The Page Titles and Meta Descriptions tabs show the metadata for every crawled page. Use the filters at the top of the tab to surface specific issues:

• Missing: Pages with no title tag or meta description. Missing titles are replaced by Google with auto-generated alternatives that may not represent the page well. Fix missing titles on high-traffic and commercially important pages first.
• Duplicate: Multiple pages sharing the same title or meta description. Duplicates reduce differentiation between pages and can contribute to keyword cannibalisation. Each page needs a unique title that reflects its specific content.
• Over length: Titles over approximately 580 pixels wide (roughly 60 characters) are truncated in search results. Meta descriptions over approximately 920 pixels (roughly 155 characters) are also truncated. Screaming Frog flags these in pixel width — focus on content quality rather than hitting exact character counts, since truncation depends on the specific characters used.
• Under length: Very short titles (under 10 characters) often indicate pages with inadequate metadata worth reviewing.

Directives Tab: Finding Accidental Noindex

The Directives tab shows every page’s canonical, noindex, nofollow, and other meta directive settings. The most critical issue to find here is unintentional noindex directives.

Filter to noindex to see every page that has a noindex tag applied. Go through this list and verify that every noindex is intentional. A noindex tag applied to an important page — often added during site development or staging and not removed — will prevent that page from appearing in search results regardless of its content quality or backlinks. (Google Search Central: Prevent indexing with noindex)

Also look for canonical conflicts: pages where the canonical URL points to a different page than intended, or where canonical tags are missing on paginated content that should self-reference. A canonical pointing to a non-indexable URL or a redirect chain is a common source of indexing problems that is easy to miss without a crawl. See the canonical tags guide for how to audit and correct canonical implementation.

Inlinks Tab: Finding Orphaned Pages

The Inlinks tab (accessible by selecting a URL and clicking the Inlinks tab in the lower panel) shows all internal pages linking to a selected URL. The Bulk Export option allows you to export inlink counts for all pages.

Sort the export by inlink count ascending to find pages with very few or zero internal links. Pages with very few internal links receive limited internal authority. True orphan pages — those with no crawlable internal links pointing to them — often cannot be discovered through a standard crawl at all; they require sitemap inclusion, external links, or direct URL submission before search engines can consistently find them.

Fix weakly linked or orphaned pages by adding contextually relevant internal links from other pages in the same topic area. Our internal linking strategy guide covers how to prioritise which internal links to add for maximum SEO impact.

Images Tab: Alt Text and File Size

The Images tab shows every image file encountered during the crawl. Key filters to use:

• Missing Alt Text: Images without alt attributes miss an opportunity to provide context to search engines and fail accessibility standards. Filter to missing alt text and prioritise fixing images on high-traffic pages first.
• Large file sizes: Sort by file size descending to find images contributing meaningfully to page weight. Prioritise unusually large images, especially above-the-fold assets or images that affect Largest Contentful Paint (LCP) on key pages. Images not yet in a next-gen format (WebP, AVIF) are candidates for compression and format conversion. (web.dev: Core Web Vitals and LCP)

Sitemaps Tab: Audit Your XML Sitemap

The Sitemaps tab compares your crawled pages against your XML sitemap. Two key issues to identify:

• Indexable pages not in sitemap: Pages that should be indexed but are not included in your sitemap. XML sitemaps help search engines discover canonical URLs more efficiently, particularly for large or newly updated websites — though inclusion alone does not guarantee crawling or indexing. (Google Search Central: Build and submit a sitemap)
• Non-indexable pages in sitemap: Pages with noindex tags, 4XX errors, or canonicals pointing elsewhere that are listed in your sitemap. Sitemaps should contain only the canonical, indexable versions of your URLs. Non-indexable URLs in sitemaps send conflicting signals and should be removed.

Screaming Frog vs Other Audit Tools

Screaming Frog is one of several tools used in technical SEO audits. Understanding where it fits prevents using it for tasks better suited to other tools:

Common Screaming Frog Mistakes

• Enabling JavaScript rendering on every crawl: JS rendering dramatically increases crawl time and resource usage. Use it only for sites where JavaScript-loaded content is important to SEO — not as a default on every project.
• Not setting the user agent correctly: The default Screaming Frog user agent is not Googlebot. Always set the user agent explicitly for audits where server-side rendering or user-agent-based redirects may affect results.
• Ignoring robots.txt before crawling: If a site has significant robots.txt exclusions, crawling without reviewing them means the crawl may miss blocked sections or behave differently from Googlebot. Check Configuration → Robots.txt → Settings before starting.
• Fixing every issue equally: Screaming Frog flags hundreds of issues on most sites. Treating missing image alt text on a low-traffic archive page with the same urgency as a noindex tag on a high-converting product page wastes time and misprioritises client work.
• Exporting data without merging with traffic data: A list of technical issues without traffic context is an incomplete audit. Always merge crawl data with GSC or GA4 export before prioritising fixes.
• Ignoring the Search Console integration: The paid licence’s GSC integration adds real organic visibility data to every URL in the crawl. Auditing without it means ranking all issues by technical severity rather than business impact.
• Treating crawl data as ground truth for Google’s index: Screaming Frog shows what it can crawl. Google’s index may differ due to crawl budget constraints, JavaScript rendering differences, or noindex directives discovered after initial crawl. Use Search Console’s URL Inspection to verify indexing status for specific pages.

Free vs Paid: What You Get With Each Version

The free version of Screaming Frog crawls up to 500 URLs per crawl. For small sites, microsites, or quick spot-checks, 500 URLs is sufficient. It includes all major tabs — Response Codes, Page Titles, Meta Descriptions, H1, Directives, Images, and Links.

The paid licence (currently around £259 per year) removes the 500 URL limit and adds several features that matter for larger sites:

• Google Analytics and Search Console API integration (traffic data inside Screaming Frog)
• Scheduled crawls with crawl comparison (automatically detects changes between crawls)
• JavaScript rendering (renders pages before crawling)
• Custom extraction (pull specific data points using XPath, CSS selectors, or regex)
• Crawl storage and historical comparison

For SEO professionals auditing client sites regularly, the paid licence is a standard business expense. For one-off audits of small sites, the free tier is a practical starting point.

Using Screaming Frog Alongside Other Tools

Screaming Frog works best as part of a broader audit toolkit rather than in isolation. The combination that most technical SEOs use:

• Screaming Frog: Crawl data, on-page metadata, redirect analysis, internal link mapping, sitemap auditing
• Google Search Console: Index status, search performance data, Core Web Vitals field data, manual actions
• PageSpeed Insights / Lighthouse: Page-level performance metrics and specific technical recommendations
• Chrome DevTools: Real-time inspection of rendering, network requests, and JavaScript execution for individual pages
• Ahrefs or SEMrush: Backlink profile, keyword rankings, competitive data that Screaming Frog does not cover

The technical SEO audit checklist covers how to structure a full audit combining these tools into a prioritised findings report.

Frequently Asked Questions

Does Screaming Frog replace Google Search Console?

No. Screaming Frog crawls your site from the outside and surfaces technical issues at scale. Google Search Console shows real data from Google’s actual crawls and index — including which pages are indexed, which have manual actions, and how your pages perform in Search. The two tools are complementary: use Screaming Frog for technical discovery and Search Console for verifying real-world search outcomes.

Can Screaming Frog find orphan pages?

Partially. Screaming Frog identifies pages with zero or very few internal links within the crawl. True orphan pages with no crawlable internal links will not appear in a standard crawl at all, because Screaming Frog discovers URLs by following links. To find them, upload your XML sitemap and compare against crawled URLs, or connect Search Console or GA data to cross-reference known URLs against those discovered by the crawl.

How often should I crawl a website with Screaming Frog?

For most sites, a full technical crawl once per quarter is a reasonable baseline. Sites with frequent content changes, active development, or large-scale migrations benefit from monthly crawls or crawls after each significant change. The paid licence’s scheduled crawl and crawl-comparison features automate this process and alert you to changes between crawls.

Is the free version of Screaming Frog enough?

For sites under 500 pages, the free version is sufficient for most technical audits. It includes all major tabs and surfaces the most common issues. For larger sites, the 500-URL limit makes the paid licence (around £259 per year) necessary for a complete audit.

Does running Screaming Frog affect rankings?

No. Screaming Frog is a crawler that reads your site — it does not modify content, submit pages to Google, or interact with Google’s systems. Running a crawl is equivalent to a search bot visiting your pages. For crawl-rate-sensitive servers, use Configuration → Speed to set a crawl rate limit during the audit.

Sources

• Screaming Frog SEO Spider — Official Documentation
• Screaming Frog User Guide
• Google Search Central: Introduction to robots.txt
• Google Search Central: Redirects and Google Search
• Google Search Central: Prevent indexing with noindex
• Google Search Central: Build and submit a sitemap
• web.dev: Core Web Vitals (LCP, INP, CLS)