Indexation Audit Workflow: GSC, Logs, and Crawl Comparison

Most indexation audits stop at Search Console. The Coverage report shows which pages are indexed, which are excluded, and which have errors. That is useful. But it does not tell you what the crawler did on your server, which pages it never reached, and which pages it reached but deprioritized. That is where log files and crawl comparisons add the layer that catches the failure modes GSC misses.

This workflow combines three data sources: Google Search Console coverage, server log analysis, and a crawl comparison against the sitemap. The goal is to answer one question: are the pages that should be indexed actually getting indexed, and if not, why not?

Key takeaways

• GSC alone misses three failure modes: crawler behavior on your server, pages the crawler never reached, and pages it reached but deprioritized — logs and crawl comparisons catch what the Coverage report skips.
• The three-layer audit combines GSC coverage, server logs, and sitemap comparison: each layer surfaces a different category of indexation gap; the overlap between all three is where the real problems live.
• Triage by impact: critical (high-traffic pages not indexed), high (sitemap entries not indexed with no clear reason), medium (low-traffic pages with intentional exclusions), and low (URLs discovered via links but not in sitemap).
• Submitting a sitemap does not fix indexation: it tells Google what you want crawled; fixing noindex directives, canonical mismatches, and robots.txt blocks is what actually restores indexing.
• Run the audit quarterly, not just when traffic drops: indexation problems compound quietly — a page that stops getting crawled loses ranking signals during the window it is unindexed.

The three-layer indexation audit

Layer 1: Search Console Coverage

Start with the GSC Coverage report. Export the full table — all pages in all four categories: Indexed, Excluded, Discovered (not indexed), and Error.

The important split is not the total indexed count. It is the ratio between:

• Indexed pages that are actively crawling — these should stay indexed.
• Discovered not indexed — Google found these but chose not to index. This is the actionable set.
• Excluded with a crawl directive — noindex, canonical to another page, blocked by robots.txt. These are intentional exclusions.

The Discovered-not-indexed pile is where indexation problems hide. A large Discovered pile with a small Indexed pile means Google is finding pages faster than it can process them — or it is finding reasons not to index.

Sort the Discovered pile by the number of submitted URLs. If 500 product pages were submitted and 400 are Discovered-not-indexed, the crawl budget is the bottleneck, not the content quality.

Layer 2: Log file analysis

GSC tells you what Google decided. Logs tell you what Google actually did on your server.

The log analysis that matters for indexation audits:

Crawl frequency by URL pattern. Which sections of the site are Googlebot crawling most frequently? If product pages are crawled daily but blog posts are crawled weekly, the crawl budget is being allocated to the wrong content.

HTTP response codes for crawled URLs. What percentage of Googlebot requests return 200 versus 404 versus 500? A site with 3% of Googlebot requests hitting 404s is losing crawl budget to dead ends.

Crawl depth. How many hops from the homepage does Googlebot reach before it stops? Log analysis shows the actual crawl depth, which is always deeper than the sitemap declaration suggests. Pages that require four or more clicks from the homepage are often not getting crawled frequently enough to stay indexed.

Robots.txt decisions. Logs show which URLs Googlebot requested but did not fetch because of a robots.txt block. These are not indexation failures — they are deliberate exclusions. But if the exclusion was unintentional, the log is the only place it shows up.

Layer 3: Crawl comparison against sitemap

The sitemap is your declaration of what should be indexed. The actual index is what Google chose. The gap between the two is the audit finding.

The comparison workflow:

1. Export all URLs from the XML sitemap.
2. Compare against the GSC Coverage report for the same period.
3. Flag URLs that are in the sitemap but not in the Indexed set.
4. Flag URLs that are in the Indexed set but not in the sitemap.
5. For each flagged URL, determine why it is in the wrong pile.

The sitemap-not-indexed failures are the most actionable. Common causes:

• Canonical mismatch — the page is in the sitemap but canonicalized to another URL.
• Noindex in the <head> — the page is included in the sitemap but blocked from indexing with a meta tag.
• Blocked by robots.txt — the page URL path is disallowed in robots.txt even though it appears in the sitemap.
• Soft 404 — the page returns 200 but has no meaningful content.
• Crawl budget exhaustion — Googlebot reached the crawl budget limit before processing this page.

The specific indexation defects that appear most often

After running this workflow across dozens of sites, certain failure patterns appear consistently.

Pagination canonicalized to page 1. When page 2, page 3, and subsequent paginated pages canonicalize to page 1, Google indexes only page 1 and treats the paginated pages as duplicates. The traffic that would have gone to pages 2 through 10 disappears from the index.

Staging or development pages left in the sitemap. Sites that publish a sitemap from their CMS without filtering staging URLs often have dev or uat subdomain URLs in the sitemap. If those URLs resolve to live content, Google indexes the wrong version.

Filtered views with too many parameters. URL parameters for filtering product catalogs create an explosion of URL combinations. If the site allows Googlebot to crawl all combinations without proper canonical or noindex treatment, the crawl budget fills with near-duplicate filtered pages instead of the canonical category pages.

HTTP/HTTPS canonical mismatch. The HTTPS version of a page is canonical but the sitemap includes the HTTP version, or vice versa. Google resolves the discrepancy and indexes one version — but it may pick the wrong one if the signal is mixed.

404 pages that still return 200. Soft 404s are pages that return a 200 status code but have no useful content. Googlebot detects these over time and removes them from the index, but they waste crawl budget until it does.

The triage priority

Not every indexation gap needs fixing the same way. Triage by impact:

Critical — high-traffic pages not indexed. If a page that should be sending traffic is Discovered-not-indexed, the cause needs to be identified and fixed. Run a site audit using the SEO Analyzer against the URL.

High — sitemap entries not indexed with no obvious reason. Check canonical, noindex, and robots.txt. The indexation defects post covers the specific fixes for each cause.

Medium — low-traffic pages not indexed with clear self-canonicalization or intentional noindex. These do not need fixing; they need confirming the directive is correct.

Low — URLs in GSC but not in sitemap. These are usually pages discovered through links. Verify they should be indexed and add to sitemap if yes.

The workflow in practice

Run this quarterly, not just when traffic drops. Indexation problems compound — a page that stops getting crawled does not just lose rankings, it starts accumulating stale content signals as the cached version grows older.

Quarterly audit steps:

1. Export GSC Coverage report for all pages.
2. Pull server logs for the same period — filter for Googlebot user agents.
3. Compare sitemap URL list against GSC Coverage.
4. For every Discovered-not-indexed page, determine cause.
5. For every sitemap-not-indexed page, fix or confirm the directive.
6. Submit the fixed sitemap and request reindexing for fixed pages.

Use the SEO Analyzer for the initial scan. The crawled currently not indexed guide covers the specific fixes for each Discovered-not-indexed cause.

Frequently asked questions

How often should I run a full indexation audit?

Quarterly is the minimum. If the site has frequent content changes (daily news, product catalog updates), monthly is better. Each audit catches problems before they compound into traffic loss.

Does submitting a sitemap fix indexation problems?

No — submitting a sitemap tells Google what you want crawled. It does not change what Google chooses to index. If a page is not indexed because of a noindex directive, canonical mismatch, or robots.txt block, fixing the directive and re-requesting indexing is what works.

How do I know if my crawl budget is exhausted?

In Search Console, if the Discovered-not-indexed pile is large and growing, Googlebot is finding pages faster than it is processing them. Check the crawl stats report for the daily crawl request volume. If it has hit the ceiling, reduce the total URLs in the index and improve crawl efficiency by fixing the issues above.

What does it mean if Googlebot is crawling a page but not indexing it?

Crawl without index is usually one of three things: a noindex directive on the page, a canonical pointing to another page, or a quality signal that Googlebot recorded but decided the page does not deserve inclusion. In all three cases, check the page-level GSC report for the specific reason under the Indexation tab.

The bottom line

Search Console tells you what Google decided. Logs tell you what Googlebot actually did. Sitemaps tell you what you declared. The audit that uses all three surfaces the gap between intention and reality — and that gap is where indexation problems live.

Run the full three-layer audit quarterly. Fix critical and high-priority gaps immediately. Confirm medium and low ones. The log file analysis guide covers the log parsing workflow in detail.