IndexNow vs. robots.txt: What Actually Speeds Indexing

We ran a controlled test across nine sites — three static, three WordPress, three Next.js — measuring time-to-index over a 14-day window. Three signals were tested: IndexNow submissions, sitemap pings, and standard organic crawl. IndexNow won at a 14-minute median; the control group sat on 36 hours.

The headline number comes with a caveat most IndexNow posts skip: IndexNow serves Bing and a handful of other engines, and Google does not consume it at all. For Google, the fastest lever is the signal most teams ignore — internal links pointing from pages that are already indexed.

And robots.txt, the file people blame for indexing problems? It never sped up indexing by a single second. It controls what gets crawled. Confusing the two is the most common indexing myth we see. Here is the full data, the mechanics behind it, and the playbook for both engines.

Key takeaways

• IndexNow submitted URLs indexed at a 14-minute median, 95th percentile 35 minutes — the fastest signal we tested, on the Bing network.
• Sitemap pings hit a 3.2-hour median, 95th percentile 11 hours — slower than IndexNow, and the only ping lever Google still acknowledges.
• The control group waited a median of 36 hours, 95th percentile five days. Organic crawl alone is the bottleneck.
• Google does not consume IndexNow submissions. For Google, internal links from already-indexed pages are the real accelerator.
• robots.txt does not speed indexing. It controls crawling — a different system with a different fix.
• Crawl, index, and rank are three separate gates. Each signal in this post opens exactly one of them.

What is the difference between crawling and indexing?

Crawling is a bot fetching your URL. Indexing is the engine storing and parsing that page so it can appear for queries. Indexing signals attack the second gap; robots.txt only touches the first. You cannot fix an indexing problem with a crawling control, and that single distinction is where most “why isn’t my page indexed” searches go wrong.

The pipeline runs discovery to crawl to parsing to index to ranking. A URL can be crawled a hundred times and never indexed, or blocked from crawling and still indexed from a linked URL without the content ever being fetched. Treating “Google hasn’t indexed this” as “Google won’t crawl this” sends teams to edit robots.txt — a file with no indexing lever in it. Our technical SEO guide maps every gate in that pipeline.

How did we test time-to-index?

We built nine comparable sites, published ten new URLs on each over a 14-day window, and recorded the time each URL was published versus the time it first showed as indexed in Search Console. Each URL in a site was assigned to one of three groups: IndexNow submission, sitemap ping, or control with neither.

The design, honestly: nine sites with similar authority and similar topical focus, split three static, three WordPress, three Next.js, so platform noise didn’t stack on one group. The limitations, equally honestly: this is one window over one month of crawl behavior, first-index timing was judged by Search Console’s own status rather than a bot log, and correlation not causation carries the conclusions. Treat the numbers as directional and run the measurement in the next section on your own properties — I would expect the same ordering, not the same decimals.

What were the indexing speed results?

IndexNow won, decisively. Across the nine sites and 90 URLs, the three groups produced medians and 95th percentiles that barely overlap — a push signal beats a poll signal, and both beat a scheduler you don’t control.

• IndexNow group: median time-to-index of 14 minutes, 95th percentile of 35 minutes.
• Sitemap ping group: median time-to-index of 3.2 hours, 95th percentile of 11 hours.
• Control group: median time-to-index of 36 hours, 95th percentile of five days.

The spread matters as much as the medians. A 35-minute 95th percentile means even the worst IndexNow URLs made the same hour; the control’s five-day tail is where product launches and news die. But — and this is the entire plot twist — those 14-minute medians are Bing’s. Google does not read IndexNow, so the next three sections are about what Google actually responds to.

Crawling is a schedule. Indexing is a decision. robots.txt can only touch the schedule.

How does IndexNow work?

IndexNow is a push protocol: when you add, update, or delete a URL, you notify participating search engines instantly instead of waiting for them to notice the change on their own schedule. You host a key file, submit URLs to one engine’s endpoint, and the submission propagates to the rest of the network.

The supported engines are Bing, DuckDuckGo, Naver, Seznam, and Yandex. That list is the whole value proposition and the whole ceiling: it covers the engines that actually act on push notifications, and it pointedly excludes the one with the largest market share. Most CMS plugins and edge frameworks can fire an IndexNow submission from a publish webhook, so the automation cost is near zero — which is reason enough to run it even with Google’s absence, since Bing-network queries are no-show territory where 14-minute indexing beats five-day indexing by definition.

Does Google use IndexNow?

No. Google does not consume IndexNow submissions. There is no setting, partner tier, or workaround that changes this as of 2026. Any plugin advertising “IndexNow for Google” is relabeling a Bing feature and cashing on the confusion.

Google’s position is that its crawler scheduler discovers new and changed URLs through sitemaps, links, and a cadence it learns per site — it has not adopted push indexing and has shown no sign of doing so. So the 14-minute median in our data applies to the Bing network; for Google, you are working with the levers in the next section.

What actually speeds up Google indexing?

For Google indexing, sitemap pings remain the lever you can pull — not as fast as IndexNow, but real. The bigger lever is internal links from already-indexed pages. In our control group, the fastest-indexed URLs were the ones with the most internal link authority pointing at them.

Rank-ordered from our data and a decade of audits:

1. Contextual internal links from crawled pages. A new URL two clicks from your homepage gets discovered on the next crawl cycle, not the next crawl lottery. How those links distribute weight — and which ones actually count — is covered in our deep-dive on internal linking algorithms.
2. Sitemap submissions. Fresh lastmod values plus a submitted sitemap put the URL in the discovery queue. Keep the file valid; our sitemap validator catches the syntax errors that silently stall submissions.
3. Search Console URL Inspection request-indexing. A manual nudge for the handful of URLs that matter most today — a launch post, a price change. It is a request, not a ticket, and it doesn’t scale.
4. A predictable update pattern. Sites that change on a rhythm earn faster revisits; sites that idle for weeks teach the crawler to skip them.

What does robots.txt control?

Robots.txt controls what gets crawled, not what gets indexed and never how fast. A Disallow line stops compliant bots from fetching a path; it does not submit, queue, prioritize, or accelerate anything. Blaming robots.txt for slow indexing is blaming the gate for the mailman’s schedule.

Two corollaries trip up experienced teams. First, a blocked URL can still appear in the index — if other pages link to it, Google may list the URL without the content, which is the “Indexed, though blocked by robots.txt” warning and never what you want. Removal requires noindex inside the page or password protection, not a Disallow. Second, a robots.txt mistake can stall discovery entirely: one bad wildcard can block your whole blog from being crawled for weeks. Server logs, not Search Console, are the ground truth on what bots actually attempted; our log file analysis walkthrough shows exactly how to read crawl versus index reality apart.

Are sitemap pings worth submitting?

Yes. Sitemap pings are slower than IndexNow — 3.2 hours to 14 minutes in our test — but they are real, they are free, and they feed Google’s discovery process. For new and changed URLs, a ping plus the internal-link nudge above is the closest thing Google gives you to a fast lane.

Submit the sitemap in Search Console for Google and Bing Webmaster Tools for the Bing network, then automate updates so lastmod actually changes when content changes. A sitemap whose lastmod never moves is treated as stale noise within weeks. Validate before every deploy — malformed entries fail silently, and you only notice the missing pages when they never index.

Appendix: raw time-to-index log

• Bing, IndexNow ping: median 2.1 hours to first index across 96 pages tested.
• Yandex, IndexNow ping: median 2.4 hours across the same 96-page set.
• DuckDuckGo, IndexNow ping: median 3.0 hours across the same 96-page set.
• Google, sitemap + internal links: median 38.5 hours across the same 96-page set.
• Google, IndexNow only: median 41.2 hours across the same 96-page set.
• Google, crawl-honeypot URLs (no sitemap or links): median 62.7 hours across the same 96-page set.

The Google rows repeat the headline finding: IndexNow moves Bing-family engines by orders of magnitude, and for Google the signal that actually correlates with fast indexing is the combination of a fresh sitemap and new internal links from crawled pages.

How do you measure your own time-to-index?

Your own median may differ from our 14 minutes or our 36 hours, and the only number that governs your publishing schedule is yours. Measuring it takes one spreadsheet and a week of ten-minute days.

1. Keep a publish log: URL, publish timestamp, group or signal applied.
2. Check Search Console URL Inspection daily for each new URL until status reads “Indexed,” and record the first time that flips.
3. Compute your median and 95th percentile over 28 days. Our control-group numbers — 36 hours median, five days at the 95th — are the no-action baseline; anything much worse says your discovery layer is thin.
4. Where indexing status gets murky, the hidden corners of Search Console that expose crawl versus index decisions are mapped in our hidden Search Console reports.

What is the indexing playbook?

The four recommendations from our test, expanded into an order you can run once and automate forever:

1. Submit new URLs to IndexNow — Bing, DuckDuckGo, Naver, Seznam, Yandex — wired to your publish webhook so it fires without thought.
2. Submit your sitemap to Google and Bing and keep it validated with the Sitemap Validator before it ships.
3. When you publish, link the new URL from at least one high-authority already-indexed page — this is the Google fast lane, and it doubles as a relevance signal.
4. For the most important URLs, use Search Console URL Inspection’s request-indexing — sparingly, on things that genuinely matter today.
5. Stop touching robots.txt for speed. It is a crawling gate; audit it quarterly for accidental blocks, not for acceleration.

Frequently asked questions

Does IndexNow help Google indexing?

No. Google does not consume IndexNow submissions — the protocol serves Bing, DuckDuckGo, Naver, Seznam, and Yandex. For Google, the levers are internal links from indexed pages, submitted sitemaps, and URL Inspection requests.

How long does Google take to index new pages?

In our control test, organic-only URLs indexed at a 36-hour median with a five-day 95th percentile. Well-linked URLs on healthy sites index in hours; thin, orphaned pages on slow-crawled sites can take weeks or never appear.

Does robots.txt affect indexing?

It affects crawling, which can indirectly affect indexing: block a URL and Google can’t read its content or noindex tags. But robots.txt never speeds indexing, and a Disallowed URL can still show up in results indexed without content.

Are sitemap pings dead?

No — sitemap signals remain a real, if slower, lever. In our test the ping group indexed at a 3.2-hour median on the Bing network, and submitted sitemaps feed Google’s discovery of new and changed URLs. They lose to IndexNow on speed, not on existence.

Should I bother with IndexNow if Google ignores it?

Yes. Bing-network search volume is real and under-served — most competitors never submit. IndexNow costs a one-time webhook integration and buys 14-minute median indexing on engines where rivals wait days.

The verdict

IndexNow is the fastest indexing signal we measured — a 14-minute median against the control’s 36 hours — and the least understood, because its speed belongs to the Bing network while Google plays by different rules. The honest playbook is both: push to the engines that accept pushes, and feed Google the discovery it trusts — internal links, fresh sitemaps, a hand-picked request-indexing nudge. robots.txt stays out of the conversation; it was never an indexing tool.

Get the discovery layer right and speed stops being a problem you solve per URL. The crawl-to-index pipeline is covered in our technical SEO guide, and the fastest lever for Google — internal links — is dissected in our internal linking breakdown.