Crawling & Indexing: How Googlebot Discovers and Indexes Pages

The first thing in Technical SEO isn't writing titles or stuffing keywords—it's confirming that search engines can smoothly discover, access, understand, and index your pages. Crawling and indexing are the gate before ranking: a page that isn't in the index has no chance to rank in organic search.

Crawling and Indexing Are Not the Same Thing#

Crawling is the process of a search engine crawler visiting a URL and downloading the page's resources. Indexing is the process of the search engine analyzing the page and deciding to add it to its index. A page can be crawled many times yet never indexed; or it may have been indexed once and later removed due to quality, duplication, or technical signal issues.

How Googlebot Discovers URLs#

Googlebot's main sources for discovering URLs include: internal links within the site, external links from other sites pointing to you, the XML sitemap, redirect targets, and historically known URLs. For new sites, the most reliable approach is to form clear links between the homepage, category pages, and article pages, then submit a sitemap.

If a page has no internal links and doesn't appear in the sitemap, it's close to an "orphan page." Search engines may take a long time to discover it, and even if they do, it's hard for them to judge its importance within the site structure.

Take this very article as an example: it gets a link entry from the Technical SEO category index page, a link from the "next article" navigation of the 30-day action checklist article, and it also appears in the sidebar navigation. These three discovery paths are completely independent, and none of them require JavaScript to follow. For Googlebot, multiple crawlable HTML links mean a higher probability of discovery and a clearer signal of in-site importance.

For more details on how Googlebot discovers and crawls URLs, see the official Google documentation: Googlebot Overview.

What Is Crawl Budget#

Crawl budget can be simply understood as: the crawling resources a search engine is willing to allocate to your site within a given time. It's not a publicly fixed number—it's influenced by factors like site size, server response, page update frequency, page quality, and link structure.

Small sites usually don't need to worry too much about crawl budget. The ones that easily run into trouble are large sites, e-commerce sites, content aggregation sites, and sites with lots of filtering parameters. For example, a shoe e-commerce site where color, size, brand, price, and sort options all generate crawlable URLs will see URL counts explode exponentially, and crawlers will waste a lot of time on low-value pages.

What Determines Whether a Page Gets Indexed#

Whether a page can be indexed depends on passing at least four checks: accessibility, index permission, valuable content, and clear canonical signals.

1. Accessible: The page returns a 200 status code and isn't a login page, error page, redirect loop, or blocked by the server.
2. Index permitted: No noindex, and no X-Robots-Tag: noindex set via response headers.
3. Valuable content: The page isn't empty, duplicate, auto-stitched, or nearly devoid of unique information.
4. Clear canonical signals: The canonical points to itself or the correct canonical version, not mistakenly to some other page.

<link rel="canonical" href="https://example.com/page.html">
<meta name="robots" content="index,follow">

JavaScript Rendering & Indexing Delay#

Google doesn't process web pages in one step. It uses a two-phase pipeline: in the first phase, Googlebot crawls the raw HTML and extracts links and text from it; in the second phase, Google's Web Rendering Service (WRS) uses a headless Chromium browser to execute JavaScript and render the full page.

The problem is the time gap between these two phases. The first phase completes almost in real time, but JavaScript rendering has to queue for WRS resources, and the delay can range from hours to weeks. This means: if your core content depends on JavaScript to be generated (such as routed content in a single-page application (SPA), or client-rendered list data), the discovery and indexing of that content will be delayed.

The solution is to make critical content visible in the raw HTML. Common approaches include:

• Server-Side Rendering (SSR): The server returns complete HTML in the response, and the client then takes over interactivity. Frameworks like Next.js and Nuxt.js support this pattern.
• Static Site Generation (SSG): Complete HTML files for all pages are generated at build time, requiring no server runtime after deployment.
• Pure static HTML: Write HTML files directly, not relying on any JavaScript framework to render content.

For more best practices on JavaScript and SEO, see the official Google documentation: JavaScript SEO Basics.

Blocking Crawling vs Blocking Indexing vs Blocking Access#

Many indexing problems come from mixing up several control mechanisms. robots.txt, noindex, password protection, and canonical solve different problems and can't substitute for one another.

Troubleshooting with Google Search Console#

Google Search Console is the core tool for troubleshooting crawling and indexing issues. Beginners most often use two features: the URL Inspection tool and the Page Indexing report.

1. Copy the target URL and paste it into the URL inspection bar at the top of GSC.
2. Check "URL is on Google" to determine whether it's not discovered, crawled-not-indexed, or indexed.
3. Check the user-declared canonical vs the Google-selected canonical to confirm whether the canonical URL is consistent.
4. Test the live URL to check whether the current page is accessible, crawlable, and free of noindex.
5. Request indexing after fixing, but don't treat it as a ranking guarantee.

The result page returned by the URL Inspection tool contains several key fields, and understanding them is crucial for accurately diagnosing problems:

The Page Indexing report is good for viewing site-wide trends: which URLs are excluded by noindex, which are duplicate, which are crawled but not indexed, which return 404. Don't look at just one URL—see whether the same type of problem appears in batches.

For detailed instructions, see the official Google Help: URL Inspection tool.

A Practical Troubleshooting Workflow#

When you find a page isn't indexed, troubleshoot in this order:

• The URL opens normally and returns 200
• No login wall, geo-block, or abnormal redirects
• robots.txt doesn't block the URL or key resources
• No meta robots noindex on the page
• No X-Robots-Tag: noindex in HTTP response headers
• canonical points to the current page or correct canonical
• The page appears in sitemap.xml
• The page has at least one crawlable internal link entry
• Content isn't duplicate, too thin, or auto-stitched
• GSC live test result is crawlable and indexable

If everything above is fine but the page still shows "Crawled - currently not indexed", you usually need to return to content quality and the site's overall value. Technical correctness is just the entry ticket; it doesn't mean the page is necessarily worth indexing.

Understanding Common GSC Statuses#

Advanced: Server Log File Analysis#

Google Search Console provides sampled data from Google's perspective, while server access logs record every real request. By analyzing logs, you can obtain more complete and precise crawl-behavior data than GSC.

Each request record in a server log usually contains: client IP, request time, requested URL, HTTP status code, response size, User-Agent string, etc. By filtering records whose User-Agent contains Googlebot, you can extract all Googlebot access behavior.

66.249.66.1 - - [23/Jun/2026:10:15:32 +0800] "GET /en/technical-seo/crawling-indexing.html HTTP/1.1" 200 18562 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

From the logs you can analyze these key metrics:

• Crawl frequency distribution: Which URLs Googlebot visits frequently, and which rarely or never. Frequently crawled pages are usually ones Google considers important or frequently updated.
• Status code distribution: How many 200, 301, 404, 500 responses Googlebot encountered. A large number of 4xx or 5xx indicates dead links or server problems.
• Response time anomalies: If some pages respond to Googlebot significantly slower than average, there may be a performance bottleneck affecting crawl efficiency.
• Crawl waste: Whether Googlebot is crawling lots of low-value pages (parameter URLs, pagination, search result pages) while ignoring important content pages.

Common log analysis tools include Screaming Frog Log File Analyser (paid, with a GUI) and custom scripts based on grep and awk. For large sites, log analysis is an important means of optimizing crawl budget allocation and uncovering technical issues.

Making Pages Easier to Discover and Index#

• Build a clear hierarchy: Homepage → category pages → article pages; don't bury important pages too deep.
• Keep internal links crawlable: Use real <a href> for important links; don't rely solely on complex script-driven clicks.
• Submit and maintain a sitemap: Include only indexable, canonical, 200-returning important URLs.
• Reduce duplicate pages: Have clear strategies for parameter pages, tag pages, and filter pages.
• Keep the server stable: Frequent 5xx errors, timeouts, and very slow responses reduce crawl efficiency.
• Continuously improve content quality: Unique experience, clear structure, factual evidence, and useful examples matter more than raw word count.

Frequently Asked Questions#

Why isn't my page indexed even though Google crawled it?

Crawling only means Googlebot visited the page. Indexing also depends on whether the page is indexable, whether it's duplicate, whether it has enough unique value, whether the canonical points to itself, and whether Google deems it worth adding to the index.

Can robots.txt stop a page from being indexed?

robots.txt mainly blocks crawling, not indexing. If a URL blocked by robots.txt is still referenced by external links, search engines may display that URL based on link information alone. To block indexing, use noindex—but a noindexed page must be crawlable to be seen.

Do small sites need to care about crawl budget?

Small sites with dozens to hundreds of pages usually don't need to over-focus on crawl budget. What really matters is clear internal links, stable status codes, avoiding orphan pages and low-quality duplicates. Large sites, e-commerce filter pages, and lots of parameter URLs are far more prone to crawl budget waste.

Can the site: operator tell if a page is indexed?

The site: operator works for a quick spot-check but isn't suitable as a precise indexing report. A more reliable method is the URL Inspection tool and Page Indexing report in Google Search Console.