Crawling versus indexing

Crawling and indexing are two separate steps that are commonly confused. Crawling means a bot visits the page and downloads its content. Indexing is the search engine's subsequent decision on whether to add the page to its database and display it in the results. This distinction has a practical consequence for troubleshooting: if a page is not in the results, you first need to find out at which step it stopped. An uncrawled page tends to be blocked in the robots.txt file, missing a link, or affected by a server error. A crawled but unindexed page signals something else – insufficient content value, duplication with another page, or a canonical link pointing elsewhere. Both states can be verified in webmaster tools, which also give the reason.

See also: Index coverage report, Robots.txt, Canonical link.