Robots.txt

Robots.txt is a text file located in the root directory of a website that tells search engine crawlers which parts of the site they should—or should not—crawl. This file does not directly prevent the indexing of a page that is blocked in another way, but it effectively controls crawlers’ access to individual sections of the website and conserves the so-called crawl budget.