An AI Crawler is an automated bot that downloads website content for the needs of language models. It is not a single category – distinguish between training bots collecting data to train a model, search bots building an index for generative answers, and bots that fetch a page only at the moment a user asks about it. The best known are GPTBot and OAI-SearchBot from OpenAI, ClaudeBot from Anthropic, PerplexityBot, and the Google-Extended token, which governs the use of content for Google's generative features. They obey the robots.txt file, where they can be allowed or blocked individually. This granularity is at the heart of the decision: by blocking training bots you protect your content from being used to train a model, but by blocking search bots you also close off your path into AI answers and lose out on both visibility and traffic.
See also: Grounding, Passage indexing, Share of voice in AI responses.
