llms.txt

llms.txt is a proposed standard for a text file on the web, similar to a robots.txt file, that provides AI models and their crawlers with a structured and clear summary of a page’s key content suitable for training or later citation. The goal of this standard is to make it easier for language models to quickly understand and process web content without having to crawl the entire site structure.