Content chunking

Content chunking is the division of a document into smaller blocks of text that are indexed, retrieved and inserted into the model's context separately. A generative search engine almost never works with an entire page – it picks out one or two blocks that best match the specific query. A practical rule follows from this: the unit of optimisation stops being the page and becomes the paragraph. A block must make sense even when pulled out of context, meaning it should contain its own subject instead of a pronoun, a repeated entity name, and a complete claim including any number or condition. It helps to have a short heading above each block, one idea per paragraph, and to avoid sentences that refer back to "the above". Content written as continuous narrative falls apart under chunking, and the individual blocks lose their informative value.

See also: Passage indexing, Grounding, Query fan-out.