Semantic caching is a technique in which the system stores answers to queries and, for a new, semantically similar query, uses the stored answer instead of processing it anew. The reason is cost and speed – calling the model costs money and takes time, whereas returning a stored answer is nearly instant. For companies running their own assistant or chatbot, it's one of the most effective ways to cut costs, since a large share of customer questions repeat. It does, however, have a consequence to bear in mind: after information changes on the website or in the price list, the system may keep answering with old data for a while. That's why a validity period is set for stored answers, and caching is switched off for critical data such as prices and availability.
See also: Indexing delay in AI, Content freshness signal, How much does an AI solution cost.
