Inference

Inference is the phase in which a trained and finalized model processes new input and, based on that input, generates an output—such as a response to a prompt—unlike training, during which the model is still learning from the data. The speed and cost of inference are key to the real-world commercial deployment of AI products, since inference occurs every time an end user uses the model.