A token is the basic unit of text into which a language model divides both input and output text—typically a part of a word, a whole word, a character, or a punctuation mark. Models process text in tokens, not whole words, and the cost and speed of most AI APIs are calculated based on the number of tokens processed and generated. Understanding tokenization is also important when estimating the costs of deploying AI in business processes.
Token
The AI Act for Board Members: What Every C-Level Executive Needs to Know to Keep the Company from Paying Millions
6 min read
AI Governance in the Workplace: How to Set Rules for AI Use Before Regulators Do It for You
5 min read