Tokenization

Tokenization is the process of splitting text into tokens and mapping each one to a numerical ID from a fixed vocabulary, so a language model can process it.

Tokenization sits in the Foundations part of the Agentik {OS} glossary, which defines the words used to build and run AI agent systems.

Also called Tokenizer.