Byte pair encoding

Byte pair encoding is a tokenization method that starts from individual characters or bytes and repeatedly merges the most frequent adjacent pairs into new tokens, building a vocabulary of common subwords.

Byte pair encoding sits in the Prompting & context part of the Agentik {OS} glossary, which defines the words used to build and run AI agent systems.

Also called BPE.