SentencePiece

SentencePiece is an open-source tokenization library that learns subword units directly from raw text, treating spaces as ordinary symbols, which makes it language independent and widely used in multilingual models.

SentencePiece sits in the Prompting & context part of the Agentik {OS} glossary, which defines the words used to build and run AI agent systems.