Continuous batching

Continuous batching is a serving technique that adds new requests to a running batch as soon as others finish, instead of waiting for the whole batch to complete, improving GPU utilization for language models.

Continuous batching sits in the Infrastructure part of the Agentik {OS} glossary, which defines the words used to build and run AI agent systems.

Also called In-flight batching.