Batching

Batching is the technique of grouping multiple inputs or requests and processing them together on hardware such as a GPU, improving throughput and cost efficiency at the price of some added latency.

Batching sits in the Infrastructure part of the Agentik {OS} glossary, which defines the words used to build and run AI agent systems.

Also called Request batching.