vLLM

vLLM is an open-source library for fast, memory-efficient serving of large language models, known for PagedAttention and continuous batching, and it can expose an OpenAI-compatible API.

vLLM sits in the Infrastructure part of the Agentik {OS} glossary, which defines the words used to build and run AI agent systems.