wtf is vllm?

vLLM is a tool for running large language models (the technology behind text generators like ChatGPT) quickly and efficiently. Instead of making you build the infrastructure to serve these models yourself, vLLM handles the heavy lifting so that requests come back fast and the computer's memory is used well. It was originally developed at UC Berkeley's Sky Computing Lab and has since grown into a large open-source project with contributions from thousands of people across universities and companies. The main reason people use vLLM is speed. It achieves high throughput through a technique called PagedAttention, which manages the memory used by the model in a smart way. It also groups incoming requests together so they can be processed at the same time, and it supports several methods for compressing models into smaller representations so they run faster. The project supports a wide range of hardware, including NVIDIA GPUs, AMD GPUs, various CPUs, Google TPUs, Apple Silicon, and others. vLLM works with over 200 model architectures from Hugging Face, a popular platform for sharing models. This includes standard text generation models, mixture-of-experts models that route work to specialized subcomponents, hybrid models that combine different internal approaches, multi-modal models that can handle images alongside text, and embedding models used for search and retrieval tasks. On the usability side, vLLM provides an API server compatible with OpenAI's API, meaning existing applications designed for OpenAI can be pointed at a vLLM server instead. It supports streaming outputs, structured output generation, distributed inference across multiple machines, and multi-LoRA support, which lets you serve multiple fine-tuned variants of a model from a single deployment. Installation is straightforward: you install it with a single command using either uv or pip. The project documentation covers installation, a quickstart guide, and the full list of supported models. The project is actively maintained and welcomes contributions from the community.