vLLM

What it does

vLLM is a library for running language-model inference and serving models. Its documentation directs people running open-source models, building applications, and developing vLLM to different guides. Developers can submit prompts and sampling settings for offline batched generation, or launch a server and send completion or chat requests; generated text is returned to the caller. The server supports an OpenAI-compatible API, with documented support also for Anthropic Messages API and gRPC.

The library includes continuous batching, streaming, structured outputs, tool calling, and distributed inference. It is installed and operated on user-managed compute rather than accessed as a hosted web application. The quickstart lists Linux and Python 3.11–3.14; Apple Silicon is documented through the separate vLLM-Metal path. Hardware and installation instructions depend on the backend. The offline `generate` method does not automatically apply a chat template, and the quickstart server hosts one model at a time by default.

What it can do

• Offline batched inference from prompts
• Online completion and chat serving
• OpenAI-compatible API server
• Continuous batching and streaming outputs
• Structured output generation
• Tool calling and distributed inference

Works with

• Hugging Face models
• OpenAI-compatible API clients
• Anthropic Messages API
• gRPC

Before you choose

• Installation and supported configurations vary by backend; some hardware paths use separate plugins or instructions.
• Offline `generate` does not apply a chat template automatically, so chat or instruction models may need template handling.
• The quickstart server hosts one model at a time by default; compute and model hosting are user-managed.

Listed under

• Coding
• Sandboxes and infrastructure
• Developers
• Developer tools
• GPU inference
• LLM inference
• LLM serving
• Model serving

Get started

The guide documents package installation followed by offline inference or server startup.

1. Check the quickstart prerequisites and choose the instructions for your hardware backend.
2. Install vLLM in a Python environment using the applicable package or backend-specific route.
3. Run the offline inference example or start `vllm serve` with a model and send requests to the server.

Open-source serving software; model licenses, hardware and hosting costs are separate.

Source: github.com/vllm-project/vllm, checked 1 Nov 2026. Prices can change; check before you buy.

Sources for this overview

Product pages, documentation, and pricing checked 1 Nov 2026. Each source supports part of the overview; availability and prices can change.

• vLLM ↗docs.vllm.ai
• Quickstart - vLLM ↗docs.vllm.ai
• GitHub - vllm-project/vllm: A high-throughput and memory-efficient inference and serving engine for LLMs · GitHub ↗github.com

Company website

docs.vllm.ai: product, pricing and policy pages

YC directory mirror

Batch and company name

Human review

Checked by a person

knowledge_base

manual_review

Last checked 1 Nov 2026. Every field keeps its source and date; unknown means we couldn’t confirm it.