LLM Compressor

What it does

LLM Compressor takes a Hugging Face-compatible language model, plus an optional calibration dataset, and applies post-training compression. It outputs a smaller or faster checkpoint saved in the compressed-tensors format. Compression options include weight, activation, KV cache and attention quantization, along with pruning such as 2:4 sparsity and REAP expert pruning.

You write a short Python recipe that picks an algorithm (for example GPTQ, AWQ, SmoothQuant or simple round-to-nearest) and a scheme (such as W8A8, FP8, INT4 or NVFP4), then call the oneshot function. The saved checkpoint loads directly in vLLM for serving. Its distinguishing feature is support for activation quantization in vLLM alongside weight-only methods, in one library rather than several separate tools. It also handles very large models through sequential onloading, disk offloading and distributed data parallel runs.

It suits ML and inference engineers who deploy open models with vLLM and want lower memory use or higher throughput. You need Python, a pip install and suitable hardware; the announcement ties its speedups mainly to NVIDIA GPUs, with other hardware still on the roadmap. Accuracy should be checked per model, since results vary by scheme and calibration data. It is a library, not a hosted service.

What it can do

• Weight and activation quantization (W8A8 int8/fp8, microscale NVFP4/MXFP4)
• Weight-only quantization (W4A16, W8A16) with GPTQ, AWQ and AutoRound
• KV cache and attention quantization
• Pruning including SparseGPT 2:4 sparsity and REAP expert pruning
• Compression of very large models using sequential onloading, disk offloading and DDP
• Saves compressed-tensors checkpoints that vLLM loads directly

Works with

• vLLM
• Hugging Face models and repositories

Example workflows

Quantize a model to FP8 for serving

Input: a Hugging Face model ID. Steps: install llmcompressor, define a QuantizationModifier recipe with an FP8 scheme, run oneshot, save the result. Output: a compressed-tensors checkpoint ready to load in vLLM.

Compress a model with a custom calibration set

Input: an INT8 model target and your own calibration dataset. Steps: pair SmoothQuant with GPTQ in a recipe, run oneshot with the dataset, then load the output in vLLM. Output: a W8A8 checkpoint.

Before you choose

• Speed gains depend on scheme and GPU generation; weight-only methods may give little benefit under compute-bound serving load.
• Support for non-NVIDIA hardware was listed as roadmap work in the announcement.
• Quantized accuracy should be evaluated on your own tasks.

Listed under

• Coding
• Sandboxes and infrastructure
• Developers
• Inference optimization
• LLM inference
• Machine learning
• Machine learning engineers
• Model compression

Get started

Then run the oneshot API on a model

1. Run pip install llmcompressor in a Python environment with GPU access.
2. Choose a model and a recipe from the repository examples matching your target scheme.
3. Run oneshot, save the checkpoint, then install vllm and load it to test.

The library is open source under Apache-2.0. GPU compute for compression is not included. The enterprise nm-vllm distribution is mentioned without prices.

Source: github.com/vllm-project/llm-compressor, checked 29 Oct 2026. Prices can change; check before you buy.

Sources for this overview

Product pages, documentation, and pricing checked 29 Oct 2026. Each source supports part of the overview; availability and prices can change.

• LLM Compressor is here: Faster inference with vLLM | Red Hat Developer ↗developers.redhat.com
• GitHub - vllm-project/llm-compressor: State-of-the-art LLM compression, built for production inference with vLLM · GitHub ↗github.com

Company website

docs.vllm.ai: product, pricing and policy pages

YC directory mirror

Batch and company name

Web search

Confirms the official site

Social accounts

Accounts the company links, and its posts on X

Human review

Checked by a person

knowledge_base

Last checked 29 Oct 2026. Every field keeps its source and date; unknown means we couldn’t confirm it.