Back
ALeeXSD @ALeeXSD
vLLM: High-Throughput and Memory-Efficient LLM Serving Engine
GitHubIndustry-standard LLM serving engine featuring PagedAttention, continuous batching, FP8 quantization, and multi-GPU tensor parallelism.
Also available in:RU
vLLM is an open-source library (over 36,000 GitHub stars) widely regarded as the industry standard for production LLM serving and inference throughput.
Key Technical Advantages
- PagedAttention Algorithm: manages Key-Value cache memory using operating system virtual memory paging concepts, virtually eliminating memory fragmentation and boosting throughput by 2–4x.
- Continuous Batching: dynamically schedules incoming requests into active forward passes without waiting for…
vllminferenceaipython
Versions
Versions and changelog are tracked in the GitHub repository.
License terms
Apache-2.0
Install command
pip install vllm
Version
0.7.2
Comments · 0
Nothing here yet
Sign in to participate