ExLlamaV3: High-Performance LLM Inference Engine

Fri, 01 May 2026 00:00:00 +0000

Running large language models on consumer hardware requires efficient inference engines that squeeze every drop of performance from available GPU memory. ExLlamaV3, developed by the turboderp team, is one of the fastest inference engines available for Llama-family models, particularly when using the EXL3 quantization format.

ExLlamaV3 achieves its speed through a combination of optimized CUDA kernels, efficient memory management, and quantization-aware computation. It supports both 4-bit and 8-bit EXL3 quantization, dynamic batching, and speculative decoding. For users running local models on consumer GPUs, it consistently delivers the highest tokens-per-second throughput available.

Performance Benchmarks

Model	GPU	Quantization	Speed (tokens/s)	Memory Usage
Llama 3.1 8B	RTX 4090 24GB	EXL3 4-bit	180	6 GB
Llama 3.1 70B	RTX 4090 24GB	EXL3 4-bit	30	22 GB
Mistral 7B	RTX 3060 12GB	EXL3 4-bit	85	5 GB
Qwen 2.5 32B	RTX 4090 24GB	EXL3 4-bit	55	18 GB

Key Features

Feature	Description	Benefit
EXL3 quantization	Specialized 4-bit and 8-bit formats	Highest quality per bit
CUDA kernel optimization	Fused attention, flash decoding	Maximum throughput
Dynamic batching	Process multiple requests concurrently	Higher utilization
Speculative decoding	Draft-then-verify for faster generation	2x speedup on some tasks
LoRA support	Load and swap LoRA adapters at runtime	Flexible fine-tuning

Inference Pipeline

flowchart LR
 A[Input Tokens] --> B[Embedding Layer]
 B --> C[Transformer Layer 1]
 C --> D[Layer 2]
 D --> E[Layer N]
 E --> F[Attention with<br/>FlashAttention]
 F --> G[Feed-Forward<br/>with Quantized GEMM]
 G --> H{More Layers?}
 H -->|Yes| D
 H -->|No| I[Output Logits]
 I --> J[Sampling]
 J --> K[Generated Token]
 K --> L[KV Cache Update]
 L --> C

The pipeline processes tokens through transformer layers with specialized CUDA kernels for attention and feed-forward computation. The KV cache is maintained efficiently in GPU memory, and speculative decoding can accelerate generation by validating multiple tokens at once.

EXL3 on SoloSoft

ExLlamaV3: High-Performance LLM Inference Engine

Performance Benchmarks

Key Features

Inference Pipeline