LLM Inference on SoloSoft

ExLlamaV3: High-Performance LLM Inference Engine

Fri, 01 May 2026 00:00:00 +0000

Running large language models on consumer hardware requires efficient inference engines that squeeze every drop of performance from available GPU memory. ExLlamaV3, developed by the turboderp team, is one of the fastest inference engines available for Llama-family models, particularly when using the EXL3 quantization format.

ExLlamaV3 achieves its speed through a combination of optimized CUDA kernels, efficient memory management, and quantization-aware computation. It supports both 4-bit and 8-bit EXL3 quantization, dynamic batching, and speculative decoding. For users running local models on consumer GPUs, it consistently delivers the highest tokens-per-second throughput available.

Performance Benchmarks

Model	GPU	Quantization	Speed (tokens/s)	Memory Usage
Llama 3.1 8B	RTX 4090 24GB	EXL3 4-bit	180	6 GB
Llama 3.1 70B	RTX 4090 24GB	EXL3 4-bit	30	22 GB
Mistral 7B	RTX 3060 12GB	EXL3 4-bit	85	5 GB
Qwen 2.5 32B	RTX 4090 24GB	EXL3 4-bit	55	18 GB

Key Features

Feature	Description	Benefit
EXL3 quantization	Specialized 4-bit and 8-bit formats	Highest quality per bit
CUDA kernel optimization	Fused attention, flash decoding	Maximum throughput
Dynamic batching	Process multiple requests concurrently	Higher utilization
Speculative decoding	Draft-then-verify for faster generation	2x speedup on some tasks
LoRA support	Load and swap LoRA adapters at runtime	Flexible fine-tuning

Inference Pipeline

flowchart LR
 A[Input Tokens] --> B[Embedding Layer]
 B --> C[Transformer Layer 1]
 C --> D[Layer 2]
 D --> E[Layer N]
 E --> F[Attention with<br/>FlashAttention]
 F --> G[Feed-Forward<br/>with Quantized GEMM]
 G --> H{More Layers?}
 H -->|Yes| D
 H -->|No| I[Output Logits]
 I --> J[Sampling]
 J --> K[Generated Token]
 K --> L[KV Cache Update]
 L --> C

The pipeline processes tokens through transformer layers with specialized CUDA kernels for attention and feed-forward computation. The KV cache is maintained efficiently in GPU memory, and speculative decoding can accelerate generation by validating multiple tokens at once.

KTransformers: Flexible LLM Inference with Advanced Kernel Optimization

Fri, 01 May 2026 00:00:00 +0000

The efficiency of LLM inference directly determines the cost, latency, and scalability of AI applications. KTransformers (kvcache-ai/ktransformers on GitHub) is a flexible inference framework that pushes the boundaries of what is achievable with kernel-level optimizations, enabling faster and more cost-effective deployment of large language models in production environments.

Developed by the kvcache-ai team, KTransformers takes a comprehensive approach to inference optimization. Rather than focusing on a single technique, it combines multiple strategies – advanced CUDA kernels, dynamic batching, speculative decoding, quantization, and attention optimizations – into a unified framework that can be tuned for different deployment scenarios.

The framework’s architecture is designed for flexibility. Users can configure which optimizations to apply based on their specific hardware, model characteristics, and performance requirements. This makes KTransformers suitable for a wide range of deployments, from single-GPU local inference to distributed multi-GPU production systems serving thousands of concurrent requests.

llama.cpp: High-Performance LLM Inference on CPU and GPU

Fri, 01 May 2026 00:00:00 +0000

The dream of running powerful language models entirely on your own hardware, without sending data to cloud APIs, was once considered impractical for anyone outside of large tech companies. llama.cpp shattered that assumption. This single-header C++ implementation has become the most popular tool for running LLMs locally, democratizing access to AI computation across virtually every hardware configuration.

Created by Georgi Gerganov, llama.cpp started as a focused implementation of Meta’s Llama architecture and has since grown into a universal inference engine supporting hundreds of model architectures, multiple backends (CPU, CUDA, Metal, ROCm, Vulkan), and a rich ecosystem of tools and integrations.

LocalAI: Self-Hosted OpenAI API-Compatible Inference Server

Fri, 01 May 2026 00:00:00 +0000

Running AI models locally offers undeniable advantages: complete data privacy, no API costs, offline operation, and full control over model choice and configuration. But replacing cloud AI services with local alternatives typically requires a patchwork of different tools – one for LLMs, another for image generation, a third for speech recognition. LocalAI solves this fragmentation by providing a single, OpenAI API-compatible server that covers the full spectrum of AI capabilities.

LocalAI is a drop-in replacement for OpenAI’s API that runs entirely on your own hardware. Any application that works with OpenAI’s API – from simple chat interfaces to complex agent frameworks – can be redirected to LocalAI by changing a single configuration parameter: the API base URL.

MLX LM: LLM Inference and Fine-Tuning on Apple Silicon

Fri, 01 May 2026 00:00:00 +0000

The promise of running LLMs locally on a MacBook has been seductive but incomplete. Ollama and llama.cpp made it possible, but performance left room for improvement — models ran, but they did not fully leverage Apple Silicon’s architecture. The gap between what a MacBook could theoretically do and what inference engines delivered was visible in every benchmark.

MLX LM closes this gap. Built on Apple’s own MLX framework, it runs LLM inference and fine-tuning at speeds that previously required dedicated GPU hardware. The key is MLX’s unified memory architecture — no data copying between CPU and GPU, no PCI-e bottlenecks, just direct access to the full memory bandwidth of Apple Silicon. For a MacBook Pro with an M4 Max, MLX LM delivers inference performance that rivals mid-range NVIDIA GPUs.

PowerInfer: High-Speed LLM Inference on Consumer GPUs via CPU-GPU Hybrid Design

Fri, 01 May 2026 00:00:00 +0000

Running large language models locally has always been constrained by a hard wall: GPU memory. A 175-billion parameter model in FP16 requires approximately 350GB of VRAM – far beyond the 24GB available on consumer GPUs like the RTX 4090. Server-grade solutions exist (A100, H100), but they cost tens of thousands of dollars. PowerInfer, developed by Tiiny-AI (formerly from Shanghai Jiao Tong University), smashes through this wall with a clever insight that exploits a fundamental property of how neural networks actually compute.

The insight is called activation locality: for any given input token, only a small fraction of a model’s neurons are active. The rest are essentially idling. PowerInfer exploits this by pre-analyzing the model to identify which neurons are “hot” (frequently activated) and which are “cold” (rarely activated). Hot neurons are kept on the GPU for fast access; cold neurons remain in CPU memory and are only loaded when needed.

SGLang: Efficient LLM Inference with Structured Generation

Fri, 01 May 2026 00:00:00 +0000

The open-source LLM ecosystem has solved many problems — model quality, fine-tuning, deployment — but one challenge persists: getting models to produce reliable, structured output. A model asked to output JSON might add explanatory text, use inconsistent key names, or fail to close brackets. For production systems that feed LLM output into downstream APIs, databases, or parsers, this unpredictability is a blocker.

SGLang approaches this problem from the inference engine level rather than the prompting layer. It is a high-performance LLM inference framework that builds structured generation into the core inference pipeline. Instead of asking the model nicely to output JSON and hoping for the best, SGLang constrains the token generation process so that every token is guaranteed to conform to a specified grammar, schema, or pattern.

TensorRT-LLM: NVIDIA's Open-Source Library for Optimized LLM Inference

Fri, 01 May 2026 00:00:00 +0000

Deploying large language models in production requires more than just loading weights onto a GPU. To achieve acceptable throughput and latency, you need kernel fusion, attention optimization, memory management, and quantization – all tuned for your specific hardware. NVIDIA’s TensorRT-LLM provides all of this in a single open-source library that extracts maximum performance from NVIDIA GPUs for LLM and visual generation inference.

TensorRT-LLM, hosted at github.com/NVIDIA/TensorRT-LLM, is NVIDIA’s official inference optimization library for large language models and visual generative models. It includes state-of-the-art kernel implementations for attention (FlashAttention, PageAttention), quantization (FP8, INT4, INT8, INT4-AWQ), and in-flight batching. The library compiles models into optimized engine files that run efficiently across NVIDIA’s GPU lineup from Turing to Blackwell architectures.

vLLM: High-Throughput LLM Inference with PagedAttention

Fri, 01 May 2026 00:00:00 +0000

Serving LLMs in production is fundamentally a memory management problem. The KV cache — the set of attention key-value pairs stored during generation — grows with each token produced. For a 70B parameter model serving multiple concurrent requests, the KV cache consumes hundreds of megabytes per sequence. Poor memory management means wasted GPU memory, lower throughput, and higher cost per token.

vLLM solves this with PagedAttention, a breakthrough that applies operating system virtual memory concepts to LLM inference. By managing the KV cache in fixed-size blocks (pages) rather than contiguous memory regions, vLLM eliminates fragmentation — the dominant memory waste in naive inference — and achieves near-perfect memory utilization. The result is 2-4x higher throughput than any previous open-source inference engine.