Inference
- AI
AMD Acquires Taalas: Etching LLM Weights Into Silicon at 17,000 Tokens/Second
AMD is acquiring Taalas, the Toronto startup that etches model weights directly into silicon — its HC1 chip serves Llama 3.1 8B at 17,000 tokens/sec, 48-73x faster than …
- AI
Xorbits Inference: Scalable LLM Serving Platform
Deploying large language models in production is a fundamentally different challenge from training them. Training requires massive clusters and weeks...
- AI
SGLang Omni: Multimodal LLM Inference with SGLang
Multimodal AI — models that understand images, audio, and video alongside text — has moved from research novelty to production necessity. Document...
- AI
ik_llama.cpp: Fork of llama.cpp with IQ4_NL and Advanced Quantization
The ecosystem around llama.cpp has produced numerous forks, each exploring different optimization strategies for running LLMs efficiently on consumer...
- AI
Gemma.cpp: Google's Lightweight C++ Inference Engine for Gemma Models
The landscape of LLM inference has largely been shaped by two approaches: heavyweight frameworks like PyTorch with full GPU acceleration, or highly...
- AI
NVIDIA Stock Rises on AI Demand: What Chip Investors Should Watch
Geopolitics and AI Demand: What Truly Underpins NVIDIA’s Stock Resilience? A slight easing in geopolitical tensions might offer temporary relief for market …