Bitsandbytes

bitsandbytes: Essential k-bit Quantization Library for LLM Training and Inference

Large language models have grown far beyond the memory capacity of consumer hardware. A 70-billion-parameter model requires 140 gigabytes of GPU …