wtf is oscar-kv-quant?

OScaR is a research project from teams at Tsinghua, Hong Kong University, Meituan's LongCat team, and Edinburgh that tries to shrink the memory cost of running large language models without hurting their accuracy. When a large language model reads a long input or holds a long conversation, it stores intermediate values called the KV cache. That cache grows with the length of the input and can dominate memory use. The standard trick is quantization, which stores those values in fewer bits, like 2 bits per number instead of 16. The trade off is that aggressive quantization usually loses accuracy. The paper, posted on arXiv on May 20, 2026, identifies what the authors see as the real reason 2 bit quantization tends to fail. They call it Token Norm Imbalance, or TNI: across the tokens in a sequence, some sit far away from the others in size, and the usual per channel quantization scheme cannot represent both groups well at once. In plain language, a few outlier tokens drag down the precision of the rest. The authors show that TNI shows up in text only models, in multi modal models, and in omni modal models that handle audio and images alongside text. OScaR, short for Omni-Scaled Canalized Rotation, is the proposed fix. It is described as following Occam's razor: it does only two things, a Canalized Rotation step and an Omni-Token Scaling step. It does not need extra training data and does not need a calibration pass on real samples. The reported results put 2 bit OScaR roughly even with the original 16 bit baseline on LongBench-E, OCRBench, and the MMAU-Pro audio benchmark, across models like Llama 3.1 8B, Qwen3 8B, Qwen3-VL, and Qwen3-Omni 30B. Implementation wise the authors built custom CUDA kernels on top of BitDecoding and HadaCore, and report 3.0 times faster decoding, 5.3 times less memory, and 4.1 times higher throughput against a BF16 FlashDecoding-v2 baseline. The repository ships the code, an evaluation suite, and an installation flow built around the uv package manager, Python 3.10, PyTorch 2.6 with CUDA 12.4, and flash-attn 2.8.3, plus a CUTLASS submodule. Integration with the vLLM and SGLang serving frameworks is listed as upcoming.