ResidualQuant
KV Cache Quantization for Looped Transformers with 2-Bit Residuals
1 KAIST · 2 Yonsei University · 3 Seoul National University
† Corresponding author
Paper Figure 1 · Ouro-1.4B · MATH500 · group size g=32 · Throughput: RTX 5090, 16k context, 128 output tokens, decode only.
Looped Transformers share weights across depth, but not their KV caches. ResidualQuant compresses the loop-specific information instead of repeatedly storing nearly redundant full-precision states.
ResidualQuant
Looped Transformers repeatedly apply shared modules to extend computational depth without increasing parameter count. Each loop, however, produces a distinct KV cache, so memory still grows with recurrent depth, context length, and batch size. This limits the larger batches and higher throughput that weight sharing should enable.
ResidualQuant exploits the strong similarity between KV states across loops. It stores the final loop as a shared INT4 anchor, predicts earlier loops with least-squares scaling, and encodes only the remaining residuals in INT2. Shared rotations and loop-wise precision allocation preserve loop-specific information without a separate BF16 region for past KV states.
Inter-loop redundancy
Repeated application of the same projection produces closely related KV patterns across loops. After least-squares prediction, the residual has fewer large-magnitude outliers and is substantially more robust to aggressive low-bit quantization.

Share the pattern. Store the difference.
Four loops. Nearly the same pattern.
Look at the repeated blue tiles. Orange marks are the small differences specific to each loop.
Actual KV examples are loading in the background…
Decode throughput
Reducing KV memory traffic accelerates fixed-batch decoding, while the smaller cache footprint also admits larger batches. On an RTX 5090, ResidualQuant improves fixed-batch decode throughput by up to 3.66× (16k context, batch 1) and, by supporting 4–8× larger tested batches, raises peak decode throughput by up to 5.38× over BF16. In the single example run recorded below at 16k context, ResidualQuant achieved 5.34× the aggregate decode throughput of vLLM BF16, with batch 8 versus batch 2.
2k context
4k context
8k context
16k context
Ouro-1.4B · RTX 5090 · group size g=32 · 128 output tokens · decode only · one warm-up, three measured trials. Independent axis ranges; batch capacity means the largest feasible tested power-of-two batch.
The accuracy–memory tradeoff
Across Ouro-1.4B, Ouro-2.6B, and Huginn-3.5B, ResidualQuant preserves accuracy across math and code benchmarks with low-bit KV caches. Under mixed INT2/4 with g = 32, its twelve-score average nearly matches BF16 (62.30 vs. 62.58) while reducing logical KV storage by 80.7%. On Ouro-1.4B, it reaches 76.0% MATH500 accuracy compared with 75.0% for BF16.
| Method | Ouro-1.4B 4 loops | Ouro-2.6B 4 loops | Huginn-3.5B 32 loops | Avg. | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| GSM8K | MATH500 | HumanEval | MBPP | GSM8K | MATH500 | HumanEval | MBPP | GSM8K | MATH500 | HumanEval | MBPP | ||
| BF16 | 79.23 | 75.00 | 71.95 | 74.34 | 86.58 | 79.80 | 79.27 | 81.75 | 43.14 | 16.40 | 23.78 | 39.68 | 62.58 |
| Uniform INT2g = 32 · beff = 2.5 | |||||||||||||
| Direct | 33.66 | 27.80 | 41.46 | 55.56 | 68.54 | 62.40 | 65.85 | 71.43 | 18.95 | 7.80 | 20.73 | 33.60 | 42.32 |
| OptR-H | 62.70 | 55.40 | 59.15 | 70.90 | 81.50 | 72.20 | 72.56 | 73.28 | 28.58 | 10.20 | 22.56 | 36.51 | 53.80 |
| Residual quantization | 64.67 | 60.60 | 55.49 | 66.40 | 79.53 | 73.60 | 78.66 | 76.46 | 34.95 | 9.80 | 24.39 | 36.24 | 55.07 |
| Residual quantization + OptR-H | 73.31 | 71.60 | 68.90 | 71.69 | 83.85 | 75.20 | 77.44 | 80.16 | 38.21 | 12.60 | 23.78 | 38.89 | 59.64 |
| Uniform INT2g = 16 · beff = 3.0 | |||||||||||||
| Direct | 55.95 | 54.20 | 62.20 | 66.40 | 78.77 | 71.40 | 75.61 | 74.34 | 24.94 | 7.00 | 21.95 | 31.22 | 52.00 |
| OptR-H | 65.96 | 64.20 | 70.73 | 70.90 | 83.40 | 75.00 | 72.56 | 76.72 | 31.61 | 10.80 | 21.34 | 40.74 | 57.00 |
| Residual quantization | 70.96 | 67.80 | 69.51 | 70.37 | 84.15 | 76.60 | 77.44 | 77.51 | 36.77 | 14.40 | 25.61 | 37.57 | 59.06 |
| Residual quantization + OptR-H | 75.06 | 75.20 | 70.12 | 71.16 | 85.22 | 79.00 | 75.61 | 81.22 | 38.97 | 14.40 | 26.22 | 39.42 | 60.97 |
| Mixed INT2/4g = 32 · beff = 3.0 | |||||||||||||
| Direct | 48.67 | 47.40 | 56.71 | 67.99 | 76.27 | 73.20 | 70.12 | 78.04 | 30.10 | 9.80 | 21.95 | 34.39 | 51.22 |
| OptR-H | 65.73 | 63.00 | 68.29 | 70.63 | 84.76 | 78.20 | 78.66 | 78.31 | 36.39 | 11.00 | 23.17 | 39.15 | 58.11 |
| Residual quantization | 77.79 | 73.80 | 71.34 | 73.02 | 86.66 | 80.20 | 79.88 | 81.75 | 42.46 | 14.60 | 23.17 | 40.74 | 62.12 |
| Residual quantization + OptR-H | 77.03 | 76.00 | 70.73 | 73.81 | 86.35 | 79.60 | 80.49 | 82.28 | 42.76 | 14.60 | 23.78 | 40.21 | 62.30 |
| Mixed INT2/4g = 16 · beff = 3.4 | |||||||||||||
| Direct | 63.15 | 61.80 | 66.46 | 72.75 | 81.96 | 76.60 | 73.78 | 79.63 | 32.83 | 11.60 | 23.17 | 36.24 | 56.66 |
| OptR-H | 69.98 | 67.00 | 73.78 | 71.16 | 86.05 | 78.60 | 77.44 | 80.16 | 36.24 | 10.20 | 21.34 | 37.83 | 59.15 |
| Residual quantization | 77.03 | 75.00 | 71.95 | 74.07 | 85.82 | 80.40 | 79.27 | 80.95 | 41.02 | 15.40 | 23.17 | 41.53 | 62.13 |
| Residual quantization + OptR-H | 77.63 | 74.60 | 72.56 | 73.28 | 87.41 | 80.40 | 79.27 | 82.01 | 42.61 | 15.60 | 22.56 | 39.42 | 62.28 |
Values and bold highlights reproduce the paper’s main table. Teal rows are ResidualQuant (residual quantization + OptR-H). Residual variants use the final-loop anchor and least-squares scaling. beff = effective bits per KV value (Direct and OptR-H); mixed INT2/4 uses g = 32 for INT4. Avg. is the unweighted mean of the twelve model–benchmark scores. Huginn-3.5B repeats each four-loop schedule eight times.
Citation
@article{kim2026residualquant,
title = {ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals},
author = {Kim, Heejun and Lee, Junyoung and Cho, SangLyul and Han, Dongsu and Han, Insu and Kim, Sehoon},
year = {2026},
eprint = {2610.10381},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2610.10381}
}