ResidualQuant

KV Cache Quantization for Looped Transformers with 2-Bit Residuals

1 KAIST · 2 Yonsei University · 3 Seoul National University

† Corresponding author

Building ResidualQuant 76.0% accuracy
Accuracy ↑ · KV storage ↓00252550507575100100Accuracy (%)KV storage (% of BF16)100.075.0%15.6127.8%15.6257.2%16.2360.6%16.2471.6%19.35ResidualQuant76.0%BF16INT2+NaiveResidual+LSScaling+Rotation+MixedPrecisionBF16100%ResidualQuant19.3%80.7% less KV storage050100150200Decode throughput · 16k contextBF16 · Batch capacity: 2ResidualQuant1248Batch sizeTokens/s5.38× peak throughput4× larger tested batchREQUESTS DECODING IN PARALLELBF16 · 2ResidualQuant · 8

Paper Figure 1 · Ouro-1.4B · MATH500 · group size g=32 · Throughput: RTX 5090, 16k context, 128 output tokens, decode only.

ResidualQuant recovers BF16-level accuracy with 80.7% lower logical KV storage and reaches 5.38× peak decode throughput at 16k context.

Looped Transformers share weights across depth, but not their KV caches. ResidualQuant compresses the loop-specific information instead of repeatedly storing nearly redundant full-precision states.

ResidualQuant

Looped Transformers repeatedly apply shared modules to extend computational depth without increasing parameter count. Each loop, however, produces a distinct KV cache, so memory still grows with recurrent depth, context length, and batch size. This limits the larger batches and higher throughput that weight sharing should enable.

ResidualQuant exploits the strong similarity between KV states across loops. It stores the final loop as a shared INT4 anchor, predicts earlier loops with least-squares scaling, and encodes only the remaining residuals in INT2. Shared rotations and loop-wise precision allocation preserve loop-specific information without a separate BF16 region for past KV states.

Inter-loop redundancy

Repeated application of the same projection produces closely related KV patterns across loops. After least-squares prediction, the residual has fewer large-magnitude outliers and is substantially more robust to aggressive low-bit quantization.

Key magnitudes from two loops and their smaller prediction residual
Post-RoPE key magnitudes for two loops of Ouro-1.4B and the least-squares prediction residual, shown on the same scale.

Share the pattern. Store the difference.

THE MECHANISM · WATCH IT UNFOLD
01 / 06

Four loops. Nearly the same pattern.

Look at the repeated blue tiles. Orange marks are the small differences specific to each loop.

Loop 1Shared pattern + difference
Loop 2Shared pattern + difference
Loop 3Shared pattern + difference
Loop 4Shared pattern + difference
Illustrative tiles · mark size shows value magnitude, not bytes

Actual KV examples are loading in the background…

Decode throughput

Reducing KV memory traffic accelerates fixed-batch decoding, while the smaller cache footprint also admits larger batches. On an RTX 5090, ResidualQuant improves fixed-batch decode throughput by up to 3.66× (16k context, batch 1) and, by supporting 4–8× larger tested batches, raises peak decode throughput by up to 5.38× over BF16. In the single example run recorded below at 16k context, ResidualQuant achieved 5.34× the aggregate decode throughput of vLLM BF16, with batch 8 versus batch 2.

Ouro-1.4B · RTX 5090 · 16,384 input / 512 output tokens per request. BF16 batch 2: 35.57 tokens/s · ResidualQuant batch 8: 190.03 tokens/s. Actual xterm windows were recorded during inference and joined at 1× speed; both methods ran separately on the same GPU with CUDA Graph enabled. ResidualQuant uses prefill CPU offloading; prefill is excluded from decode timing. The title card, live counters, and result card are rendered from the recorded token timestamps.
ResidualQuantBF16

2k context

3.68×peak throughput
0375750112515001248163264Batch sizeTokens/s
8× batch capacityBF16 8 → ResidualQuant 64

4k context

2.73×peak throughput
020040060080012481632Batch sizeTokens/s
4× batch capacityBF16 8 → ResidualQuant 32

8k context

3.35×peak throughput
0100200300400124816Batch sizeTokens/s
4× batch capacityBF16 4 → ResidualQuant 16

16k context

5.38×peak throughput
0501001502001248Batch sizeTokens/s
4× batch capacityBF16 2 → ResidualQuant 8

Ouro-1.4B · RTX 5090 · group size g=32 · 128 output tokens · decode only · one warm-up, three measured trials. Independent axis ranges; batch capacity means the largest feasible tested power-of-two batch.

The accuracy–memory tradeoff

Across Ouro-1.4B, Ouro-2.6B, and Huginn-3.5B, ResidualQuant preserves accuracy across math and code benchmarks with low-bit KV caches. Under mixed INT2/4 with g = 32, its twelve-score average nearly matches BF16 (62.30 vs. 62.58) while reducing logical KV storage by 80.7%. On Ouro-1.4B, it reaches 76.0% MATH500 accuracy compared with 75.0% for BF16.

Downstream performanceAccuracy (%) · higher is better
MethodOuro-1.4B 4 loopsOuro-2.6B 4 loopsHuginn-3.5B 32 loopsAvg.
GSM8KMATH500HumanEvalMBPPGSM8KMATH500HumanEvalMBPPGSM8KMATH500HumanEvalMBPP
BF1679.2375.0071.9574.3486.5879.8079.2781.7543.1416.4023.7839.6862.58
Uniform INT2g = 32 · beff = 2.5
Direct33.6627.8041.4655.5668.5462.4065.8571.4318.957.8020.7333.6042.32
OptR-H62.7055.4059.1570.9081.5072.2072.5673.2828.5810.2022.5636.5153.80
Residual quantization64.6760.6055.4966.4079.5373.6078.6676.4634.959.8024.3936.2455.07
Residual quantization + OptR-H73.3171.6068.9071.6983.8575.2077.4480.1638.2112.6023.7838.8959.64
Uniform INT2g = 16 · beff = 3.0
Direct55.9554.2062.2066.4078.7771.4075.6174.3424.947.0021.9531.2252.00
OptR-H65.9664.2070.7370.9083.4075.0072.5676.7231.6110.8021.3440.7457.00
Residual quantization70.9667.8069.5170.3784.1576.6077.4477.5136.7714.4025.6137.5759.06
Residual quantization + OptR-H75.0675.2070.1271.1685.2279.0075.6181.2238.9714.4026.2239.4260.97
Mixed INT2/4g = 32 · beff = 3.0
Direct48.6747.4056.7167.9976.2773.2070.1278.0430.109.8021.9534.3951.22
OptR-H65.7363.0068.2970.6384.7678.2078.6678.3136.3911.0023.1739.1558.11
Residual quantization77.7973.8071.3473.0286.6680.2079.8881.7542.4614.6023.1740.7462.12
Residual quantization + OptR-H77.0376.0070.7373.8186.3579.6080.4982.2842.7614.6023.7840.2162.30
Mixed INT2/4g = 16 · beff = 3.4
Direct63.1561.8066.4672.7581.9676.6073.7879.6332.8311.6023.1736.2456.66
OptR-H69.9867.0073.7871.1686.0578.6077.4480.1636.2410.2021.3437.8359.15
Residual quantization77.0375.0071.9574.0785.8280.4079.2780.9541.0215.4023.1741.5362.13
Residual quantization + OptR-H77.6374.6072.5673.2887.4180.4079.2782.0142.6115.6022.5639.4262.28

Values and bold highlights reproduce the paper’s main table. Teal rows are ResidualQuant (residual quantization + OptR-H). Residual variants use the final-loop anchor and least-squares scaling. beff = effective bits per KV value (Direct and OptR-H); mixed INT2/4 uses g = 32 for INT4. Avg. is the unweighted mean of the twelve model–benchmark scores. Huginn-3.5B repeats each four-loop schedule eight times.

Citation

@article{kim2026residualquant,
  title  = {ResidualQuant: KV Cache Quantization for Looped Transformers with 2-Bit Residuals},
  author = {Kim, Heejun and Lee, Junyoung and Cho, SangLyul and Han, Dongsu and Han, Insu and Kim, Sehoon},
  year   = {2026},
  eprint = {2610.10381},
  archivePrefix = {arXiv},
  primaryClass = {cs.LG},
  url    = {https://arxiv.org/abs/2610.10381}
}