Abstract
A runtime precision-control system is proposed herein for reducing the memory occupied by a large language model (LLM) key-value (KV) cache by assigning tokenspecific precision levels based on importance and measured reconstruction error. During live generation, attention-based importance identifies candidates for trial compression. An error gate accepts a lower-precision candidate only when its measured error satisfies a threshold, and a bidirectional control loop can promote a compressed entry when later verification indicates excessive error. System and policy tokens remain at full precision. By avoiding fixed-precision compression and a persistent full-cache uncompressed copy, the approach can increase available cache capacity while preserving output quality. In an experimental evaluation using a three-billion-parameter model, approximately 67% of KV cache memory was saved with a +0.14% change in perplexity.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Calmic, Sergiu; Demian, Joseph; and Manza, Stefan-Alexandru, "INTELLIGENT LIVE KV CACHE OPTIMIZATION ENGINE WITH BIDIRECTIONAL ERROR-GATED PRECISION CONTROLLER", Technical Disclosure Commons, (November 05, 2026)
https://www.tdcommons.org/dpubs_series/12758