Abstract
A system performs large-scale information retrieval using a large language model (LLM) by pre-computing a key-value (KV) cache for a document corpus. The retrieval prompt is structured into static and dynamic parts, where the static portion includes system instructions and the entire document corpus. In an offline process, the LLM processes this static portion to generate and store a corresponding KV cache. During online inference, the system loads the pre-computed KV cache and appends the dynamic user query, which enables the LLM to perform its attention-routing task without reprocessing the entire corpus. A layerwise prefilling technique manages the memory footprint during the offline pre-computation, supporting the processing of corpora containing millions of tokens. This architecture facilitates full cross-attention between the query and every document token, providing high-precision retrieval for complex queries with negations or compositional constraints. Bypassing the prefill phase at query time reduces retrieval latency from over 100 seconds to under two seconds for a 10,000-item corpus and allows the system to scale to a corpora of 100,000 items or more.
Keywords: in-context retrieval, corpus key-value (KV) caching, layerwise prefill, offline pre-computation, large language model (LLM), context window, product cluster, prompt organization, inference latency, query answering
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Rabiee, Ladan; Yu, Felix; Guo, Ruiqi; Collins, Chris; and Zhou, Tianyin, "Efficient In-Context Retrieval for Search Ads via Pre-computed Corpus Key-Value Caching", Technical Disclosure Commons, (September 16, 2026)
https://www.tdcommons.org/dpubs_series/11760