Abstract
A hierarchical tiered storage subsystem manages variable-size per-user inference state for LLM-based recommendation services. Per-user attention KV cache objects are persisted across sessions and placed across HBM, DRAM, and SSD. SSD storage uses a slab-class allocator with aligned chunks for variable-size objects. A tier placement engine assigns users to tiers based on an activity metric. Cache validity is controlled using version metadata including model and user-related versions, enabling profile-version-aware invalidation. A session-start pipeline asynchronously prefetches validated KV cache objects into HBM prior to a first recommendation request. Cold-tier objects are compressed using attention-head pruning and quantization. Cross-replica consistency is maintained via lease-based write-through and cache sequence numbers to detect staleness and refresh from a shared tier.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Anonymous, "Hierarchical Tiered Storage for Variable-Size Per-User Inference State in LLM-Based Recommendation Systems", Technical Disclosure Commons, (June 30, 2026)
https://www.tdcommons.org/dpubs_series/10661