Abstract
A system and method for "Semantic-Aware Distributed Replica Orchestration" (SADRO) that optimizes inference latency and cache hit rates in large-scale AI deployments. Unlike traditional load balancers, SADRO utilizes a Semantic Router that maps incoming natural language queries to a high-dimensional vector space. Based on the "cluster" or "topic" of the request (e.g., Legal, Coding, Medical), the system identifies or spins up service replicas on hardware nodes containing specialized data-locality (e.g., specialized KV-caches, hot-loaded LoRA adapters, or domain-specific vector database shards). This reduces the "cold start" penalty and maximizes throughput by ensuring high semantic affinity between the request and the local memory state of the replica.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Shivhare, Shubham and Ahmed, Thousif, "Semantic-Aware Distributed Replica Orchestration (SADRO)", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/11802