Abstract
The present disclosure provides a method and a system for account-context-affinity routing in a distributed transformer inference grid. The method includes recording, by a cache state monitor embedded in each inference stack, KV cache hit and miss events for each account bucket and determining a rolling hit rate for each account bucket over a sliding window. The method includes aggregating the rolling hit rates from the inference stacks into a preference matrix mapping account buckets and inference stacks to hit rates and publishing a snapshot of the preference matrix to a routing signal bus. The method includes receiving an authorization request for an account, determining the account bucket of the account, and selecting, by a soft affinity router, an inference stack having a greater hit rate for the account bucket, subject to a capacity guard limiting the fraction of total traffic routed to any single inference stack.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Durga S. Kala; Rabani, Tomer; and Nativi, Benjamin, "ACCOUNT-CONTEXT-AFFINITY ROUTING FOR TRANSFORMER-BASED PAYMENT AUTHORIZATION WITH KEY-VALUE CACHESTATE FRESHNESS AS A ROUTING SIGNAL", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/12403