Abstract
Limited accelerator memory on mobile devices may restrict the concurrent execution of parallel large language model workloads. The capacity of the key‑value cache may often constrain the number and length of these tasks. A workload scheduling method is disclosed to manage these memory constraints. When the cache capacity is reached, the state of a specific workload is serialized and moved to an alternative memory location. This process frees cache slots, which allows other parallel tasks to proceed. Once memory is vacated by completed tasks, the serialized state is restored to an accelerator cache for continued processing. This approach enables the handling of numerous simultaneous batches within a finite memory environment, which maintains performance by avoiding frequent switching between model variants and mitigates execution failures. A variant may be a pre-compiled model execution graph for a hardware accelerator, with certain shape parameters (e.g., cache sizes, compute size (i.e., the number of work items the variant can process simultaneously), batch size). In this way, the total end‑to‑end latency is improved by dynamically balancing memory usage across independent workloads
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Alrahma, Saeed; Butler, Michael; and Markwell, Matthew, "Managing a Plurality of Parallel Batches", Technical Disclosure Commons, (July 07, 2026)
https://www.tdcommons.org/dpubs_series/10825