Abstract
Systems and methods for serving machine-learning inference are provided. A host runtime loads a model artifact a single time into host-owned memory, providing a single resident instance of the model artifact. Inference workflows sharing the single instance each execute in a respective interpreter process isolated from the host runtime and from one another. Each interpreter process holds an opaque handle referencing the single instance without a copy of the model artifact in an address space of the interpreter process. The host runtime receives, over a shared-memory channel, a scoring request issued by a first inference workflow through the opaque handle of a first interpreter process. A host thread of the host runtime scores the request using the single instance and returns a scoring result over the shared-memory channel to the first inference workflow.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Qin, Hao; Xia, Ruiyang; Genty, Victor; Shi, Ningyu; Zhang, Bin; Li, Yi; and Fu, Wenting, "System and Method for Serving a Plurality of Process-Isolated Inference Workflows from a Single Host-Resident Model Artifact Through an Opaque Shared-Memory Handle on a Real-Time Transaction Path", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/12337