Abstract
The present disclosure relates to dynamically loading transformation logic in a distributed streaming system executing on a Java Virtual Machine (JVM). A record is received from a streaming data source at an executor of a plurality of distributed executors. A routing attribute of the record is determined, and a transformation version is selected from a plurality of transformation versions based on the routing attribute. Each transformation version is packaged as an independently deployable transformation artifact stored in distributed storage accessible to the executors. When the selected transformation version is not present in a transformation cache of the executor, the corresponding transformation artifact is loaded from the distributed storage at runtime using a custom class loader and stored in the transformation cache. The selected transformation version is applied to the record without restarting a streaming job.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Ahmed, Faiz , VISA and Prakash, Anusha Asimane , VISA, "EVENT-DRIVEN DYNAMIC TRANSFORMATION LOADING FOR JVM-BASED DISTRIBUTED STREAMING PIPELINES", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/12816