Abstract
The present disclosure provides a computer-implemented method and a system for receiving events from an event streaming platform, processing the events using a stream processing engine for a processing epoch to generate an output, and writing the output to one or more sinks. Source progress information indicating a consumption position, and execution state information comprising a state version of a state store and a watermark position, are collected for the processing epoch. Sink durability information indicating durable commitment of the output to the one or more sinks is determined. A candidate unified checkpoint manifest binding the source progress, execution state, and sink durability information is generated. When consistency conditions are satisfied, including the consumption position, state version, and watermark position corresponding to the processing epoch and durable commitment of the output, the candidate manifest is sealed, designated as a recoverable restart point, and stored in a persistent manifest store.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
VERMA, ANANYA and KHAN, RAOUF, "CROSS-SYSTEM CHECKPOINT COHESION FOR KAFKA AND SPARK STREAMING PIPELINES", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/12338