Abstract
The present disclosure relates to a method and a system (103) for managing data flow between a source system (101) and a target system (105). Here, the system comprises a Metadata Definer (MD) 201 for defining metadata of the input dataset associated with the source system (101), output dataset associated with the target system (105), the data transfer pipeline between the input and output datasets and its internal transformation. The system further comprises a Data Flow Manager (DFM) (203) which extracts metadata input from the MD 201, and determines source type, target type, transfer type, data quality strategy, frequency of pipeline, and column mapping from source to target based on the metadata. Further, the DFM 203 generates query to be run on the source system (101) to obtain records (113,117) in target format (109) at target system (105), selects a template for instantiating pipeline. Thus, the present disclosure enables metadata driven data transfer capability between disparate systems which is cost effective and time efficient, by creating an automated pipeline without requiring manual intervention.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Chandrahas, Shreyas Kunjal, "METHOD AND SYSTEM FOR MANAGING DATA FLOW BETWEEN SOURCE SYSTEM AND TARGET SYSTEM", Technical Disclosure Commons, (February 04, 2021)
https://www.tdcommons.org/dpubs_series/4052