Abstract
The present disclosure provides a solution for seamless integration of data from diverse sources by dynamically managing schema changes. The solution comprises establishing a base schema during data processing and comparing it to incoming data layouts. Any detected changes, such as new tags or fields, are automatically managed within a staging layer, preventing disruptions to the main dataset. These changes are reviewed by Business and Engineering teams for relevance and compatibility. Only after thorough validation, approved changes are processed into the final dataset. This approach ensures that only validated and reliable data is integrated, maintaining data integrity and minimizing disruptions. By automating schema evolution and involving stakeholders in the review process, the system offers a robust method for handling dynamic data sources and evolving data structures efficiently.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
Tripathi, Harshwardhan, "“DYNAMIC SCHEMA EVOLUTION”", Technical Disclosure Commons, (July 03, 2024)
https://www.tdcommons.org/dpubs_series/7159