Abstract
[0001] A system and method for managing and optimizing a distributed Apache Kafka production cluster 104. The system 102 collects real-time operational metrics from a plurality of Kafka brokers. The system generates a workload fingerprint and retrieves similar historical fingerprints from a historical case repository. Further, the system identifies top-ranked historical fingerprints using similarity computation. The system generates configuration recommendations if the similarity exceeds a threshold and rejects recommendation generation if the similarity is less than the threshold. Further, the system generates a confidence score for the recommended configuration and compares against a deployment threshold. The system deploys the recommended configuration progressively if the confidence score exceeds a deployment threshold and rejects the recommended configurations if the confidence score is not exceeding the deployment threshold. Further, the system automatically rollbacks to previous stable configuration state in response to detection of performance degradation. Further, the system updates the historical case database upon successful deployment and adjusts the fingerprint weighting model and confidence scoring parameters. Further, the system returns to telemetry acquisition.
Creative Commons License
This work is licensed under a Creative Commons Attribution 4.0 License.
Recommended Citation
GOYAL, KARTIK and CHOUDARY, DIWAKAR, "KAFKA PROACTIVE GENTUNER: INTELLIGENT SELF-OPTIMIZING CONFIGURATION TUNER WITH CONFIDENCE-SCORING", Technical Disclosure Commons, ()
https://www.tdcommons.org/dpubs_series/9520