Intelligent Performance Optimization of Distributed Databases using AI in Cloud-Native Kubernetes Environments
Abstract
The rapid adoption of the cloud-native apps has increased the demand of the distributed databases that may provide scalable, dependable and responsive data services to meet the dynamic computing environments. However, the unpredictable workload, resource contention, query performance, and data distribution, as well as the varying conditions of a cluster, also present an issue in deployments using Kubernetes. In this paper, a smart performance optimization system of distributed databases in cloud-based Kubernetes will be proposed based on AI. The framework is successful in embedding database monitoring, Kubernetes monitoring, workload-monitoring, AI-based optimization tools, and adaptive optimization systems within a single framework. It is also constantly gathering metrics concerning query latency, throughput, CPU and memory use, storage activity, network traffic and node performance in the database. These observations are used by the AI-based models to identify abnormalities in performance as well as their ability to predict the quantity of resources required, and suitable optimization actions. The framework helps to allocate resources dynamically, schedule allocations based on workloads, optimize query execution, manage replicas, and find intelligent locations of the data. The close loop control system enables continuous monitoring, decision making, optimization and validation with no manual intervention. The proposed architecture provides an organized remedy to the augmentation in the responsiveness, utilization of resources, scalability and economies of databases amid the varying workloads. The framework assessment reveals that it is a prospective smart base of autonomous and robust distributed database administration in the cloud-native infrastructures that employ Kubernetes
Keywords
DOI
PDF Download
Publication Information
| Journal | International Research Journal of Innovative Engineering |
|---|---|
| Volume | 8 |
| Number | 1 |
| Pages | 13919-13929 |
| Published | 2024-01-11 |
| Issue | Vol. 8 No. 1 (2024): International Research Journal of Innovative Engineering (IRJIE) |
References
[2] A. Pavlo, M. Butrovich, A. Joshi, et al., “External vs. internal: An essay on machine learning agents for autonomous database management systems,” IEEE Data Engineering Bulletin, vol. 42, no. 2, pp. 32–46, 2019.
[3] R. Jain, “Beyond stateless: A production architecture for running distributed databases on Kubernetes at scale,” International Journal of Advanced Engineering Science and Information Technology (IJAESIT), vol. 1, no. 2, pp. 416–423, 2018.
[4] A. Ouared, A. Chadli, and M. A. Daoud, “DeepCM: Deep neural networks to improve accuracy prediction of database cost models,” Concurrency and Computation: Practice and Experience, vol. 34, Art. no. e6724, 2022.
[5] S. A. Mohsin, S. M. Darwish, and A. Younes, “QIACO: A quantum dynamic cost ant system for query optimization in distributed database,” IEEE Access, vol. 9, pp. 15833–15846, 2021.
[6] A. Ouared and Y. Ouhammou, “Capitalizing the database cost models process through a service-based pipeline,” Concurrency and Computation: Practice and Experience, Art. no. e6463, 2021.
[7] A. C. Almeida, F. A. Baião, S. Lifschitz, D. Schwabe, and M. L. M. Campos, “Tun-OCM: A model-driven approach to support database tuning decision making,” Decision Support Systems, vol. 145, Art. no. 113538, 2021.
[8] Z. Ding, Z. Wei, and H. Chen, “A software cybernetics approach to self-tuning performance of on-line transaction processing systems,” Journal of Systems and Software, vol. 124, pp. 247–259, 2017.
[9] S. Chaudhuri and V. R. Narasayya, “Self-tuning database systems: A decade of progress,” in Proc. 33rd Int. Conf. Very Large Data Bases (VLDB), 2007, pp. 3–14.
[10] B. Mozafari, E. Z. Y. Goh, and D. Y. Yoon, “CliffGuard: A principled framework for finding robust database designs,” in Proc. ACM SIGMOD Int. Conf. Management of Data, 2015, pp. 1167–1182.
[11] G. C. Durand, R. Piriyev, M. Pinnecke, D. Broneske, B. Gurumurthy, and G. Saake, “Automated vertical partitioning with deep reinforcement learning,” in Proc. European Conf. Advances in Databases and Information Systems (ADBIS), Springer, 2019, pp. 126–134.
[12] A. Ghosh, J. Parikh, V. S. Sengar, and J. R. Haritsa, “Plan selection based on query clustering,” in Proc. 28th Int. Conf. Very Large Data Bases (VLDB), 2002, pp. 179–190.
[13] T. Siddiqui, A. Jindal, S. Qiao, H. Patel, and L. Le, “Cost models for big data query processing: Learning, retrofitting, and our findings,” in Proc. ACM SIGMOD Int. Conf. Management of Data, 2020, pp. 99–113.