CICD for Real-Time Analytics on Azure Databricks
Keywords:
CI/CD for Data Engineering, DataOps Frameworks, MLOps Pipelines, DevSecOps Integration, Infrastructure as Code (IaC), Data Lakehouse Architecture, Data Mesh Adoption, Azure Databricks Pipelines, ETL Pipeline Automation, Data Pipeline Orchestration, Near Real-Time Analytics, Streaming Data Processing, Data Availability Optimization, Data Quality Governance, Secure Data Engineering, Observability in Data Systems, Cloud Data Platforms, Automated Data Deployment, Enterprise Data Engineering, Scalable Analytics Systems.Abstract
An end-to-end Continuous Integration/Continuous Deployment (CI/CD) framework for data engineering is proposed, enabling automation from source control to development, testing, deployment, and monitoring. Supporting advanced development patterns like Infrastructure as Code (IaC), DataOps, and MLOps, these best practices enhance collaboration, streamline provisioning, and improve observability while ensuring consistent quality, security, and compliance. As the underpinning for a Zoomcar production-grade data engineering pipeline on Azure Databricks, the architecture supports a demand-based data engineering approach that favors data availability over access latency for Data Mesh adoption.
Cloud-based Data and Analytics platforms support advanced analytics, Artificial Intelligence, and Machine Learning on Near Real-Time Streaming data, enabling data-driven enterprise decision-making. Continuous Integration/Continuous Deployment (CI/CD) with DevSecOps is ubiquitous in Software Development and Application Development. Expanding CI/CD to ETL pipelines supporting MLOps and DataOps brings powerful, tested, and fast-productized code into Production for Cloud-Based Data Engineering Process. Data Availability on Demand supersedes Data Availability in Real-Time. A well-governed Data Lakehouse serving a Data Mesh architecture with a focus on Data Availability, Quality, and Security over Streaming Latency meets the demands of a Data Vision for a Data-Driven Enterprise.
References
1. Avirneni, S. T. (2025). Establishing workload identity for zero trust CI/CD: From secrets to SPIFFE-based authentication. arXiv Preprint arXiv:2504.14760.
2. Baqar, M., Naqvi, S., & Khanda, R. (2025). AI-augmented CI/CD pipelines: From code commit to production with autonomous decisions. arXiv Preprint arXiv:2508.11867.
3. Kumar, S. S., Garapati, R. S., Segireddy, A. R., Kalisetty, S., Inala, R., & Nagabhyru, K. C. (2026). Hybrid Deep Neural Network–DevOps Pipeline Optimization for Risk Prediction in Cloud-Native Workers’ Compensation Platforms. In 2026 IEEE International Conference for Convergence in Computing Technology (I3CTCON) (pp. 1–6). IEEE. 2026 IEEE International Conference for Convergence in Computing Technology (I3CTCON). https://doi.org/10.1109/i3ctcon68242.2026.11507219
4. Valiaiev, D. (2025). DataOps-driven CI/CD for analytics repositories. arXiv Preprint arXiv:2511.12277.
5. Madabathula, L. (2025). Autonomous data ecosystem: Self-healing architecture with Azure Event Hub and Databricks. Journal of Computer Science and Technology Studies, 7(8), 866–873.
6. Kundavaram, V. N. K. (2025). Implementation of real-time event-driven data engineering using Azure Fabric with stream analytics. International Journal of Data Science and IoT Management System, 4(3), 341–349.
7. Kudo, K., & Devi, S. (2025). Scalable CI/CD for legacy modernization: An industrial experience addressing internal challenges related to the 2025 Japan cliff. arXiv Preprint arXiv:2510.17430.
8. Sudha Rani, P. R., Amistapuram, K., Pamisetty, V., Singireddy, S., Kummari, D. N., & Sheelam, G. K. (2025). Hybrid Knowledge Graph–Deep Learning Framework for Automated Exception Handling and Investigation in Complex Insurance Claims. In 2025 IEEE 3rd Global Conference on Wireless Computing and Networking (GCWCN) (pp. 1–6). IEEE. 2025 IEEE 3rd Global Conference on Wireless Computing and Networking (GCWCN). https://doi.org/10.1109/gcwcn66157.2025.11448301
9. Zaharia, M., Xin, R., Wendell, P., Das, T., Armbrust, M., Dave, A., & Stoica, I. (2023). Apache Spark: Unified analytics engine for big data processing. Communications of the ACM, 66(4), 72–81.
10. Armbrust, M., Ghodsi, A., Xin, R., & Zaharia, M. (2023). Lakehouse architecture for modern analytics. VLDB Journal, 32(2), 201–219.
11. Akidau, T., Chernyak, S., & Lax, R. (2023). Streaming systems and low-latency analytics architectures. IEEE Software, 40(5), 41–50.
12. Kimball, R., & Ross, M. (2023). Real-time dimensional modeling for cloud analytics. Data Management Review, 15(2), 18–34.
13. Kumar, S. S., Garapati, R. S., Segireddy, A. R., Kalisetty, S., Inala, R., & Nagabhyru, K. C. (2026). Hybrid Deep Neural Network–DevOps Pipeline Optimization for Risk Prediction in Cloud-Native Workers’ Compensation Platforms. In 2026 IEEE International Conference for Convergence in Computing Technology (I3CTCON) (pp. 1–6). IEEE. 2026 IEEE International Conference for Convergence in Computing Technology (I3CTCON). https://doi.org/10.1109/i3ctcon68242.2026.11507219
14. Gupta, P., & Sharma, R. (2023). DataOps automation for cloud-native data platforms. International Journal of Cloud Applications, 9(3), 55–72.
15. Singh, A., & Verma, N. (2023). DevOps practices for scalable enterprise analytics pipelines. Journal of Software Engineering Research, 14(1), 23–39.
16. Chen, L., & Wang, Y. (2023). CI/CD automation in distributed data engineering systems. IEEE Access, 11, 45891–45910.
17. Krishnan, M., Aitha, A. R., Amistapuram, K., Nandan, B. P., Kaulwar, P. K., & Singireddy, J. (2025). Human-in-the-Loop Hybrid Neuro-Symbolic AI Model for Reliable Data Engineering in High-Stakes Industrial Systems. In 2025 IEEE 3rd Global Conference on Wireless Computing and Networking (GCWCN) (pp. 1–7). IEEE. 2025 IEEE 3rd Global Conference on Wireless Computing and Networking (GCWCN). https://doi.org/10.1109/gcwcn66157.2025.11448516
18. Kumar, S., & Patel, H. (2023). Infrastructure-as-code for cloud analytics deployment. Journal of Cloud Computing, 12(4), 88–103.
19. Gupta, D. K., Purushotham, K., Dheer, G., P, S., Gottimukkala, V. R. R., & Kapoor, S. (2025). Semantic Feature Learning Using Transformer-Based Deep Neural Networks. In 2025 IEEE 5th International Conference on ICT in Business Industry & Government (ICTBIG) (pp. 1–6). IEEE. 2025 IEEE 5th International Conference on ICT in Business Industry & Government (ICTBIG). https://doi.org/10.1109/ictbig68706.2025.11323734
20. Brown, T., & Smith, J. (2023). Continuous delivery in cloud-native enterprise systems. ACM Computing Surveys, 55(9), 1–33.
21. Jones, R., & Miller, K. (2023). Automated testing strategies for data pipelines. Software Quality Journal, 31(2), 401–422.
22. Davis, M., & Wilson, E. (2023). Event-driven architecture for real-time analytics. IEEE Internet Computing, 27(4), 66–74.
23. Bhavani, B. D., SR, S., Loganathan, R., & Nagaraj, S. (2026, April). Evolutionary Gravitational Neocognitron Neural Network, Snow Leopard Optimization and Deep Graph Reinforcement Learning for Routing Protocol in WSN. In 2026 2nd International Conference on Intelligent Systems and Computational Networks (ICISCN) (pp. 1-8). IEEE.
24. Lee, H., & Park, S. (2023). Cloud-native orchestration for streaming workloads. Future Generation Computer Systems, 142, 54–70.
25. Oracle Research Team. (2023). Data observability in enterprise analytics systems. Oracle Technical Journal, 28(3), 22–38.
26. Microsoft Research. (2023). Best practices for CI/CD in Azure analytics workloads. Microsoft Technical Report.
27. Databricks Engineering. (2023). Delta Live Tables for streaming ETL automation. Databricks Technical Whitepaper.
28. Amazon Web Services Research. (2023). Stream processing patterns for enterprise analytics. AWS Research Series.
29. IBM Research. (2023). Operational analytics with automated DevOps. IBM Systems Journal, 62(1), 44–61.
30. Mangala, N. (2026). Responsible AI Data Architecture: Embedding GDPR and PII Compliance into MLOps Pipelines at Enterprise Scale. Canadian Journal of Marketing Research, 16(1), 107-124.
31. OpenAI Research Group. (2024). AI-assisted DevOps automation in modern software pipelines. AI Engineering Review, 6(1), 15–36.
32. Zhao, Q., & Liu, H. (2024). Machine learning driven CI/CD optimization. IEEE Transactions on Software Engineering, 50(2), 334–349.
33. Thomas, P., & Green, D. (2024). Automated rollback and resilience engineering in CI/CD. Software Practice and Experience, 54(3), 221–239.
34. Nguyen, T., & Hoang, L. (2024). Kubernetes orchestration for streaming analytics pipelines. Journal of Systems Architecture, 147, 102811.
35. Garcia, M., & Lopez, A. (2024). GitOps for enterprise data engineering. Journal of Cloud Infrastructure, 8(2), 90–108.
36. Lakhanpal, S., Tunjungsari, H. K., Raj, S., Chukka, A., Dawadi, D., & Mangalampalli, B. M. (2026, April). The Digital Transformation of Healthcare: Balancing Opportunities, Risks, and Ethical Considerations. In 2026 International Conference on Multidisciplinary Innovations For Smart & Sustainable Future (MISSF) (pp. 1-6). IEEE.
37. Ahmed, S., & Khan, F. (2024). Secure software delivery pipelines for cloud platforms. Cybersecurity Review, 12(1), 60–78.
38. Patel, D., & Rao, V. (2024). Data quality validation in real-time ETL systems. Journal of Big Data Analytics, 5(4), 117–132.
39. Kumar, R., & Bose, S. (2024). Azure DevOps integration with analytics pipelines. Cloud Engineering Journal, 18(2), 45–62.
40. Hernandez, J., & White, P. (2024). Observability and telemetry in streaming systems. IEEE Cloud Computing, 11(3), 55–69.
41. Raj, S., Prashanthi, P., Kolla, S. K., Bandaru, S., & Bhardwaj, N. (2026, April). Lightweight Cryptographic Schemes for Resource-Constrained IoT and Healthcare Devices. In 2026 International Conference on Multidisciplinary Innovations For Smart & Sustainable Future (MISSF) (pp. 1-6). IEEE.
42. Ghosh, A., & Mehta, R. (2024). Delta Lake transaction guarantees for enterprise analytics. Journal of Data Engineering, 10(3), 201–219.
43. Li, P., & Sun, X. (2024). Real-time anomaly detection in distributed analytics platforms. Pattern Recognition Letters, 181, 72–85.
44. Kapoor, N., & Arora, S. (2024). Spark structured streaming performance optimization. International Journal of Parallel Programming, 52(2), 145–164.
45. Databricks Engineering Team. (2024). Databricks asset bundles for CI/CD workflows. Databricks Technical Documentation.
46. Microsoft Azure Team. (2024). Continuous integration and delivery on Azure Databricks. Microsoft Learn Technical Documentation.
47. Sharma, P., & Jain, V. (2024). Real-time dashboarding and analytics pipelines. Journal of Business Intelligence Systems, 14(1), 31–48.
48. Mangala, S. B. G. N., Kolla, S. K., Segireddy, A. R., & Yandamuri, U. S. (2026). Designing Intelligent, Scalable Data Ecosystems for Precision Healthcare: A Cloud-Native Approach to Predictive and Automated Decision Systems. Advanced Engineering Sciences, 58(1), 2138-2152.
49. Foster, K., & Lee, D. (2024). Cloud-native reliability engineering for analytics infrastructure. ACM Queue, 22(4), 10–26.
50. Srinivas, K., & Mohan, R. (2024). Automation governance in enterprise analytics delivery. Information Systems Frontiers, 26(4), 721–739.
51. Mattaparthi, R. (2022). From Raw Sensor to Business Signal: An Azure Databricks Framework for Diesel Engine Performance Analytics Across Global Fleet Operations. Journal of Artificial Intelligence & Cloud Computing, 1(4), 1. https://doi.org/10.47363/jaicc/2022(1)529
52. Wu, Y., & Chen, Z. (2025). Predictive scaling for stream processing clusters. IEEE Transactions on Cloud Computing, 13(1), 118–132.
53. Morgan, T., & Ellis, B. (2025). Self-healing data pipelines using AI monitoring. Journal of Intelligent Systems, 34(2), 155–173.
54. Rao, M., & Sinha, K. (2025). Data lineage automation for governance-aware analytics. Journal of Data Governance, 7(1), 1–19.
55. Mangalampalli, B. M., & Kolla, T. (2026). FHIR-Based Interoperability Frameworks For Real-Time Healthcare Data Exchange: Architecture Patterns And Performance Optimization. International Journal Of Advances in Signal and Image Sciences, 1514-1536.
56. Verma, S., & Reddy, P. (2025). Secure CI/CD pipeline architecture for cloud analytics. International Journal of Information Security, 24(2), 89–108.
57. Bose, D., & Das, A. (2025). Streaming data validation in production pipelines. Data Engineering Bulletin, 48(1), 66–82.
58. Chandra, R., & Nair, V. (2025). DevSecOps controls for enterprise data platforms. Cyber Defense Journal, 10(3), 93–111.
59. Kolla, S. H., & Peddi, R. K. (2024). Designing Governance-Aligned GenAI Pipelines Using Small Language Models for Enterprise Workflow Intelligence. International Journal of Science, Research and Technology, 7(6), 13256-13268.
60. Peterson, G., & Hall, S. (2025). Real-time telemetry for distributed Spark workloads. IEEE Systems Journal, 19(2), 2011–2024.
61. Lin, H., & Zhao, F. (2025). Dynamic resource allocation in Apache Spark clusters. Concurrency and Computation, 37(5), e8120.
62. Fernandez, R., & Cooper, J. (2025). Policy-as-code for analytics governance. Software Impacts, 14, 100489.
63. Madhavi, B., Kolla, S. K., & Gari, V. A. K. R. R. (2026). Comment on" Using mobile applications for body composition analysis: A technical review of an artificial intelligence-based tool". Clinical nutrition ESPEN, 103365.
64. Patel, M., & Singh, G. (2025). Automated schema evolution in Delta Lake pipelines. Journal of Database Management, 36(2), 45–63.
65. Yadav, A., & Kulkarni, S. (2025). Enterprise lakehouse modernization on Azure. Cloud Transformation Review, 11(2), 29–46.
66. Databricks Research. (2025). Declarative automation bundles for production analytics. Databricks Research Notes.
67. Davuluri, P. N. (2026). Autonomous Compliance Systems: AI, Event Streaming, and the Future of Financial Crime Prevention. Journal of Informatics Education and Research.
68. Microsoft Corporation. (2025). Azure Event Hubs and Databricks for real-time analytics. Azure Architecture Center.
69. Bandi, V. D. V. K. Autonomous Data Platforms: Converging AI, MLOps, and Cloud Engineering for Digital.
70. Google Cloud Research. (2025). Modern DataOps patterns for large-scale analytics. Google Cloud Whitepaper.
71. Open Source Initiative. (2025). GitOps patterns in data platform automation. Open Infrastructure Report.
72. Rani, S., Thavara, S. S., Kr, A., Naguband, A. R., Garg, A., & Vajpayee, A. Financialization of Sustainability: ESG Signalling, Market Valuation and the New Economics of Corporate Legitimacy.
73. Wang, T., & Li, Y. (2026). Autonomous CI/CD for data-intensive systems. IEEE Access, 14, 11901–11925.
74. Ramesh, P., & Narayan, V. (2026). AI agents for DevOps orchestration in analytics platforms. Journal of AI Engineering, 8(1), 52–71.
75. Silva, C., & Martins, R. (2026). Intelligent rollback systems for production pipelines. Software Reliability Engineering Journal, 29(1), 12–28.
76. Hassan, A., & Rehman, M. (2026). Adaptive observability for cloud-native analytics workloads. Future Internet, 18(2), 94.
77. Krishnan, M., Bandi, V. D. V. K., Mangala, N., Kolla, S. H., & Mangalampalli, B. M. Engineering Intelligent Cloud-Native Data Ecosystems for Predictive Decision-Making in Industry.
78. Iyer, S., & Gupta, N. (2026). Zero-downtime deployment for streaming data applications. Journal of Cloud Operations, 9(1), 77–95.
79. Nelson, B., & Clark, R. (2026). Data reliability engineering in lakehouse architectures. ACM Digital Library Proceedings, 211–226.
80. Kulkarni, K. (2026). Building Context-Aware Enterprise Intelligence: A Secure and Scalable Architecture for AI-Augmented Data Systems. Minnesota Journal of Business Law and Entrepreneurship, (1), 1315-1336.
81. Databricks Engineering. (2026). Best practices and recommended CI/CD workflows on Databricks. Databricks Documentation.
82. Microsoft Azure Team. (2026). CI/CD workflows on Azure Databricks. Microsoft Learn Documentation.
83. IEEE Standards Association. (2026). Standards for software delivery automation in cloud systems. IEEE Standards Publication.
84. Inala, R., Garapati, R. S., Aitha, A. R., Komaragiri, V. B., Gottimukkala, V. R. R., Recharla, M., ... & Varri, D. B. S. (2026). U.S. Patent Application No. 19/389,108.
85. ACM DevOps Working Group. (2026). Emerging practices in continuous delivery for analytics engineering. ACM Technical Bulletin.
Additional Files
Published
Issue
Section
License
Copyright (c) 2026 Mallesham Goli (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
The American Online Journal of Science and Engineering (AOJSE) is an open-access journal, and all published articles are made freely available to readers worldwide under the terms of the Creative Commons Attribution 4.0 International License (CC BY 4.0). This license governs the use, sharing, adaptation, and distribution of all content published in AOJSE and is designed to promote the broadest possible dissemination and reuse of scholarly research.
Creative Commons Attribution 4.0 International License (CC BY 4.0)
Under the Creative Commons Attribution 4.0 International License, any user is free to share, copy, and redistribute the published material in any medium or format, and to adapt, remix, transform, and build upon the material for any purpose, including commercial use, provided that appropriate credit is given to the original authors and source. The license cannot be revoked as long as the terms are followed.
When using or redistributing content published in AOJSE, users must provide proper attribution by citing the original authors' names, the article title, the journal name (AOJSE), the volume and issue number, the year of publication, and the DOI or URL of the article. Users must also indicate if any changes were made to the original work. Attribution must be provided in a reasonable manner but must not suggest that the authors or AOJSE endorse the user or their use of the material.
Author Rights and Copyright Retention
AOJSE respects and upholds the intellectual property rights of all contributing authors. Authors who publish in AOJSE retain full copyright over their work. By submitting a manuscript to AOJSE, authors grant the journal a non-exclusive, worldwide, royalty-free license to publish, reproduce, distribute, publicly display, and archive the article in print and electronic formats. This license allows AOJSE to make the article freely accessible to readers globally while ensuring that the authors remain the rightful owners of their work.
Authors are permitted to deposit their published articles in institutional repositories, personal websites, academic networking platforms, and preprint servers, provided that the original publication in AOJSE is acknowledged and properly cited. Authors may also reuse their published content in subsequent works, presentations, teaching materials, and grant applications without requiring prior permission from the journal.
Third-Party Content
Authors are solely responsible for obtaining written permission to reproduce any third-party material, including figures, tables, images, data, or excerpts from other publications, included in their manuscript. Evidence of such permissions must be provided to the editorial office upon request. AOJSE does not assume any responsibility for copyright infringement arising from the unauthorized inclusion of third-party content in published articles.
Permitted Uses
Under the CC BY 4.0 license, the following uses of AOJSE published content are freely permitted without prior written permission from the journal or authors, provided that proper attribution is given. Users may read, download, print, and distribute articles for personal, educational, or research purposes. Researchers may reuse data, figures, and findings for meta-analyses, systematic reviews, and secondary research. Educators may incorporate published articles into course materials, syllabi, and academic presentations. Journalists, policymakers, and practitioners may reference and cite published findings for professional and public interest purposes.
Prohibited Uses
While the CC BY 4.0 license is broad and permissive, users may not apply legal terms or technological measures that legally restrict others from doing anything the license permits. Users may not misrepresent the original authorship of published content or imply endorsement by the original authors or AOJSE without explicit consent. Plagiarism, data fabrication, and any form of academic misconduct in the reuse of published material are strictly prohibited and are a violation of publication ethics standards.
Institutional and Repository Deposit Policy
AOJSE fully supports self-archiving and green open access. Authors are encouraged to deposit the final published version of their article, also known as the version of record, in institutional repositories, subject repositories, and open-access databases immediately upon publication. There is no embargo period. Authors should always link back to the original article on the AOJSE website and include the full citation and DOI when depositing their work in any repository.
Digital Preservation and Archiving
AOJSE is committed to the long-term digital preservation of all published content to ensure its permanent availability and accessibility. The journal utilizes the Open Journal Systems (OJS) platform, which supports internationally recognized digital archiving standards. AOJSE encourages participation in archiving networks such as LOCKSS (Lots of Copies Keep Stuff Safe) and CLOCKSS (Controlled LOCKSS) to safeguard published articles against data loss and ensure continued access for future generations of researchers and readers.
Disclaimer
The views, opinions, findings, and conclusions expressed in articles published in AOJSE are solely those of the authors and do not necessarily reflect the official position, policy, or views of the journal, its editorial board, or its affiliated organizations. AOJSE makes every effort to ensure the accuracy and integrity of published content but does not accept legal responsibility for any errors, omissions, or claims arising from the use of information published in the journal.
Contact for Licensing Inquiries
For any questions regarding the licensing terms, copyright, or permitted use of content published in AOJSE, please contact the editorial office through the journal website at https://aojse.org. The editorial team will be happy to assist authors, readers, and institutions with any licensing-related queries.
