Real-Time Data Pipelines for Healthcare Information Systems
Keywords:
Real-time pipelines, healthcare data, streaming analytics, data governance, privacy, interoperability, operational reliability.Abstract
Optimizing the ingestion, transformation, and storage of real-time healthcare data pipelines can enhance information quality and enable critical alert systems and advanced analytics. Stream processing architectures, telemetric data models, and smart telemetric enrichment guarantee low-latency data delivery while upholding provider and regulatory data quality constraints. A rich set of storage technologies meets real-time and near-real-time data availability needs and enables high-performance analytics. Practical considerations such as security, compliance, and operability further strengthen streaming data flows, which support timely detection of critical patient events, accelerate population health insights, and shed light on the impact of data changes on clinical workflows.
Healthcare information systems face increasing pressure to deliver and utilize data in near-real time. Optimized end-to-end pipelines—tailored for ingestion, transformation and enrichment, and storage—boost data quality and internal responsiveness while unlocking more complex external uses, such as alerts for critical events. Timely detection of incidents ranging from falling out of bed to rapid deterioration can help mitigate patient harm and improve clinical outcomes. Broad support for population health and research is likewise crucial, particularly as data privacy legislation toward more open sharing of health data begins to emerge. Acknowledging operational domains and feasibility for wider adoption enables a more complete roadmap for practical implementations.
References
1. Ayaz, M., Pasha, M. F., Alzahrani, M. Y., Budiarto, R., & Stiawan, D. (2021). The Fast Health Interoperability Resources (FHIR) standard: Systematic literature review of implementations, applications, challenges and opportunities. JMIR Medical Informatics, 9(7), e21929.
2. Cheng, A. C., Duda, S. N., Taylor, R., Delacqua, F., Lewis, A. A., Bosler, T., Johnson, K. B., & Harris, P. A. (2021). REDCap on FHIR: Clinical Data Interoperability Services. Journal of Biomedical Informatics, 121, 103871.
3. Mattaparthi, R. (2023). Deep Learning-Driven Combustion Anomaly Detection in Diesel Powertrains: A Multi-Sensor Fusion Approach for Real-Time ECM Adaptation. International Journal of Intelligent Systems and Applications in Engineering, 11, 1084.
4. Xiao, D., Song, C., Nakamura, N., & Nakayama, M. (2021). Development of an application concerning fast healthcare interoperability resources based on standardized structured medical information exchange version 2 data. Computer Methods and Programs in Biomedicine, 208, 106232.
5. Turbow, S., Hollberg, J. R., & Ali, M. K. (2021). Electronic health record interoperability: How did we get here and how do we move forward? JAMA Health Forum, 2(3), e210253.
6. Seong, D., Jung, S., Bae, S., Chung, J., Son, D. S., & Yi, B. K. (2021). Fast Healthcare Interoperability Resources (FHIR)–based quality information exchange for clinical next-generation sequencing genomic testing: Implementation study. Journal of Medical Internet Research, 23(4), e26261.
7. Kolla, S. H., & Loganathan, R. (2023). Cloud-Native Deep Learning Architectures For Secure Generative AI Deployment In Enterprise Workflow Platforms. Journal of International Crisis and Risk Communication Research, 603-618.
8. Ziminski, T., Demurjian, S., & Agresta, T. (2021). Extending the Fast Healthcare Interoperability Resources (FHIR) with meta resources. In Proceedings of the 16th International Conference on Software Technologies (pp. 167–176).
9. Gabetta, M., Alloni, A., Polce, F., Lanzola, G., Parimbelli, E., & Barbarini, N. (2021). Development of a FHIR layer on top of the OMOP Common Data Model for the CAPABLE project. Studies in Health Technology and Informatics, 287, 89–93.
10. Mangala, N. (2022). Implementing Databricks Unity Catalog For Centralized Data Governance In Multi-Business-Unitenterprises. Journal of International Crisis and Risk Communication Research, 101-122.
11. Rinaldi, E., & Thun, S. (2021). From openEHR to FHIR and OMOP Data Model for microbiology findings. Studies in Health Technology and Informatics, 281, 313–317.
12. Klopfenstein, S. A. I., Vorisek, C. N., Shutsko, A., Lehne, M., Sass, J., Löbe, M., Schmidt, C. O., & Thun, S. (2021). Fast Healthcare Interoperability Resources (FHIR) in a FAIR metadata registry for COVID-19 research. Studies in Health Technology and Informatics, 287, 170–174.
13. Garza, M. Y., Rutherford, M. W., Adagarla, B., Eisenstein, E., Kumar, K. R., Zimmerman, K. O., Topaloglu, U., & Zozus, M. (2021). Evaluating site-level implementations of the HL7 FHIR standard to support eSource data exchange in clinical research. Studies in Health Technology and Informatics, 281, 576–580.
14. Yandamuri, U. S. (2023). An Intelligent Analytics Framework Combining Big Data and Machine Learning for Business Forecasting. International Journal Of Finance, 36(6), 682-706.
15. Brandt, C. A., van der Sijs, H., & others. (2021). A FHIR-based approach for standardized clinical data exchange and research data integration. Studies in Health Technology and Informatics, 281, 343–347.
16. Deppenwieser, M., Lehne, M., & others. (2021). Transformation of oncology data into HL7 FHIR for clinical research and interoperability. Studies in Health Technology and Informatics, 281, 346–350.
17. González-Castro, L., Sánchez-Pérez, M. J., & others. (2021). A FHIR-based approach for the collection and aggregation of survivorship data. Studies in Health Technology and Informatics, 281, 388–392.
18. Syed, S. (2023). Shaping The Future Of Large-Scale Vehicle Manufacturing: Planet 2050 Initiatives And The Role Of Predictive Analytics. Nanotechnology Perceptions, 19(3), 103-116.
19. Nan, Y., Del Ser, J., Walsh, S., Schönlieb, C., Roberts, M., Selby, I., Howard, K., Owen, J., Neville, J., Guiot, J., Ernst, B., Pastor, A., Alberich-Bayarri, A., Menzel, M. I., Walsh, S., Vos, W., Flerin, N., Charbonnier, J.-P., van Rikxoort, E., Chatterjee, A., Woodruff, H., Lambin, P., Cerdá-Alberich, L., Martí-Bonmatí, L., Herrera, F., & Yang, G. (2022). Data harmonisation for information fusion in digital healthcare: A state-of-the-art systematic review, meta-analysis and future research directions. Information Fusion, 82, 99–122.
20. Vorisek, C. N., Lehne, M., Klopfenstein, S. A. I., Mayer, P. J., Bartschke, A., Haese, T., & Thun, S. (2022). Fast Healthcare Interoperability Resources (FHIR) for interoperability in health research: Systematic review. JMIR Medical Informatics, 10(7), e35724.
21. de Mello, B. H., Rigo, S. J., da Costa, C. A., Righi, R. da R., Donida, B., Bez, M. R., & Schunke, L. C. (2022). Semantic interoperability in health records standards: A systematic literature review. Health and Technology, 12(2), 255–272.
22. Mangalampalli, B. M. (2023). AI-Driven Anomaly Detection in Healthcare Claims Data: A Business Intelligence Perspective. Journal of Rare Cardiovascular Diseases.
23. Cheng, K. Y., Pazmino, S., & Schreiweis, B. (2022). ETL processes for integrating healthcare data—Tools and architecture patterns. Studies in Health Technology and Informatics, 294, 344–348.
24. Yu, Y., Zong, N., Wen, A., Liu, S., Stone, D. J., Knaack, D., Chamberlain, A. M., Pfaff, E., Gabriel, D., Chute, C. G., Shah, N., & Jiang, G. (2022). Developing an ETL tool for converting the PCORnet CDM into the OMOP CDM to facilitate the COVID-19 data integration. Journal of Biomedical Informatics, 127, 104002.
25. Xiao, G., Pfaff, E., Prud'hommeaux, E., Booth, D., Sharma, D. K., Huo, N., Yu, Y., Zong, N., Ruddy, K. J., Chute, C. G., & Jiang, G. (2022). FHIR-Ontop-OMOP: Building clinical knowledge graphs in FHIR RDF with the OMOP Common Data Model. Journal of Biomedical Informatics, 134, 104201.
26. Bönisch, C., Kesztyüs, D., & Kesztyüs, T. (2022). Harvesting metadata in clinical care: A crosswalk between FHIR, OMOP, CDISC and openEHR metadata. Scientific Data, 9, 659.
27. Yandamuri, U. S. (2022). Big Data Pipelines for Cross-Domain Decision Support: A Cloud-Centric Approach. International Journal of Scientific Research and Modern Technology, 1(12), 227-237.
28. Marteau, B. L., Zhu, Y., Giuste, F., Shi, W., Carpenter, A., Hilton, C., & Wang, M. D. (2022). Accelerating multi-site health informatics with streamlined data infrastructure using OMOP-on-FHIR. In Proceedings of the 44th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (pp. 4687–4690).
29. Nan, Y., Del Ser, J., Walsh, S., Schönlieb, C., Roberts, M., Selby, I., Howard, K., Owen, J., Neville, J., Guiot, J., Ernst, B., Pastor, A., Alberich-Bayarri, A., Menzel, M. I., Walsh, S., Vos, W., Flerin, N., Charbonnier, J.-P., van Rikxoort, E., Chatterjee, A., Woodruff, H., Lambin, P., Cerdá-Alberich, L., Martí-Bonmatí, L., Herrera, F., & Yang, G. (2022). Data harmonisation for information fusion in digital healthcare: A state-of-the-art systematic review, meta-analysis and future research directions. Information Fusion, 82, 99–122.
30. Torab-Miandoab, A., Samad-Soltani, T., Jodati, A., & Rezaei-Hachesu, P. (2023). Interoperability of heterogeneous health information systems: A systematic literature review. BMC Medical Informatics and Decision Making, 23, 18.
31. Reddy, V. A. R. (2022). Designing Fault-Tolerant Data Ingestion Pipelines for High-Volume Healthcare Transactions. Frontiers in Health Informatics, 11, 861-889.
32. Essaid, S., Andre, J., Brooks, I. M., Hohman, K. H., Hull, M., Jackson, S. L., Kahn, M. G., Kraus, E. M., Mandadi, N., Martinez, A. K., Mui, J. Y., Zambarano, B., & Soares, A. (2023). MENDS-on-FHIR: Leveraging the OMOP common data model and FHIR standards for national chronic disease surveillance. Journal of the American Medical Informatics Association, 30(12), 2074–2083.
33. Wilcox, A. B., Murtaugh, C. M., & others. (2023). A standardized clinical data harmonization pipeline for scalable AI application deployment (FHIR-DHP): Validation and usability study. JMIR Medical Informatics, 11, e43847.
Additional Files
Published
Data Availability Statement
None
Issue
Section
License
Copyright (c) 2024 Ganesh Pambala (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
This work is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Authors retain full copyright of their published work. Under this license, others are free to share, copy, distribute, transmit, remix, transform, and build upon the published work for any purpose, including commercial use, provided that appropriate credit is given to the original authors, a link to the license is provided, and any changes made are clearly indicated. No additional restrictions may be applied that limit others from doing anything the license permits. All published articles are freely and permanently accessible online to readers worldwide without any subscription or access fees.
License URL: https://creativecommons.org/licenses/by/4.0/
