Multimodal Temporal Representation Learning for Early ICU Deterioration Forecasting from Physiologic Streams and Clinical Narratives

Abstract

Intensive care environments generate dense streams of physiologic observations, laboratory measurements, and narrative documentation. Converting these heterogeneous signals into reliable short-horizon forecasts remains difficult because clinical trajectories are irregularly sampled, event prevalence is low, and much of the most informative context is embedded in free text rather than in structured fields. These characteristics make deterioration forecasting a representation learning problem as much as a clinical one. This paper develops a multimodal deep learning framework for early prediction of patient deterioration in the intensive care unit by combining structured time-series electronic health record data with contextualized embeddings from contemporaneous radiology reports. The study uses 74,822 adult ICU stays and 5,720,904 hourly prediction samples, with deterioration defined as death in the ICU, vasopressor initiation, or invasive mechanical ventilation within the subsequent 24 hours. The proposed system uses a bidirectional recurrent temporal encoder with attention for 48-hour physiologic windows, a frozen domain-specific language model for note representation, and gated fusion to adaptively integrate both modalities under substantial missingness. On a held-out test set of 823,641 samples, the multimodal model achieves an AUROC of 0.7857 and an AUPRC of 0.1908, outperforming a structured-only deep model, XGBoost, and logistic regression. Relative to the structured-only baseline, note integration yields a 2.5 percentage point AUROC gain and a 39.2% AUPRC increase, indicating that narrative documentation contributes complementary predictive signal even when text alone is weak. The empirical analysis further shows rapid convergence, small validation-to-test drift, and strong post-hoc recoverability of calibration, suggesting that multimodal temporal learning is a viable computational framework for real-time ICU risk stratification.