A machine learning approach for predicting mortality in unvaccinated COVID-19 patients using a novel web application

INTRODUCTION

The COVID-19 pandemic profoundly impacted global health, leading to millions of infections and fatalities since its emergence in late 2019. The causative virus, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), presented particularly severe consequences for unvaccinated individuals, who faced a heightened risk of severe illness and mortality. In unvaccinated populations, the absence of immunity resulted in increased susceptibility to complications such as acute respiratory distress syndrome, multi-organ failure, and death.

Our team previously conducted a retrospective observational analysis of 401 COVID-19-positive patients admitted between July and December 2020, where we meticulously collected clinical details and laboratory investigations within 3 days of detecting COVID-19 positivity and documented patient outcomes from medical records.^[1] Our findings revealed significant alterations in total leukocyte count (TLC), absolute neutrophil count (ANC), neutrophil-tolymphocyte ratio (NLR), C-reactive protein (CRP), IL-6, lactate dehydrogenase (LDH), ferritin, and lymphocyte-toCRP ratio (LCR) at presentation in severe cases compared to non-severe cases and in deceased compared to surviving patients. Notably, a combination of CRP, IL-6, ferritin, and LCR robustly predicted mortality, even in patients initially presenting with non-severe disease.

Building on these insights, this study aims to enhance the predictive accuracy and clinical utility of our model through the application of machine learning (ML) algorithms. ML, a subset of artificial intelligence, entails the development of algorithms capable of learning from and making predictions based on data.^[2] In the healthcare context, ML models can analyze complex datasets to identify patterns and predict outcomes, such as disease progression and patient mortality. Applying ML to COVID-19 mortality prediction involves training algorithms on clinical and biochemical data to discern the most indicative factors of severe outcomes. This approach promises to enhance early intervention strategies and optimize the allocation of medical resources by identifying high-risk patients requiring intensive care.

Several studies have underscored the utility of clinical and biochemical markers in predicting COVID-19 outcomes. For instance, Zhao et al. (2021)^[3] reported that elevated D-dimer levels were associated with increased mortality in COVID-19 patients. Similarly, Devang et al. (2022)^[4] demonstrated that markers such as CRP, LDH, and ferritin were significant predictors of severe disease and mortality.^[4] These studies emphasize the critical role of biochemical markers in managing COVID-19; however, comprehensive models integrating a broader range of inflammatory and hematological markers are missing, especially in the Indian context. The unique demographic, genetic, and environmental factors in India necessitate localized studies to ensure the findings’ applicability and effectiveness within this population.

Traditional statistical methods, such as logistic regression and linear discriminant analysis, assume linear relationships between predictors and outcomes, which limit their ability to capture complex, non-linear interactions among clinical and laboratory variables in COVID-19 patients. To address this limitation and enhance predictive accuracy and clinical utility, we aim to develop an ML-based prediction model that can identify intricate patterns and non-linear relationships in the data to predict mortality in COVID-19 patients using a combination of clinical, inflammatory, and hematological markers. Our objective is to harness these models’ predictive accuracy and clinical utility, developing an ML-based web application for triaging of patients in any future pandemic/epidemic-like situations to facilitate resource allocation and improved patient management.

MATERIALS AND METHODS

RESULTS

The study included 401 COVID-19-positive patients, with 277 survivors and 124 deaths. Missing data were minimal, with only four patients missing LDH values and six missing IL-6 values, both of which were imputed using median imputation. The mean age of the survivors was 51.52 years (standard deviation [SD] = 16.06), whereas the mean age of the deceased was 53.89 years (SD = 14.62), which was not statistically significant (P = 0.147). The sex distribution was similar between survivors and deceased patients, with males constituting 65.34% of survivors and 69.35% of deceased patients (P = 0.492). However, disease severity at presentation was significantly higher among those who died (90.32%) compared to those who survived (19.86%), with a P < 0.001.

Key laboratory findings revealed significant differences between the two groups. Deceased patients had higher levels of IL-6, LDH, ferritin, CRP, TLC, neutrophil percentage, and NLR, while survivors had higher lymphocyte percentage, monocyte percentage, platelet count, and lymphocyte-toCRP ratio [Table 2].

The performance of various ML models was evaluated using multiple metrics, including accuracy, sensitivity, specificity, precision, NPL, and the AUC, with detailed results provided [Supplementary Table 1]. The support vector machine (SVM) exhibited a test sensitivity of 0.84, test specificity of 0.74, and an AUC of 0.82, alongside a high training accuracy of 0.86 but a lower test accuracy of 0.77, suggesting a degree of overfitting. The extra trees model, another ensemble method, achieved a test sensitivity of 0.81, test specificity of 0.79, and an AUC of 0.81, with a consistent test accuracy of 0.79. Random forest, notable for its ensemble approach, had the highest training accuracy at 0.97 yet showed a test sensitivity of 0.74 and an AUC of 0.82, with a test accuracy of 0.8, indicating potential overfitting. In contrast, XGBoost presented perfect training metrics (accuracy, sensitivity, specificity, and AUC all equal to 1), but its test sensitivity dropped to 0.58, and test accuracy to 0.75, highlighting significant overfitting. Observing these patterns, it is evident that while ensemble methods such as Extra Trees, CatBoost, and Random Forest provided high training performance, they were prone to overfitting, as indicated by the disparity between training and test metrics, especially in models like XGBoost, which showed extreme overfitting despite its initial perfect training scores.

Figure 1 illustrates the performance metrics of various models, highlighting the accuracy in training and test datasets, test sensitivity, specificity, and AUC for each model. The SVM model was noted for its balanced performance, particularly its sensitivity, which is crucial for minimizing false negatives in predicting mortality. SVM hyperparameters were optimized using GridSearchCV with 5-fold stratified cross-validation and recall scoring. The parameter grid included C = [0.01, 0.1, 1, 10], kernel = [“linear,” “rbf,” “poly”], gamma = [“scale,” “auto”], and degree = [2, 3, 4] for polynomial kernels. The best estimator was selected based on cross-validated recall performance, balancing sensitivity, and generalization.

Close

Figure 1:

Performance metrics of various machine learning models. AUC: Area under curve

Export to PPT

Given the high sensitivity of the SVM model, further optimization was performed by reducing the feature set to the top five most predictive features using permutation importance analysis of AUC: Clinical severity, IL-6, LDH, neutrophil percentage, and the NLR. The number of features (5) was selected based on the AUC [Supplementary Figure 1]. This reduced-feature SVM model slightly increased the AUC from 0.82 to 0.83 and improved test accuracy from 0.77 to 0.78 [Figure 2]. In addition, the confusion matrices for the SVM models with all features and with reduced features demonstrated a decrease in false positives (from 18 to 15) while maintaining the same number of true positives (26). The final model’s sensitivity, specificity, PPV, and NPV values are 0.83, 0.78, 0.63, and 0.91, respectively.

Close

Figure 2:

Receiver operating characteristic (ROC) curves and confusion matrices for support vector machine models with all features versus reduced features.

Export to PPT

Overall, the reduced-feature SVM model provided a practical balance between complexity and predictive accuracy, making it well-suited for clinical implementation. The web application developed using Streamlit (https://covidprediction.streamlit.app/) allows clinicians to feed patient data as input and receive real-time mortality risk predictions, enhancing decision-making in managing COVID-19 patients [Figure 3].

Close

Figure 3:

COVID-19 survival prediction web application. (Available from - https://covid-prediction.streamlit.app/)

Export to PPT

DISCUSSION

COVID-19 presented the medical fraternity with an enormous challenge in terms of resource allocation. This spurred extensive research into predictive modeling for patient outcomes, with ML techniques emerging as powerful tools for mortality prediction. In our previous study,^[1] we used a combination of inflammatory and hematological markers with traditional statistical analysis, which helped us pinpoint key predictors of mortality, like CRP, IL-6, ferritin, and LCR. This method gave us results for identifying high-risk patients or those who are likely to face mortality irrespective of the severity of disease at the onset. This type of analysis could, however, not capture the non-linear relationships or interactions between variables, which can be crucial, especially in complex diseases like COVID-19.

Comparing the results of our previous study with the current ML approach reveals significant improvements in predictive accuracy. In the earlier study, individual laboratory parameters showed poor discriminatory performance, with AUC values ranging from 0.614 to 0.710.^[1] The combination of four markers (CRP, IL-6, ferritin, and 1/LCR) improved prognostication but lacked a unified AUC measure. In contrast, our current ML model, particularly the optimized SVM with reduced features, achieved an AUC of 0.83. This marked improvement in AUC demonstrates the superior discriminative ability of the ML approach. Furthermore, the current model’s balanced performance across various metrics (sensitivity: 0.83, specificity: 0.78) suggests a more robust and clinically applicable tool for predicting COVID-19 mortality. ML models could uncover hidden patterns and interactions that traditional methods might miss. In addition, techniques like cross-validation and hyperparameter tuning make ML models more accurate and less prone to overfitting, particularly with large, multi-dimensional datasets. Another strength of ML is its flexibility, which allows for dynamic updates and feature reduction. For instance, with the SVM model, we were able to optimize the most important markers while keeping the model’s performance intact, thus bringing in cost-efficiency in laboratory testing.

Earlier studies have demonstrated the efficacy of various ML algorithms in identifying high-risk patients and predicting adverse outcomes.^[6,7] Gao et al. (2020)^[8] developed a mortality risk prediction model using four ML methods, achieving impressive AUC scores of 0.9621 in internal validation and 0.9760 and 0.9246 in two external validation cohorts. Their study highlighted the potential of ensemble-based models in accurately predicting COVID-19 mortality. Similarly, Subudhi et al. (2020)^[9] compared 18 ML algorithms, finding that ensemble-based models consistently outperformed others in predicting both ICU admission and 28-day mortality. These studies underscored the utility of ML in processing complex clinical data to generate actionable insights for patient care. Vaid et al. (2020)^[10] also demonstrated that ensemble models outperformed individual classifiers, with accuracy rates exceeding 90% in predicting near-term mortality.

Key predictive variables identified across multiple studies have included a combination of demographic factors, vital signs, and laboratory parameters. Gao et al. and Subudhi et al.^[8,9] both emphasized the importance of age, vital signs (such as blood pressure, respiratory rate, and oxygen saturation), and laboratory markers (including LDH, D-dimer, troponin, and creatinine) in predicting mortality risk.^[8,9] Ikemura et al. (2021)^[11] utilized automated ML (autoML) to develop high-performing models, identifying the top 10 influential variables for mortality prediction. Their approach demonstrated the potential of automated techniques in streamlining the model development process. Tezza et al. (2021)^[12] found that Random Forest outperformed other ML techniques in predicting in-hospital mortality, achieving a receiver operating characteristic (ROC) of 0.84. Meanwhile, Yu et al. (2021)^[7] reported high accuracy (86.2%) and NPV (87.8%) using XGBoost for predicting the need for mechanical ventilation. Das et al. (2020)^[13] developed an online prognostic tool using logistic regression, highlighting the importance of accessibility in clinical settings. Emami et al.^[14] compared four ML algorithms, finding that Gradient Boosting Tree models performed best with an accuracy of 70% and ROC area of 0.857. These studies collectively highlighted the diverse array of ML algorithms being applied to COVID-19 outcome prediction, with ensemble methods often yielding superior results.

Our study builds upon this foundation while addressing several gaps in the existing literature, particularly in the context of the Indian population. Unlike many previous studies that focused solely on clinical or readily available laboratory parameters, our research incorporated a comprehensive set of inflammatory biomarkers, including IL-6, which has been less frequently studied in the Indian context. Our results are consistent with prior studies that emphasize the predictive value of laboratory markers for COVID-19 mortality. However, our study identified a unique combination of the top five most predictive features: Clinical severity, IL-6, LDH, neutrophil percent, and the NLR. This feature set is somewhat similar to those identified in the work of Alle et al. (2022),^[15] indicating that the specific characteristics of the Indian patient population and the unvaccinated status of our cohort align with previous findings. The performance of our SVM model, with an AUC of 0.83 after feature reduction, compares favorably with previous studies, though it does not reach the exceptionally high AUC values reported by Gao et al. (2020).^[8] However, our model’s balance of sensitivity (0.83) and specificity (0.78) suggests robust performance in identifying high-risk patients while maintaining a low false-positive rate. Notably, our study’s focus on unvaccinated patients provides valuable insights into mortality prediction for this vulnerable population, addressing a critical need as vaccination rates vary globally. Furthermore, our development of a web-based application for real-time risk prediction represents a significant step toward translating research findings into practical clinical tools, addressing the gap between model development and clinical implementation often seen in previous studies.

However, several limitations must be acknowledged. The data utilized were collected during the first wave of the COVID-19 pandemic, a period preceding the widespread availability of vaccines. Consequently, all patients in this study were unvaccinated, and the viral strains circulating at that time differ from current variants. Vaccinated individuals exhibit different immune responses, disease severity, and outcomes compared to unvaccinated patients. Therefore, models developed from pre-vaccination data may not accurately predict current patient outcomes, limiting the generalizability of our findings to the present scenario. Furthermore, the constantly evolving virus necessitates continuous updates to predictive models to maintain their relevance and accuracy. Hence, more than the usefulness of this study on the evaluation of COVID patients, it is the model of predictive analysis that we have developed that may be of interest.

The SVM model showed a decrease in AUC from 0.91 (training) to 0.82 (test), reflecting the common bias-variance tradeoff in ML. Overfitting, where models memorize training data rather than learning generalizable patterns, is a significant concern in clinical prediction modeling, potentially leading to inadequate or harmful clinical decisions (Kernbach and Staartjes, 2021).^[16] As these authors emphasize, modest out-of-sample performance degradation is expected when prioritizing generalization, whereas massive performance gaps indicate problematic overfitting. Importantly, our SVM demonstrated the smallest performance degradation among all tested algorithms; XGBoost declined from AUC 0.99 to 0.75, while the SVM declined from 0.91 to 0.82, indicating effective regularization through feature reduction and hyperparameter optimization. Contributing factors for this overfitting may include limited sample size, single-center data collection, and inherent biological variability.

Another limitation is the absence of clinical symptoms and examination findings from the model. Parameters such as respiratory rate, oxygen saturation, comorbidities, and imaging results are crucial for a comprehensive assessment of a patient’s condition. The integration of clinical data could potentially enhance the model’s predictive power by providing additional context and capturing aspects of the disease not reflected in laboratory values alone. Initial clinical conditions in our patients were evaluated on the basis of their admission to the wards or the ICU. Future studies should consider incorporating a broader range of clinical features to develop more robust and accurate predictive models.

Despite these limitations, our study underscores the potential of ML approaches in improving mortality predictions in COVID-19 and similar infectious diseases. The ML model not only confirmed the importance of key biomarkers identified in our previous research but also enhanced predictive accuracy by effectively handling complex interactions between variables. This enhancement is evident in our results, where the SVM model with reduced features achieved an AUC of 0.83, compared to our previous study’s individual parameter AUCs ranging from 0.614 to 0.710. This advancement over traditional methods highlights the value of ML in clinical decision-making, enabling earlier identification of high-risk patients and more efficient allocation of medical resources.

Above all, this approach adopted for risk stratification, when coupled with the web application for mortality prediction, may come very handy in any future unforeseen pandemics/epidemics.

CONCLUSIONS

Machine learning, especially the Support Vector Machine model, proved to be a reliable tool for predicting mortality in unvaccinated COVID-19 patients. Using just a few routinely used markers—clinical severity, IL-6, LDH, neutrophil percentage, and NLR—the model achieved strong accuracy and was easy to apply in practice. The web-based tool developed from this model can help clinicians quickly identify high-risk patients and make better decisions about care and resource use during outbreaks.