The MaD-CLINYC score: An easy tool for the prediction of the outcome of hospitalized COVID-19 patients

Posted on by admin

RESEARCH ARTICLE

Hippokratia 2021, 25(3): 119-125

Koutroulos MV, Bakola SA, Kalpakidis S, Avramidou D, Panagaris S, Melissopoulou E, Souleiman H, Partsalidis A, Metaxa E, Feresiadis I, Kampaki E, Papadopoulos V
Department of Internal Medicine, Xanthi General Hospital, Xanthi, Greece

Abstract

Background: Most outcome-predictive models for COVID-19 patients use hospital admission data, offering a spontaneous mortality risk estimation. We aimed to elaborate on a tool that could be applied repeatedly, thus being more suitable for these patients’ rapidly changing clinical course.

Methods: In this prospective study, we evaluated 560 samples derived from 156 patients hospitalized for COVID-19 in a single center. Age >61 years, male sex, comorbidities >2, need for intensive care unit admission, lactate dehydrogenase (LDH) >408 U/L, Neutrophil/Lymphocyte Ratio (NLR) >17, C-reactive protein (CRP) >10 mg/dl, and D-dimers >3,200 ng/ml were incorporated in an eight-scale score (MaD-CLINYC) after optimal scaling, ridge regression, and bootstrapping, which was documented to correlate with outcome independently of one or more samples analyzed, day from admission at sampling, and need for delivery. Validation process was performed over 574 samples derived from three centers.

Results: The developing and the validation cohort Area under Curve (AUC) was 0.90 (95 % Confidence Interval: 0.82-0.98) and 0.91 (0.88-0.94), respectively (p =0.822). A MaD-CLINYC score ≥4 had 75 % sensitivity and 81 % specificity to predict fatal outcome.

Conclusions: MaD-CLINYC score is a powerful, feasible, easy-to-use, dynamic tool to assess the risk of the outcome, thus assisting clinicians in close monitoring and timely decisions in COVID-19 hospitalized patients. HIPPOKRATIA 2021, 25 (3):119-125.

Keywords: Coronavirus disease 2019, COVID-19, Neutrophil/Lymphocyte Ratio, C-reactive protein, lactate dehydrogenase, D-dimers, NLR, CRP, LDH, intensive care unit, mortality

Corresponding author: Papadopoulos Vasileios, Department of Internal Medicine, Xanthi General Hospital, 67133, Xanthi, Greece, e-mail: vaspapmd@gmail.com

Introduction

Since December 2019, more than 265 million individuals have been infected worldwide by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) causing COVID-19. Consequently, mild to moderate symptoms from the upper respiratory tract usually emerged; however, a small percentage of patients developed acute respiratory distress syndrome (ARDS) that may lead to the need for mechanical ventilation or death1.

Various factors have been demonstrated to affect the survival of COVID-19 patients. Patients that have a medical record of hypertension, diabetes, cardiovascular disease, chronic obstructive pulmonary disease (COPD), cerebrovascular disease, chronic renal failure, and obesity have a higher probability of a fatal outcome than comorbidity-free ones2-4. Moreover, the male gender and increased age contributed to unfavorable outcome5,6. Several biomarkers, including elevated C-reaction protein (CRP), lactate dehydrogenase (LDH), D-dimers, and Neutrophil/Lymphocyte Ratio (NLR), are used as severity markers or outcome prognosticators in COVID-19 disease7-10.

Numerous predictive models for COVID outcomes have been reported to offer an estimation of risk based on admission values. The 4C score incorporates age, sex, comorbidities, peripheral oxygen saturation, respiratory rate, Glasgow Coma Scale, blood urea nitrogen, and CRP on admission in a 21-point score to assess COVID-19 mortality11. Moreover, the CALL score encompasses age, comorbidities, lymphocyte count, and LDH in a 13-point score to approach the risk for disease progression12. Additionally, CURB-65, a prognostic tool for the assessment of the severity of community-acquired pneumonia (CAP), comprising five variables (new onset confusion, increased urea, low blood pressure, increased respiratory rate, and age ≥65 years), had also been validated for COVID-19 disease13,14.

Even more recent prognostic tools such as the COVID-GRAM score, the COVID-19 severity score, the Mortality Risk Prediction Model for COVID-19 (MRPMC), the model developed by Yadaw et al, the COVID-19 score, and the COVID-19 risk index (CRI) still focus on admission clinical and laboratory data15-20. In one of the most exhaustive studies that had evaluated 35 parameters as potential outcome predictors at the time of admission, male sex, increased age, elevated white blood cell count, low platelet count, anemia, obesity, diabetes, malignancy, dementia, rhinorrhea, dyspnea, and unconsciousness were demonstrated to be independent risk factors for non-surviving21. A systematic living review attempts to evaluate the numerous predictive models for covid-19 qualitatively22.

Recently, a 10-point score implicating the Activins/Follistatin axis, called the FACT-CLINYCoD score, was reported to predict in-hospital mortality in COVID-19 patients in a very efficient manner [Area Under Curve (AUC): 0.95; 95 % Confidence Interval (CI): 0.92-0.98], while being independent of disease day. This tool was demonstrated to be valid for as frequent re-evaluation as the clinician could ask for, reflecting the dynamically changing pathophysiology of the disease. However, its use is limited by the fact that activins and follistatin are not routinely measured in most healthcare facilities8.

The study hypothesis was that a risk score of prognostic value regarding the outcome of hospitalized COVID-19 patients should reflect both the baseline risk, mainly attributed to non-modifiable factors as sex, age, and comorbidities, and the current risk described by markers of hyperinflammation and immunothrombosis, such as D-dimers, CRP, LDH, and NLR. Furthermore, a surrogate marker of respiratory failure, recognized as a crucial independent risk factor of the outcome, had to be additionally considered; in keeping with the FACT-CLINYCoD score, the need for mechanical ventilation was preferred to the P/F ratio (PaO2 divided by FiO2) as such.

In the present study, inspired by the dynamic nature of the FACT-CLINYCoD score, we have focused on establishing and evaluating a more straightforward prognostic tool for the outcome of COVID-19 patients using an initial and a validation cohort.

Material and Methods

The present study is a non-interventional, both retrospective and prospective, cohort one. A single outcome (final outcome recorded as survival or death) has been introduced. All consecutive COVID-19 patients who were hospitalized between December 2020 and May 2021 in Xanthi General Hospital, Greece, and received standard-of-care treatment entered the study. Inclusion criteria were: i) age >18 years, ii) positive SARS-CoV-2 testing in either nasopharyngeal swab or bronchoalveolar lavage [using either reverse transcription-polymerase chain reaction (RT-PCR) or rapid antigen detection plus the presence of bilateral infiltrates in chest X-ray), iii) symptomatic COVID-19 disease requesting hospitalization, and iv) known final outcome (either survival or death). Exclusion criteria were: i) duration of symptoms attributable to COVID-19 for over 14 days, and ii) medical record of illicit drug abuse, psychiatric illness, and mental retardation.

All samples derived from enrolled patients, either single at admission or multiple as requested during their hospitalization period, were analyzed for NLR, CRP, LDH, and D-dimers. For every sample, gender, age, number of comorbidities, intensive care unit (ICU) admission, day from admission, necessity for delivery, and outcome were additionally collected. Diabetes mellitus, arterial hypertension, obesity, dyslipidemia, cardiovascular disease (including coronary artery, heart failure, and atrial fibrillation), asthma,  COPD, renal failure, autoimmunity, immunosuppression, and cancer have been considered comorbidities.

Data from samples derived from a recent publication has been used as a validation cohort after permission8. The study conformed to the TRIPOD statement23. All patients’ data were handled as secured record numbers during analysis to achieve anonymity and confidentiality.   The study protocol was submitted to the Local Scientific Committee of the Xanthi General Hospital in December 2020 (Decision of approval No 102/17-05-2021).

Statistical analysis

Pearson’s chi-square or two-sample independent t-test were used to compare discrete and continuous variables, respectively, between survivors and non-survivors. In case of deviation from normality as evaluated by the Kolmogorov-Smirnov test, the non-parametric Mann-Whitney U test was preferred instead of the t-test.  Means are accompanied by their ±95 % standard deviations (SD). The level of statistical significance was set to p =0.05.

Eight parameters, namely sex, ICU admission, LDH, CRP, NLR, D-dimers, age, and the number of comorbidities, were considered predictors. The SPSS CATREG procedure was used to transform all available non-binary parameters (LDH, CRP, NLR, D-dimers, age, and the number of comorbidities) to binary ones selecting the best cutoff through nominal optimal scaling/discretization to two groups after bootstrapping application. For this purpose, ridge regression was preferred, as it can both tolerate variance and handle collinearity; to avoid overfitting, models including parameters with tolerance <0.6 were rejected.

The best fit model was used to construct and evaluate a scoring system. Each sample was matched to a score that is the sum of one additive point for each parameter documented to be independently correlated with the unfavorable outcome as indicated after the discretization process. Binary regression was used to assess the correlation of final scores with the outcome. Probability of death was computed as previously described8. A Generalized linear model was used to detect potential confounders, including sampling approach, day from admission at sampling, and need for delivery, by assessing their correlation with the score.

The accuracy of the scoring system was assessed by AUC, as defined by analysis of the Receiver Operating Characteristics (ROC), along with Precision-Recall Curves. The minimum sample size to ensure AUC >0.7 with a <0.05 and b <0.20, as well as the difference between independent ROC curves, was performed using MedCalc Statistical Software version 20.011 (MedCalc Software Ltd, Ostend, Belgium; https://www.medcalc.org; 2021).

Multiple imputations were used for handling missing data. Outcome, age, sex, comorbidities, and ICU admission were used only for prediction, while NLR, CRP, LDH, and d-dimers were used for prediction and imputation. Imputed data tolerated missing values at an overall maximum of 10 %.

Statistical analysis was performed using IBM SPSS Statistics for Windows, Version 26.0 (IBM Corp., Armonk, NY, USA).

Results

Comparability between survivors and non-survivors

A number of 156 consecutive COVID-19 hospitalized patients (118 survivors and 38 non-survivors) were enrolled, and a total of 560 randomly acquired samples (270 from survivors and 290 from non-survivors) were analyzed. Samples from survivors and non-survivors were found to differ regarding the hospitalization period of the patient, sampling approach (single sample on the first day of hospitalization or multiple samples during hospitalization period), day from admission at sampling (in case of multiple sampling), need for delivery, ICU admission, sex, age, number of comorbidities, NLR, CRP, LDH, and D-dimers (Table 1).

Establishment of MaD-CLINYC score

A predictive model was based on the eight parameters that are considered to be potential predictors, namely ICU admission, sex, age, number of comorbidities, NLR, CRP, LDH, and D-dimers, and optimal cutoffs were proposed by optimal scaling procedure. Based on this model, a scoring system under the acronym MaD-CLINYC [(Ma)le sex, (D)-dimers, (C)RP, (L)DH, (I)CU admission, (N)LR, (Y)ears of age, (C)omorbidities] was proposed to quantify outcome probability. In detail, an additive point was given for male sex, D-dimers >3,200 ng/ml, CRP >10 mg/dl, LDH >408 U/L, ICU admission, NLR >17, age >61 years, and comorbidities >2 (Table 2). Relevant ridge regression paths are depicted in Figure 1. Assessment of quantification of outcome predictability was performed using binary regression (Table 3 and Table 4). Total missing values were 7.8 %; this was attributed to protocol violation in cases of emergency situations.

Figure 1: Ridge regression paths for model leading to the MaD-CLINYC score.

CRP: C-reactive protein, ICU: intensive care uni, LDH: lactate dehydrogenase, NLR: Neutrophil/Lymphocyte Ratio

Figure 2: The MaD-CLINYC score: Receiver Operating Characteristics (ROC) curve based on original (A) and imputed (B, C, D, E, F) data as produced after five iterations. The outcome, age, sex, comorbidities, and Intensive care Unit (ICU) admission were used for prediction only, while Neutrophil/Lymphocyte ratio (NLR), C-reactive protein (CRP), lactate dehydrogenase (LDH), and D-dimers were used for both prediction and imputation. Original data yielded to Area under the curve (AUC) =0.897 (95 % Confidence Interval: 0.818-0.976), while imputed data to AUC 0.906-0.927. Score ≥ 4 has 75% sensitivity and 81% specificity to predict fatal outcomes.

ROC: Receiver Operating Characteristics.

Assessment of potential confounding effect

The sampling approach, the day from admission at sampling, and the need for delivery were recognized as potential confounding factors. A generalized linear model incorporating these parameters as independent variables along with MaD-CLINYC as dependent was used to check this hypothesis. No independent correlation of the above-mentioned covariates with the MaD-CLINYC score was documented (p =0.643 for sampling approach, p =0.783 for day from admission at sampling, and p =0.980 for delivery) (Table 5).

Assessment of the MaD-CLINYC score predictability

The relevant MaD-CLINYC score ROC curve presented AUC =0.897 (95 % CI: 0.818-0.976). A MaD-CLINYC score ≥4 has 75 % sensitivity and 81 % specificity to predict fatal outcome (Figure 2). As 182 samples (91 from survivors and 91 from non-survivors) are needed as a minimum to discriminate an AUC of 0.7 from baseline (AUC =0.5) with 0.05 type I error and 0.20 type II error, the sum of 560 samples collected (270 from survivors and 290 from non-survivors) is considered adequate.

Validation of the MaD-CLINYC score

The MaD-CLINYC score was applied to a validation cohort; for that purpose, data derived from 574 samples published elsewhere were used8. The relevant AUC was found to be 0.907 (0.878-0.936); this result was comparable (p =0.822) to that resulted from the development cohort (Figure 3). 

Figure 3: Initial (blue line) and validation (red line) cohort on 575 samples (data derived from Synolaki et al after permission): A) Receiver Operating Characteristics (ROC) curve and B) Precision-Recall curve; p =0.822 for the Area under the ROC curve (AUROC).

ROC: Receiver Operating Characteristics.

Discussion

An eight-point prediction model for mortality of hospitalized COVID-19 patients is described in the present study. The model was demonstrated to be independent of the day elapsed from disease onset, the severity of the disease, the number of samples collected during hospitalization (single or multiple), and the timing of each sampling, while it retained its validity even in case of patients requesting delivery to another hospital. Thus, it can be used repeatedly at any time to serve the clinicians’ need for re-evaluation of the risk estimation of outcomes in hospitalized COVID-19 patients. Moreover, the model is feasible as it comprises easy-to-collect parameters, including demographics (sex and age), clinical characteristics (comorbidities and need for ICU admission), and laboratory parameters that are available in a secondary hospital setting (CRP, NLR, LDH, and D-dimers). Lastly, its simplicity in computation (assessing either 0 or 1 for each of the above-mentioned parameters and summing up to a total score) renders it an easy tool for emergency use by clinicians. The additional prospective nature of the cohort under a predefined protocol empowered the validity of the results.

The current score, under the name MaD-CLINYC, was based on the conception of the FACT-CLINYCoD score. However, the resulting MaD-CLINYC score differed from the FACT-CLINYCoD score as the A/F axis has been omitted, while the male sex was demonstrated to be an additional predictor through the evaluation of data from the developing cohort. The score incorporated the need for mechanical ventilation, which is undeniably a crucial predictor of outcome. This dynamic component reflects the altering clinical course attributed to hyperinflammation and immunothrombosis (CRP, d-dimers, LDH, and NLR), and a significantly sizeable non-modifiable component, including sex, age, and comorbidities, which is considered to reflect the baseline risk. A very recent article evaluated scoring systems to predict mortality in patients with COVID-19. The most common parameter used in predictive models is age, after which lymphocyte count, D-dimers, oxygen saturation, CRP, platelet count, respiratory rate, LDH, and NLR. All three predictive models with the higher AUROC included D-dimers and lymphocyte count24; this is in keeping with the MaD-CLINYC score, as NLR and lymphocyte count are highly collinear in our analysis.

The majority of scores used for predicting outcomes in hospitalized COVID-19 patients have been validated for risk estimation at the time of hospital admission. Hence, they are inadequate to follow the need for re-estimation of risk and updated guidance due to the rapidly changing clinical course of COVID-19 disease throughout the hospitalization period16-20. Until present, the only prognostic tool developed to approach the dynamic and rapidly changing clinical course of COVID-19 was the FACT-CLINYCoD score, which incorporated the Activins/Follistatin axis (A/F axis) as an independent predictor of outcome in hospitalized patients with COVID-19. The strength of this tool is that it can be used not only as a guide at admission but repeatedly and independently at any time during hospitalization8. However, despite its pathophysiological value and accuracy, the FACT-CLINYCoD score is impractical since activin-A, activin-B, and follistatin cannot be measured routinely.

Apart from the strengths of the MaD-CLINYC score, there might be some limitations, too. One can debate that the number of patients included in the developing cohort is limited. However, the minimum of 182 samples (91 from survivors and 91 from non-survivors) needed for both derivation and validation cohort is far outnumbered.

Another limitation is that the developing cohort endorsed patients from a single center. Despite the fact that the MaD-CLINYC score has been developed from a single center cohort (Xanthi General Hospital), its validation cohort, derived from three referral centers, has produced strikingly matching results. However, further external prospective validation would support the findings of the study.

The MaD-CLINYC score lacks significant clinical parameters used in other prognostic models as temperature or SpO2/FiO2 ratio25,26. Despite that this might not be a true limitation or disadvantage, the necessity of close monitoring based on clinical evaluation of the patient should not be disregarded or substituted by stochastic approaches such as the MaD-CLINYC score by clinicians. Furthermore, as the MaD-CLINYC score incorporates CRP, its accuracy might be influenced by proposed treatments that dramatically impact this marker, such as the interleukin-6 inhibitor tocilizumab27 or the interleukin-1 inhibitor anakinra28.

In conclusion, the MaD-CLINYC score is a powerful, feasible, easy-to-use, dynamic tool that might serve as a guide for clinicians to assist in close monitoring and timely decisions in the rapidly changing clinical course of COVID-19 hospitalized patients. Further prospective studies are needed to consolidate these results.

Conflict of interest

The authors declare that they have no conflict of interest.

Acknowledgements

The assistance provided by Gampra Maria, Amanatidou Anastasia, Politou Maria, Georgiou Georgios, and Vakalos Aristeidis in the clinical assessment of the patients is of utmost significance for the present study. The assistance provided by Agorasti Athanasia, Midioura Triantafyllia, Tsiliggiris Athanasios, Nikolakopoulou Eftychia, Kotanidou Agapi, Orfanidou Eva, Kalinina Sophia, and Petridou Eleni in the interpretation of the laboratory results is greatly appreciated.

All authors express their gratitude to Dr. Paschalis Sideras for his permission to use raw data to validate the MaD-CLINYC score.

References

  1. Grasselli G, Zangrillo A, Zanella A, Antonelli M, Cabrini L, Castelli A, et al. Baseline Characteristics and Outcomes of 1591 Patients Infected With SARS-CoV-2 Admitted to ICUs of the Lombardy Region, Italy. JAMA. 2020; 323: 1574-1581.
  2. Fernández Villalobos NV, Ott JJ, Klett-Tammen CJ, Bockey A, Vanella P, Krause G, et al. Effect modification of the association between comorbidities and severe course of COVID-19 disease by age of study participants: a systematic review and meta-analysis. Syst Rev. 2021; 10: 194.
  3. Cai Z, Yang Y, Zhang J. Obesity is associated with severe disease and mortality in patients with coronavirus disease 2019 (COVID-19): a meta-analysis. BMC Public Health. 2021; 21: 1505.
  4. Corona G, Pizzocaro A, Vena W, Rastrelli G, Semeraro F, Isidori AM, et al. Diabetes is most important cause for mortality in COVID-19 hospitalized patients: Systematic review and meta-analysis. Rev Endocr Metab Disord. 2021; 22: 275-296.
  5. Nuño M, García Y, Rajasekar G, Pinheiro D, Schmidt AJ. COVID-19 hospitalizations in five California hospitals: a retrospective cohort study. BMC Infect Dis. 2021; 21: 938.
  6. Cannistraci CV, Valsecchi MG, Capua I. Age-sex population adjusted analysis of disease severity in epidemics as a tool to devise public health policies for COVID-19. Sci Rep. 2021; 11: 11787.
  7. Katzenschlager S, Zimmer AJ, Gottschalk C, Grafeneder J, Schmitz S, Kraker S, et al. Can we predict the severe course of COVID-19 – a systematic review and meta-analysis of indicators of clinical outcome? PLoS One. 2021; 16: e0255154.
  8. Synolaki E, Papadopoulos V, Divolis G, Tsahouridou O, Gavriilidis E, Loli G, et al. The Activin/Follistatin Axis Is Severely Deregulated in COVID-19 and Independently Associated With In-Hospital Mortality. J Infect Dis. 2021; 223: 1544-1554.
  9. Taylor EH, Marson EJ, Elhadi M, Macleod KDM, Yu YC, Davids R, et al. Factors associated with mortality in patients with COVID-19 admitted to intensive care: a systematic review and meta-analysis. Anaesthesia. 2021; 76: 1224-1232.
  10. Ulloque-Badaracco JR, Ivan Salas-Tello W, Al-Kassab-Córdova A, Alarcón-Braga EA, Benites-Zapata VA, Maguiña JL, et al. Prognostic value of neutrophil-to-lymphocyte ratio in COVID-19 patients: A systematic review and meta-analysis. Int J Clin Pract. 2021, 75: e14596.
  11. Docherty AB, Harrison EM, Green CA, Hardwick HE, Pius R, Norman L, et al. Features of 20 133 UK patients in hospital with covid-19 using the ISARIC WHO Clinical Characterisation Protocol: prospective observational cohort study. BMJ. 2020; 369: m1985.
  12. Ji D, Zhang D, Xu J, Chen Z, Yang T, Zhao P, et al. Prediction for Progression Risk in Patients With COVID-19 Pneumonia: The CALL Score. Clin Infect Dis. 2020; 71: 1393-1399.
  13. Lim WS, van der Eerden MM, Laing R, Boersma WG, Karalus N, Town GI, et al. Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax. 2003; 58: 377-382.
  14. Tomlins J, Hamilton F, Gunning S, Sheehy C, Moran E, MacGowan A. Clinical features of 95 sequential hospitalised patients with novel coronavirus 2019 disease (COVID-19), the first UK cohort. J Infect. 2020; 81: e59-e61.
  15. Liang W, Liang H, Ou L, Chen B, Chen A, Li C, et al. Development and Validation of a Clinical Risk Score to Predict the Occurrence of Critical Illness in Hospitalized Patients With COVID-19. JAMA Intern Med. 2020; 180: 1081-1089.
  16. Altschul DJ, Unda SR, Benton J, de la Garza Ramos R, Cezayirli P, Mehler M, et al. A novel severity score to predict inpatient mortality in COVID-19 patients. Sci Rep. 2020; 10: 16726.
  17. Gao Y, Cai GY, Fang W, Li HY, Wang SY, Chen L, et al. Machine learning based early warning system enables accurate mortality risk prediction for COVID-19. Nat Commun. 2020; 11: 5033.
  18. Yadaw AS, Li YC, Bose S, Iyengar R, Bunyavanich S, Pandey G. Clinical features of COVID-19 mortality: development and validation of a clinical prediction model. Lancet Digit Health. 2020; 2 :e516-e525.
  19. Obremska M, Pazgan-Simon M, Budrewicz K, Bilaszewski L, Wizowska J, Jagielski D, et al. Simple demographic characteristics and laboratory findings on admission may predict in-hospital mortality in patients with SARS-CoV-2 infection: development and validation of the covid-19 score. BMC Infect Dis. 2021; 21: 945.
  20. Raad M, Gorgis S, Abshire C, Yost M, Dabbagh MF, Chehab O, et al. COVID-19 risk index (CRI): a simple and validated emergency department risk score that predicts mortality and the need for mechanical ventilation. J Thromb Thrombolysis. 2022; 53: 567-575.
  21. Her AY, Bhak Y, Jun EJ, Yuan SL, Garg S, Lee S, et al. A Clinical Risk Score to Predict In-hospital Mortality from COVID-19 in South Korea. J Korean Med Sci. 2021; 36: e108.
  22. Wynants L, Van Calster B, Collins GS, Riley RD, Heinze G, Schuit E, et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. BMJ. 2020; 369: m1328.
  23. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Ann Intern Med. 2015; 162: 55-63.
  24. Chu K, Alharahsheh B, Garg N, Guha P. Evaluating risk stratification scoring systems to predict mortality in patients with COVID-19. BMJ Health Care Inform. 2021; 28: e100389.
  25. Moon HJ, Kim K, Kang EK, Yang HJ, Lee E. Prediction of COVID-19-related Mortality and 30-Day and 60-Day Survival Probabilities Using a Nomogram. J Korean Med Sci. 2021; 36: e248.
  26. Fukuda Y, Tanaka A, Homma T, Kaneko K, Uno T, Fujiwara A, et al. Utility of SpO2/FiO2 ratio for acute hypoxemic respiratory failure with bilateral opacities in the ICU. PLoS One. 2021; 16: e0245927.
  27. Alkofide H, Almohaizeie A, Almuhaini S, Alotaibi B, Alkharfy KM. Tocilizumab and Systemic Corticosteroids in the Management of Patients with COVID-19: A Systematic Review and Meta-Analysis. Int J Infect Dis. 2021; 110: 320-329.
  28. Kyriazopoulou E, Poulakou G, Milionis H, Metallidis S, Adamis G, Tsiakos K, et al. Early treatment of COVID-19 with anakinra guided by soluble urokinase plasminogen receptor plasma levels: a double-blind, randomized controlled phase 3 trial. Nat Med. 2021; 27: 1752-1760.