Prediction of prognosis in elderly patients with sepsis based on machine learning (random survival forest)

Abstract

Background

Elderly patients with sepsis have many comorbidities, and the clinical reaction is not obvious. Thus, clinical treatment is difficult. We planned to use the laboratory test results and comorbidities of elderly patients with sepsis from a large-scale public database Medical Information Mart for Intensive Care (MIMIC) IV to build a random survival forest (RSF) model and to evaluate the model’s predictive value for these patients.

Methods

Clinical information of elderly patients with sepsis in MIMIC IV database was collected retrospectively. Machine learning (RSF) was used to select the top 30 variables in the training cohort to build the final RSF model. The model was compared with the traditional scoring systems SOFA, SAPSII, and APSIII. The performance of the model was evaluated by C index and calibration curve.

Results

A total of 6,503 patients were enrolled in the study. The top 30 important variables screened by RSF were used to construct the final RSF model. The new model provided a better C-index (0.731 in the validation cohort). The calibration curve described the agreement between the predicted probability of RSF model and the observed 30-day survival.

Conclusions

We constructed a prognostic model to predict a 30-day mortality risk in elderly patients with sepsis based on machine learning (RSF algorithm), and it proved superior to the traditional scoring systems. The risk factors affecting the patients were also ranked. In addition to the common risk factors of vasopressors, ventilator use, and urine output. Newly added factors such as RDW, type of ICU unit, malignant cancer, and metastatic solid tumor also significantly influence prognosis.

Background

Despite the growing awareness of sepsis, advanced diagnostic methods, broad-spectrum antibiotics, and intensive care, sepsis remains a major public health problem worldwide [1]. Most epidemiological studies on sepsis come from developed countries. It is estimated that worldwide, about 30 million patients are affected by sepsis each year, of which about 5 million patients die [2], accounting for about 20% of global deaths [3]. With the aggravation of the aging society, the incidence of sepsis in the elderly is gradually increasing; sepsis is among the diseases that lead to the highest mortality among elderly patients [4]. Elderly patients have low immunity [5], reduced organ reserve function, comorbidities such as diabetes and coronary heart disease are more common than younger patients [6], and atypical clinical symptoms after infection; thus, it is easy to miss diagnosis or for a misdiagnosis to occur. Sepsis occurs and quickly progresses to multiple organ failure [7]. Thus, the clinical mortality rate is high. In addition, changes in the pharmacokinetics of elderly patients have also made the treatment of sepsis difficult [8]. Furthermore, a prospective cohort study haven illustrated that older sepsis survivors bear a higher burden of persistent disability and 12-month mortality compared with younger patients [9]. Other researches also very clearly demonstrated, elderly patients with sepsis are more likely to have long-term cognitive impairment and dysfunction [10, 11].

The development of medical information technology and the popularization of electronic medical record system provide the basis for the clinical application and evaluation of a prognostic model. Random survival forest (RSF) is a machine learning method based on decision trees. The algorithm uses internal data cross-validation to ensure high prediction accuracy without over-fitting, which is suitable for survival analysis of many diseases [12, 13]. The RSF model need not assume that variable for the influence of the risk function is linear, in addition to this, it can also rank the importance of variables, so as to screen variables with greater importance and reduce the dimension of variables [14, 15], which is beneficial to the application of the model in clinical practice [16, 17]. Maryam et al. have illustrated this point clearly, their research showed that the machine learning prediction model can well predict the major adverse cardiac and cerebrovascular events during long-term follow-up after percutaneous coronary intervention [18]. Sequential Organ Failure Assessment (SOFA), Simplified acute physiological score II (SAPSII), and Acute physiology score III (APSIII)  [19, 20] contain the evaluation of multiple laboratory indicators, which are often used to predict the prognosis of diseases, but they still have certain limitations. Current studies tend to add some new markers on the basis of the abovementioned scoring system [21, 22], or reconstruct the scoring system [23], to improve their performance in predicting disease prognosis.

Researches have shown that early identification and assessment of sepsis is key to improving survival in older patients with sepsis [24, 25]. At present, no study has used the RSF model to predict the prognosis of elderly patients with sepsis. We planned to use the laboratory test results and comorbidities of elderly patients with sepsis from the large-scale public database MIMIC IV to build the RSF model and evaluate its predictive value for elderly patients with sepsis.

Methods

Data source and study population

The MIMIC-IV v0.4 database is a large public database that contains hospitalization information for patients at Beth Israel Deaconess Medical Center between 2008 and 2019, which was approved by the Massachusetts Institute of Technology (Cambridge, MA) and Beth Israel Deaconess Medical Center (Boston, MA). Because the present study was an analysis of the third party anonymized publicly available database with pre-existing institutional review board (IRB) approval, our institution’s IRB approval was exempted. This database provides a strong information base for clinical studies. In the database, the true identity information about the patient is hidden. Thus, obtaining the patient’s informed consent was not needed. The author completed the relevant course training and obtained the certificate to access the database. All data are from Physionet official website (https://mimic.physionet.org/).

A total of 11,897 patients were diagnosed with sepsis in the database, including 6,567 patients aged 65 years old or older. Exclusion criteria were as follows: patients who died within 24 h of entering intensive care unit (ICU). Finally, a total of 6,503 patients were selected for the study.

Data extraction

Using Structured Query Language to extract data, the extracted variables included the general information of patients, as follows: ethnicity, sex, age, weight, ventilator use, vasopressor use, continuous renal replacement therapy (CRRT) use, and first care unit (unit). The severity of the disease was assessed using SOFA, SAPS II, and APS III. Charlson comorbidity index was used, and the comorbidities included the following: myocardial infarction, congestive heart failure, peripheral vascular disease, cerebrovascular disease, dementia, chronic pulmonary disease, rheumatic disease, peptic ulcer disease, mild liver disease, diabetes uncomplicated, diabetes complicated, paraplegia, renal disease, malignant cancer, severe liver disease, metastatic solid tumor, and AIDS. Results of the first laboratory examination after admission to the ICU included data on the following: white blood cells (WBC), red blood cells (RBC), hemoglobin, hematocrit, red cell distribution width (RDW), mean corpuscular hemoglobin (MCH), mean corpuscular hemoglobin concentration (MCHC), mean corpuscular volume (MCV), platelet count (PLT), prothrombin time (PT), partial thromboplastin time (PTT), INR PT, lactate, calculated total CO₂, PaCO₂, pH, PaO₂, alanine aminotransferase (ALT), aspartate aminotransferase (AST), albumin, alkaline phosphatase(AP), bilirubin total, urea nitrogen, creatinine, glucose, anion gap (AG), base excess, calcium total, chloride, magnesium, bicarbonate, phosphate, potassium, sodium, specific gravity, urine output. Vital signs included data on the following: mean heartrate, mean systolic blood pressure, mean diastolic blood pressure, mean blood pressure, mean respiratory rate, mean temperature, and mean SpO₂.

Statistical analysis

In this study, indicators with a missing degree greater than 20% were not included, and the remaining missing data were filled in by multiple imputation. In this study, the final complete data was generated from 10 imputed datasets obtained by the "mice" package of the R software [26].

The elderly patients with sepsis were randomly assigned to the training cohort (80%) or validation cohort (20%). The training cohort was used to construct the RSF model and perform internal validation. The validation cohort was used to verify the performance of the model. Categorical variables were described by frequency and percentage values, and differences between cohorts were determined by the chi-square test or Fisher's exact test. In some statistical guides, it is shown that for descriptive statistics, the median and quartiles are preferred over means and standard deviation values [27]. Therefore, in this study, the median and quartiles are used to describe continuous variables.

RSF is an ensemble method [28], which firstly uses the Bootstrap's sampling method to randomly select N samples from the training cohort to generate N survival trees, and then at each node of the tree, randomly select a subset of the covariates as candidate variables for splitting. Therefore, each tree is composed of categorized or split node variables, where tree nodes are split according to the maximum survival difference between child nodes, which can be calculated by four methods, namely log-rank, conservation of events, log-rank score, and random [15]. The method used in this study is the log-rank. For each bootstrap sample, about 37% of the samples in the training cohort were not extracted on average, and these samples were called out-of-bag (OOB) samples. The OOB error rate of the OOB sample was calculated. The OOB error rate and the predictive error rate of the validation set were used to evaluate the model’s performance. The lower the error rate was, the better the model performance was. In this study, the optimal parameter combination of the model was determined by calculating the error rate of the bag in the training cohort under various parameter combination conditions through grid search [29]. The parameter combination that made the total error rate of the RSF the lowest was determined. RSF model was built according to the optimal parameters, and variables were screened according to variable importance (VIMP)¹⁴. The importance score is an evaluation index used to measure the predictive ability of predictive variables to outcome variables. The greater the VIMP value was, the stronger the predictive ability was. VIMP was positive, indicating that the variable had a predictive effect. A VIMP of 0 or a negative value indicated that the variable was not a meaningful predictor. Ranking was performed according to the score of order of importance from the most important to the least important. The top 30 variables of importance were selected, and the RSF was built again. C index and calibration curves were used to evaluate the performance of the model.

In this study, data analysis was performed using R 4.0.3 software and Python 3.7; the packages used include randomForestSRC, survival, survivalROC, matplotlib, and scikit-learn.

Results

Of 6,503 elder sepsis patients, 5,202 were in the training cohort, and 1,301 were in the validation cohort. The median age of the training cohort was 77.00 (70.00, 83.00), and the median age of the validation cohort was 76.00 (70.00, 83.00). Male patients accounted for 49.9% in the training cohort and 49.4% in the validation cohort. The median weight of patients in the training cohort was 75.00 (63.30, 89.88), and that in the validation cohort was 73.30 (61.90, 88.60). Among the comorbidities, renal disease accounted for the largest proportion, which was 30.5% in the training cohort and 30.0 in the validation cohort. Other baseline characteristics are shown in Table 1. 

Table 1 Baseline characteristics of the patients

Modeling process

We calculated the OOB error rate in the training cohort under various mtry and nodesize combinations by grid search. As shown in Fig. 1a, under the parameter combination condition of mtry = 8 and nodesize = 5, the OOB error rate of the model in the training cohort reached the lowest rate (26.35%), and the OOB error rate of the model tended to be stable at 1000 survival trees. The top 30 variables in the importance diagram of variables (Fig. 2, Supplementary material) were selected to build a random forest model. The optimal mtry = 4 and nodesize = 8 were determined again in the same way (Fig. 1b), and the OOB was 27.30%, and these values were used to build a random forest model.

Fig.1

Tuning parameters of RSF model

Fig. 2

Variable importance and error rate curve of RSF

Modeling validation

The C indexes of the four models (SOFA, SAPSII, APSIII, and RSF) in the validation cohort were as follows: 0.551, 0.654, 0.669, and 0.731, respectively. The calibration curve described the calibration of the RSF model, that is, the agreement between the predicted probability and the observed 30-day survival (Fig. 3).

Fig. 3

Calibration curves for the validation cohort

Discussion

In this present study, we established a prognostic prediction model for predicting 30-day mortality risk in elderly patients with sepsis based on the machine learning (RSF), which can provide a basis for clinical decision-making. Our model is unique, it ranked clinically common laboratory examinations and comorbidities according to the variable importance through RSF, and selected the top 30 variables to build the final RSF model, which is not done in traditional scoring systems. Moreover, we used C index to compare the RSF model with the traditional SOFA, SAPSII, and APSIII scoring system, showing RSF exhibits better predictive performance. The calibration curve further confirmed that the newly constructed RFS model could be used to predict 30-day mortality in elderly patients with sepsis.

Among the variables related to the prediction of sepsis in elderly patients, the top variables are the use of vasopressor, the use of ventilator, the patient’s urine output during the first 24 h, lactate level, and mean systolic blood pressure 24 h after entering the ICU. These are important indicators that can be used to evaluate whether circulatory disorders, respiratory disorders, and other organ dysfunctions occur in elderly patients with sepsis [30, 31]. In addition to lactate, these abovementioned top indicators are also found in SOFA, SAPSII, and APSIII scoring systems, indicating their importance for disease prediction [32]. In recent years, the number of studies about the prognosis of lactate in sepsis has been increasing, because lactate can reflect the degree of hypoxia in patients. For example, one study showed that early detection of lactate was associated with 28-day mortality from sepsis [33].

RDW and the type of ICU unit are some of the new indicators added to the RFS model, which are not included in the traditional scoring system. In recent years, RDW has been of great value as a marker of poor prognosis for diseases of the nervous system, cardiovascular system, and other systemic systems [34,35,36]. The increased value of RDW can indirectly reflect the imbalance of RBC homeostasis, which may be due to the impaired RBC formation ability and abnormal RBC survival caused by the body’s abnormal metabolism [37]. The abovementioned changes in RBC may be due to the large number of inflammatory factors produced in the process of severe metabolic disorder and oxidative stress reaction in sepsis patients.

The patient’s ICU unit reflects the difference in the etiology of sepsis, the more that is known about this the more specific therapies can be, so this also occupies an important part [38].Sepsis can arise from different causes, such as traumatic infection, postoperative infection, and severe pneumonia, which have different effects on the prognosis of patients [39]. These should receive recognition in clinical practice. In addition, malignant cancer and metastatic solid tumor are also new variables. The absolute value of neutrophils in malignant tumors or solid tumors is reduced by intensive cytotoxic chemotherapy, thereby reducing the survival rate of patients [40]. Moreover, the immune system dysfunction that tumors share with sepsis is also associated with lower survival rates in older patients with sepsis [41].

In short, we use RSF to overcome the weaknesses of traditional survival analysis methods to build a model with high predictive performance. With the advent of the medical big data era, machine learning models will be increasingly used in clinical practice to help improve the prognosis of patients [42].

Strengths and limitations of the study

The advantage of this study is that it adopts machine learning method to construct an RSF model which is superior to traditional SOFA, SAPSII, and APSIII scoring system. At the same time, the importance of variables was ranked, so that clinicians can more intuitively understand the indicators that have a greater impact on the outcome. This study also has limitations, first of all, it is a single-center study and lacks external verification. Moreover, when machine learning is applied in clinical practice, the 30-day survival probability of elderly patients with sepsis can be predicted by creating web pages and inputting indicators in the model. One of our limitations is that a complete web page is not generated, which will be improved in future research.

Conclusions

We constructed a prognostic model for predicting 30-day mortality risk in elderly patients with sepsis based on the machine learning (RSF algorithm), and it proved superior to the traditional scoring system. The risk factors affecting the patients were also ranked. In addition to the common risk factors of vasopressors, ventilator use, and urine output. Newly added factors such as RDW, type of ICU unit, malignant cancer, and metastatic solid tumor also significantly influence prognosis.