ACS-NSQIP – Surgical risk calculator accurately predicts outcomes of laparotomy in a prospective study at a tertiary hospital in Tanzania

Introduction: The American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP) surgical risk calculator is excellent in predicting postoperative complications. Validation in resource-limited settings and applicability in gastrointestinal surgery is still unclear.

Objective: The study aimed to determine the accuracy of the ACS-NSQIP surgical risk calculator in predicting the 30days postoperative adverse outcomes among patients who underwent laparotomy.

Methods: A Single hospital-based, prospective cohort study was done at Muhimbili National Hospital in Tanzania from April 2021 to December 2021 recruiting patients aged 18 and above who underwent primary laparotomy. SR calculator variables were obtained from patients and entered manually to categorize patients into low and high-risk. Patients were followed up for thirty days for the outcome. The predicted risk was compared with actual occurrence to obtain personal risk ratios. The c-statistics of > 0.7, Brier of score < 0.25 and Index of Prediction Accuracy score were used for discrimination, accuracy, and usefulness of the model respectively.

Results: ACS -NSQIP SR-Calculator discriminated well the risks of cardiac complications, re-laparotomy, Anastomotic leak, and death (c-statistic > 0.7) Poor discrimination was observed for the length of hospital stay (c-statistics 0.518). However, SR-calculator shows high calibration potential for all complications with a Brier score < 0.25 (0.002 - 0.144) and an IPA score ranging from 0.225 - 0.969.

Conclusion: The ACS -NSQIP SR-calculator accurately predicted postoperative outcomes for patients requiring laparotomy. SR-Calculator is a reliable tool for preoperative shared decision-making and counseling. The model should be adopted to strengthen the healthcare system in a low-income country.

What is already known about this subject?

➣ The ACS NSQIP surgical risk calculator is accurate in predicting the outcome of the Surgical procedure.

➣ ACS NSQIP surgical risk calculator was validated in the USA

➣ The SRC is associated with surgical care improvement whenever applied.

What are the new findings?

➣ The SRC accurately predicted the 30-day outcomes associated with primary laparotomy except for the length of hospital stay, discharge for other services, and veno-thromboembolic events.

➣ The risk ratio (RR) provides a preliminary estimation of specific complications for a single patient.

➣ Index of prediction Accuracy (IPA) reliably summarizes the discrimination and calibration of the ACS-NSQIP-SR-Calculator.

➣ The SRC can be validated in a resource-limited setting and can be a source of surgical care improvement.

How might these results affect future research or surgical practice?

➣ The results of this study will provide a comparison between our institution and other international situations where a calculator had been tested and surgical service improvement locally. Our findings will provide insights into our loco-regional surgical institutions in the low-income country to conduct cross-specialties and institutional research toward the adoption of ACS-NSQIP-SRC into National surgical improvement programs. Informative to the surgical community on the performance of the SR-calculator outside the USA.

Abbreviations

ACS NSQIP: American College of Surgeons National Surgical Quality Improvement Program; ASA: American Society of Anesthesiologist; AUC: Area Under Curve; BMI: Body Mass Index; BS: Brier Score; CPT: Common Procedural Terminology; ICU: Intensive Care Unit; IPA: Index of Prediction and Accuracy; ROC: Receiver Operator Characteristic curve; SRC: Surgical Risk Calculator

Introduction

Laparotomy conducted in Low-and-Middle Income Countries (LMICs) accounts for 10% of the global total but 70% of associated mortality equivalent to 4.2million deaths annually [1-3]. Postoperative survival and complication rates are highly influenced by the preoperative patient’s characteristics and associated comorbidities [4]. According to The World Health Organization (WHO-1994) declaration, patients have full rights to information including the risk of potential complications. [5,6]. In modern practice, accurate information requires the use of surgical risk stratification tools during the whole of the patient’s encounter.

Several tools have been proposed including the American College of Surgeons National Surgical Quality Improvement Program (ACS-NSQIP) Surgical Risk Calculator (SRC [7]. The SRC utilizes 20 patient-specific domains to predict the outcome of more than 2500 surgical procedures [8]. The SRC had performed well and has been validated across the US since 2013 [9,10]. The SRC has the additional advantage of showing the exact target area for improvement to avert morbidity and mortality in surgical patients [11]. However, the accuracy of the ACS NSQIP surgical risk calculator has been shown to vary among laparotomy patients [12,13] as well as in other non-abdominal surgeries [14,15].

Despite being simple to use and the excellent prediction performance of the calculator in patients who underwent surgery, the model was less tested outside the USA [16]. There is a paucity of information on the applicability of calculators in low-income settings. In spite of laparotomy being a common procedure at Muhimbili National Hospital (MNH), its outcome has not been predicted leaving the judgment on published papers and surgeons' experience. This study, therefore, sought to validate the applicability of the SRC among patients undergoing laparotomy at MNH. The study aimed to determine the accuracy of the ACS-NSQIP surgical risk calculator in predicting the 30days postoperative adverse outcomes among patients who underwent laparotomy.

Materials and methods

Study design and patient selection

The study was conducted prospectively at the department of general surgery of Muhimbili National Hospital in Tanzania. The study population involved patients who underwent laparotomy from 1st April to 31st December 2021. Patients were included If they were eighteen years of age or older, had undergone a primary laparotomy at the index center and had 30 days of clinical follow-up information available, and provided written informed consent to participate. Patients who had laparotomy for trauma and multiple simultaneous procedures laparotomies were excluded. Ethical clearance was obtained from the MUHAS IRB and MNH research, training, and consultancy unit. The research was conducted in accordance with the declaration of Helsinki. We used logistic regression for binary interaction online calculator to estimate the accepted sample size for the study.

Variables of interest and data collection

ACS-NSQIP risk factors were used: age group (< 65, 65–74, 75–84, ≥85); sex (male, female); functional status as obtained from Eastern Cooperative Oncology Group (ECOG) scores, with 0–2 classified as independent, 3 classified as partially dependent, and 4 classifieds as totally dependent; emergency case (yes, no); American Society of Anesthesiologists (ASA) class (1-healthy patient, 2-mild systemic disease, 3-severe systemic disease, 4-severe systemic disease or constant threat to life, 5- moribund or not expected to survive surgery: ASA 5 was removed from the study; steroid use for chronic condition (yes, no); ascites within 30 days prior to surgery (yes, no); systemic sepsis within 48 hours prior to surgery (responses: none, systemic inflammatory response syndrome, sepsis, septic shock); ventilator dependent (yes, no); disseminated cancer (yes, no); diabetes (no, oral, insulin); hypertension requiring medication (yes, no); congestive heart failure in 30 days prior to surgery (yes, no); dyspnea (no, with moderate exertion, at rest); current smoker within 1year (yes, no); history of severe chronic obstructive pulmonary disease (yes, no); dialysis (yes, no); acute renal failure (yes, no); body mass index as extrapolated from World Health Organization (WHO) and Centre for Disease Control and Prevention (CDC) interpretations (Below 18.5 -Underweight, 18.5 – 24.9-Health weight, 25.0 – 29.9-Overweight, 30.0 and above - Obesity).

Current Procedural Terminology codes were assigned after confirming the intended procedure and the code was run in the SRC to predict the risk of a complication occurring. Preoperatively all patients eligible for the study were evaluated and followed for 30-days postoperatively (1st, 5th, 14th, and 30th day respectively ) to obtain the information on the 14-postoperative complications estimated by the ACS-NSQIP SR calculator: Surgical site infection (deep or superficial), death, reoperation, discharge to other care services, readmission due to condition associated with performed surgery, anastomotic leak, cardiac arrhythmias, acute renal failure, pneumonia, length of hospital stay, venous thromboembolic events, any complication, and severe complication. A composite of serious postoperative complications, defined by the ACS-NSQIP, included cardiac arrest, myocardial infarction, pneumonia, progressive renal insufficiency, acute renal failure, venous thromboembolic or pulmonary embolus, return to the operating room, deep incisional surgical site infection, organ space, surgical site infection, systemic sepsis, unplanned intubation, urinary tract infection, and bust abdomen.

Follow-up was achieved in the ward, ICU, clinic, electronic database, and personal mobile phone calls for a patient who failed to attend regular follow-up. The observed occurrence of the outcome was used as a gold standard measure of the study.

Statistical analysis

Statistical analyses were performed using IBM SPSS Statistics version 27.0. Demographics and patients’ characteristics were summarized as mean and standard deviation (SD) for continuous variables and percentage value for categorical variables. The length of postoperative hospital stay was expressed as the median ± interquartile range (IQR). The predicted risk was compared with the actual risk of the outcome using logistic regression, for all the presented statistical analyses, p < 0.05 were considered statistically significant at a 95percent Confidence Interval.

To establish the preliminary performance analysis, a predicted individual patient risk was divided by the ‘‘average risk’’ provided by the SRC to calculate the individual person risk ratio (RR). RR < 1 was considered a low-risk category group and RR > 1 was a high-risk category group. Significant difference results between the two groups were considered at a p < 0.05 at a 95% confidence interval.

The discrimination capacity of the ACS-NSQIP risk calculator was assessed using the area undercurve (c-statistics) and Brier score (BS) to measure how well the regression model differentiates patients at high vs low risk of having an event. The c-statistics was not used to judge the accuracy of the calculator due to a lack of calibration potential (no information regarding whether the overall magnitude of risk is estimated accurately or not). The c-statistic score ranges from 0 to 1 (0 no discrimination, < 0. 5 – Poor model, 0.5 – random guessing, 0.5 – 0.6 fair discrimination, 0.6 – 0.7 moderate good discrimination, 0.7 – 0.8 Good, acceptable discrimination, 0.8 – 0.9 Excellent discrimination, 0.9 – 1 Perfect discrimination) and a value of > 0.7 was used to indicate a model with high discriminative ability. Brier scores combine both model discrimination and calibration to measure model accuracy. The values range between 0 and 1, (0 – Meant perfect Accuracy, 1 – meant perfect inaccuracy, 0.5 – Random guess, and BS value < 0.25 meant the best accurate prediction). A minimum sample size of 289 patients was required to reach the minimum statistical threshold for evaluation of performance with an event per predictor (EPP) of 0.8 and power of 80% at a significant level of p < 0.05 in the logistic regression model as previously described by Pavlou, et al. [17], however, due to ease of data collection we managed to recruit 302 patients.

Null Model Brier Score was found from logistic regression pseudo-R square (Nagelkerke R square) value in an attempt to eliminate possible confounding effects brought by independent variables which may lead to misinterpretation. It assesses the variance effect, assesses the model fits (deviance), and ascertains the predictive performance of the calculator.

To assess the usefulness of a risk calculator for shared-decision making and counseling patients pertaining to the operation, an Index of Prediction Accuracy (IPA) score was determined. The IPA score is a useful score system, that combines the discrimination and calibration performance of the SR-Calculator. It was obtained by one minus the ratio of the Brier score (prediction model) and Null-Brier score (Null model) values. IPA values range from 0 and 1, (1 – Perfect prediction, 0.8 – 0.9 Best, 0.6 – 0.7 good, 0.0 – 0.5 Fair, 0 – useless, < 0 or -negative – harmful) [18].

Results

From April to December 2021, a total of 348 patients underwent laparotomy in the department of general surgery of which, 31 were excluded for having multiple procedures and another 15 for incomplete information. The remaining 302 patients were reviewed and their clinical demography is presented in Table 1 below. There were more male patients with a male-to-female (M: F) ratio of 2:1. The mean age was 47±17.68 (range 18 - 87) years with the majority 246 (81.5%) was under 65 years of age, and 4(1.3%) of the patient was aged > 85 years. Laparotomies done on an emergency basis predominated over the elective cases by a ratio of 2.4:1. Majority 140(46.7%) of the subject had a normal BMI, 14(4.6%) were underweight and 35(11.6%) were obese. Most patients 160(53%) had independent performance status and 5.6% had a dependent performance status.

In Table 2 below, we present the overall occurrence of 30-day complications as observed under two conditions: SRC predicted low risk (RR < 1) and predicted high risk (RR > 1). We report severe complications, the occurrence of any complication, and specific complications including death: a total of 14 events are thus displayed in the table. Severe complications were reported in 82(27.2%) being significantly high in the SRC high-risk group (p = 0.001). The majority of the patients had surgical site infection again being significantly high in the SRC high-risk group (p = 0.001). The morbidity rate of 63.6% and a mortality rate of 39.1% were reported to be 40% higher in the SRC high-risk group, a finding which was significant (p = 0.001). Almost 50% more patients were discharged for other services among the SRC high-risk group but failed to reach significant levels (p = 0.198). Over half of Venous thromboembolism (VTE) was observed in the high-risk group but failed to reach significant levels (p = 0.084).

The SRC failed to predict the length of hospital stay in the high-risk group. Patients who had a high risk of complications stayed less in-hospital compared to those with a low risk of complications. On further analysis of a scatter plot and regression line in Figure 1 below, the surgical risk calculator was found to structurally overestimate those with low risk for prolonged in-hospital stay. (The regression line is equal to y = 11.55+1.07x, R = 0.010, p = 0.089). In the remaining events, SRC had an accurate prediction of adverse event occurrences which were significant however, in terms of duration of hospital stay the prediction capacity of a model we found to be inaccurate.

Figure 1: Scatter plot and regression line for the predicted length of hospital stay versus the observed length of hospital stay of patients who underwent laparotomy between April 2021 - December 2021.

Table 3 below shows the discrimination capacity of the SR-Calculator by using both the c-statistics (AUC) and Brier Score. In the c-statistic (AUC), the SRC showed good discrimination capacity in predicting the occurrence of an anastomotic leak, return to the operating room, death, and cardiac complications. Discrimination and Calibration by Brier score, the SR calculator showed the best accuracy in predicting all the outcomes events. In assessing the usefulness of the SRC for shared-decision making and counseling patients pertaining to the operation, the IPA showed fair to best with the lowest score in predicting anastomotic leak.

In Figure 2 below, ROCs for the outcomes of cardiac complications, death, return to OR and anastomotic leak which were well discriminated are displayed. The AUC was 0.763 and p < 0.001 95% CI (0.664 - 0.862) for cardiac complications, AUC of 0.707 a p < 0.001 95% CI (0.645 – 0.768) for death, AUC of 0.786 and p < 0.001 95% CI (0.618 – 0.751) for anastomotic leak, AUC 0.744 for return to OR with p < 0.001 95% CI (0.673 – 0.815) among 14 outcomes of interest. The specificity of the calculator was > 60% in all outcomes. The lowest NPV was 78.8% among all tested outcomes except for re-admission. Due to the satisfactory specificity of the SR calculator, no adjustments to personal risks (RR) were required.

Figure 2: Displays the C = statistics of four well-discriminated outcome variables obtained from 302 patients who underwent laparotomy between April 2021- December 2021.

Discussion

This was the first study carried out to predict laparotomy outcomes in the region where the operation is common and lifesaving when done on an emergency basis. The ACS-NASQIP tool has not been used in this country before and the risk prediction was therefore arbitrarily based on past publications, which few came from local settings and surgeons' personal experience. This made communication between surgical teams (surgeons and anesthesia) difficult, as well as that with families and patients too. With scarce resources to care for critically ill patients and high-risk surgical cases, a prediction model was highly needed. The SRC calculator utilizes patient-specific characteristics which are easy to collect in any local setup. The CPT codes are not routinely used in local practice but proved handy during this study and can be used for future comparisons of outcomes among similar procedures. Furthermore, the calculator is easy to use in its online format and hence can be adapted for day-to-day use.

The choice to carry out this study prospectively was due to the lack of regular collection of the SRC variables hence would have proven difficult in a retrospective study. However, the study did not intervene when the patient was assigned to any risk category and the care was left with the treating physician. The team managing the patient was believed to have offered the best level of care to each patient being managed and where such a level of care was not offered, it was not due to failure to stratify the case. This relieved researchers of the burden of not intervening in the case. Furthermore, the intervention which was required by the tool was to communicate with the patient and families and not on what is to be done by the case in question.

The mortality and morbidity presented in this study were high but similar to other studies [4]. However, other studies have reported lower rates compared to ours [19,20]. While individual patients’ risks might be comparable and specific across studies, the expertise and level of care are not and this has the potential of limiting comparability between the income divide of high versus low-income settings. Secondly, the study was limited by mixing emergent and elective cases, and procedures with different CPT codes. This ignored the individual risks carried by these two variables on the morbidity and mortality risk.

In spite of the limitations above, this study represents the best attempt to address the outcome following laparotomy in an objective and open manner. The statistics used in this tool have the potential Variability of the data could be due to the low sample size and the presence of other risk factors that cannot be assessed by the SR-calculator. The SRC calculator has the tendency to overestimate those with low risk or intermediate risk to potential outcomes and underestimate the occurrence of events in those with high-risk to outcome in different settings [21]. The calculator overestimated those with a low risk of prolonged hospital stay while underestimating those with a high risk. However, upon elimination of the patients with prolonged hospital stays (> 14 days) the calculator estimated well. The findings consist of a study by Britt et and in 2017. [22].

The SR-Calculator risk estimation based on C-statistics had low performance compared to that in the US when it was originally used [7] but similar to findings by Scotton et al on emergency laparotomy patients [23]. This necessitated the use of the Brier score which added calibration to the tool and had an excellent performance at a cutoff point of BS < 0.25, similar to what was found by other studies which estimated the short-term outcomes of laparotomy [24,25]. Previous studies had relied on the use of ROC-AUC, p-value, or Brier score to make a decision on the tool [26]. Our study has introduced the Index of Prediction and Accuracy (IPA) score in determining the overall predictive performance of the SR calculator and the usefulness of the model in shared decision-making. There was variability in the performance of the calculator in the studied community from excellent performance to fair performance, consistent with what was observed by other studies [21,25]. We observed that the Brier score and IPA had high predictive potential than c-statistics alone contrasting studies on head and neck surgery patients [27].

The study has demonstrated that the SRC can reliably predict the occurrence of complications in surgical patients undergoing laparotomy except for the occurrence of VTE, length of hospital stay, and discharge to other services. The failure to detect VTE, a potentially life-threatening condition, should not be taken lightly as this can represent an underdiagnosis. The tool failed to discriminate the length of hospital stay, shown to be prolonged among patients with a low risk of complication. This need to be studied further to ascertain the specific cause of hospital stay among such patients. While the findings from this study can be generalized to similar settings locally and regionally, we call for multicenter data collection and procedure-specific analysis to further consolidate these findings.

Conclusion

The ACS-NSQIP SR-Calculator performed well in predicting 30-day outcomes after laparotomy, with accurate discrimination and calibration, hence a useful tool for shared decision-making in our study population. We propose the SR calculator be adopted into daily surgical practices in resource-limited settings.

Author contribution

i. T. B. K: Conceptualization, resources, software, investigation, methodology, project administration, data curation, formal analysis, writing an original draft for manuscript development, visualization, writing—a review and editing and submission of the manuscript.

ii. N. E. K: Conceptualization, methodology, data curation, validation, and writing review and editing of the manuscript for submission

iii. F.A.M: Conceptualization, methodology, data curation, validation, visualization, and writing review and editing of the manuscript for submission

iv. F. D. M: Conceptualization, supervision, data curation, and editing of the manuscript for submission.

v. L. O. A: Conceptualization, methodology, supervision, data curation, validation, and writing review and editing of the manuscript for submission

Institutional review board statement

Ethical review and approval were obtained from MUHAS IRB and MNH research, training, and consultancy bureau according to national law.

Informed Consent Statement

Written informed consent was obtained from all subjects involved in the study.

Data availability statement

The data will be made available upon formal and fair request from the corresponding author and the main supervisor: akokole12@gmail.com

Data are available upon reasonable request. Data are available from the corresponding author upon reasonable request.

1. Kivrak S, Haller G. Scores for preoperative risk evaluation of postoperative mortality. Best Pract Res Clin Anaesthesiol. 2021 May;35(1):115-134. doi: 10.1016/j.bpa.2020.12.005. Epub 2020 Dec 8. PMID: 33742572.
2. Wurdeman T, Strader C, Alidina S, Barash D, Citron I, Kapologwe N. In-Hospital Postoperative Mortality Rates for Selected Procedures in Tanzania’s Lake Zone. World J Surg . 45(1):41 to 49. Available from: https://doi.org/10.1007/s00268-020-05802-w
3. Ayenew Z, Gizaw AT, Workneh D, Fentahun N. Outcome of Non-Traumatic Surgical Acute Abdomen in Nekemte Referral Hospital Southwest Ethiopia: A Retrospective Cross-Sectional Study. Surg Curr Res. 2017;07(01):1–5.
4. Pearse RM, Clavien PA, Demartines N, Fleisher LA, Grocott M, Haddow J,. Global patient outcomes after elective surgery: Prospective cohort study in 27 low-, middle- and high-income countries. Br J Anaesth. 2016;117(5):601–9. Available from: http://dx.doi.org/10.1093/bja/aew316
5. Hervey ST, Peers S, Ward A, Hervey T, Mchale J. Fundamental rights and health care. 2012;282–314.
6. Declaration of the promotion of patient's rights in Europe (partim). World Health Organization--Europe, 1994. Acta Chir Belg. 1998 Jan-Feb;98(1):23-5. PMID: 9538917.
7. Bilimoria KY, Liu Y, Paruch JL, Zhou L, Kmiecik TE, Ko CY, Cohen ME. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013 Nov;217(5):833-42.e1-3. doi: 10.1016/j.jamcollsurg.2013.07.385. Epub 2013 Sep 18. PMID: 24055383; PMCID: PMC3805776.
8. Hall BL, Hamilton BH, Richards K, Bilimoria KY, Cohen ME, Ko CY. Does surgical quality improve in the American College of Surgeons National Surgical Quality Improvement Program: an evaluation of all participating hospitals. Ann Surg. 2009 Sep;250(3):363-76. doi: 10.1097/SLA.0b013e3181b4148f. PMID: 19644350.
9. Velanovich V, Rubinfeld I, Patton JH Jr, Ritz J, Jordan J, Dulchavsky S. Implementation of the National Surgical Quality Improvement Program: critical steps to success for surgeons and hospitals. Am J Med Qual. 2009 Nov-Dec;24(6):474-9. doi: 10.1177/1062860609339937. Epub 2009 Jul 7. PMID: 19584373.
10. Bilimoria KY, Liu Y, Paruch JL, Zhou L, Kmiecik TE, Ko CY, Cohen ME. Development and evaluation of the universal ACS NSQIP surgical risk calculator: a decision aid and informed consent tool for patients and surgeons. J Am Coll Surg. 2013 Nov;217(5):833-42.e1-3. doi: 10.1016/j.jamcollsurg.2013.07.385. Epub 2013 Sep 18. PMID: 24055383; PMCID: PMC3805776.
11. Kraemer K, Cohen ME, Liu Y, Barnhart DC, Rangel SJ, Saito JM, Bilimoria KY, Ko CY, Hall BL. Development and Evaluation of the American College of Surgeons NSQIP Pediatric Surgical Risk Calculator. J Am Coll Surg. 2016 Nov;223(5):685-693. doi: 10.1016/j.jamcollsurg.2016.08.542. Epub 2016 Sep 22. PMID: 27666656.
12. Li Z, Coleman J, D'Adamo CR, Wolf J, Katlic M, Ahuja N, Blumberg D, Ahuja V. Operative Mortality Prediction for Primary Rectal Cancer: Age Matters. J Am Coll Surg. 2019 Apr;228(4):627-633. doi: 10.1016/j.jamcollsurg.2018.12.014. Epub 2019 Jan 8. PMID: 30630082.
13. McMillan MT, Allegrini V, Asbun HJ, Ball CG, Bassi C, Beane JD, Behrman SW, Berger AC, Bloomston M, Callery MP, Christein JD, Dickson E, Dixon E, Drebin JA, Fernandez-Del Castillo C, Fisher WE, Fong ZV, Haverick E, Hollis RH, House MG, Hughes SJ, Jamieson NB, Kent TS, Kowalsky SJ, Kunstman JW, Malleo G, McElhany AL, Salem RR, Soares KC, Sprys MH, Valero V 3rd, Watkins AA, Wolfgang CL, Zureikat AH, Vollmer CM Jr. Incorporation of Procedure-specific Risk Into the ACS-NSQIP Surgical Risk Calculator Improves the Prediction of Morbidity and Mortality After Pancreatoduodenectomy. Ann Surg. 2017 May;265(5):978-986. doi: 10.1097/SLA.0000000000001796. PMID: 27232260.
14. Hers TM, Van Schaik J, Keekstra N, Putter H, Hamming JF, Van Der Vorst JR. Inaccurate Risk Assessment by the ACS NSQIP Risk Calculator in Aortic Surgery. J Clin Med. 2021 Nov 20;10(22):5426. doi: 10.3390/jcm10225426. PMID: 34830708; PMCID: PMC8618691.
15. van Schaik J, Hers TM, van Rijswijk CS, Schooneveldt MS, Putter H, Eefting D, van der Vorst JR. Risk assessment in aortic aneurysm repair by medical specialists versus the American College of Surgeons National Surgical Quality Improvement Program risk calculator outcomes. JRSM Cardiovasc Dis. 2021 Apr 8;10:20480040211006582. doi: 10.1177/20480040211006582. PMID: 33889384; PMCID: PMC8040563.
16. Hornor MA, Ma M, Zhou L, Cohen ME, Rosenthal RA, Russell MM, Ko CY. Enhancing the American College of Surgeons NSQIP Surgical Risk Calculator to Predict Geriatric Outcomes. J Am Coll Surg. 2020 Jan;230(1):88-100.e1. doi: 10.1016/j.jamcollsurg.2019.09.017. Epub 2019 Oct 28. PMID: 31672676.
17. Pavlou M, Qu C, Omar RZ, Seaman SR, Steyerberg EW, White IR, Ambler G. Estimation of required sample size for external validation of risk models for binary outcomes. Stat Methods Med Res. 2021 Oct;30(10):2187-2206. doi: 10.1177/09622802211007522. Epub 2021 Apr 21. PMID: 33881369; PMCID: PMC8529102.
18. Kattan MW, Gerds TA. The index of prediction accuracy: an intuitive measure useful for evaluating risk prediction models. Diagnostic Progn Res . 2018;2(1):7. Available from: https://doi.org/10.1186/s41512-018-0029-2
19. Vester-Andersen M, Lundstrøm LH, Møller MH, Waldau T, Rosenberg J, Møller AM; Danish Anaesthesia Database. Mortality and postoperative care pathways after emergency gastrointestinal surgery in 2904 patients: a population-based cohort study. Br J Anaesth. 2014 May;112(5):860-70. doi: 10.1093/bja/aet487. Epub 2014 Feb 10. PMID: 24520008.
20. El Edelbi M, Abdallah I, Jaafar RF, Tamim H, Deeba S, Doughan S. Comparing Emergent and Elective Colectomy Outcomes in Elderly Patients: A NSQIP Study. Int J Surg Oncol. 2021 Dec 6;2021:9990434. doi: 10.1155/2021/9990434. PMID: 34912578; PMCID: PMC8668335.
21. Teoh D, Halloway RN, Heim J, Vogel RI, Rivard C. Evaluation of the American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator in Gynecologic Oncology Patients Undergoing Minimally Invasive Surgery. J Minim Invasive Gynecol. 2017 Jan 1;24(1):48-54. doi: 10.1016/j.jmig.2016.10.005. Epub 2016 Oct 24. PMID: 27789387; PMCID: PMC6614862.
22. Burgess JR, Smith B, Britt R, Weireter L, Polk T. Predicting Postoperative Complications for Acute Care Surgery Patients Using the ACS NSQIP Surgical Risk Calculator. Am Surg. 2017 Jul 1;83(7):733-738. PMID: 28738944.
23. Scotton G, Del Zotto G, Bernardi L, Zucca A, Terranova S, Fracon S, Paiano L, Cosola D, Biloslavo A, de Manzini N. Is the ACS-NSQIP Risk Calculator Accurate in Predicting Adverse Postoperative Outcomes in the Emergency Setting? An Italian Single-center Preliminary Study. World J Surg. 2020 Nov;44(11):3710-3719. doi: 10.1007/s00268-020-05705-w. Epub 2020 Jul 24. PMID: 32710123; PMCID: PMC7527359.
24. Scotton G, Del Zotto G, Bernardi L, Zucca A, Terranova S, Fracon S, Paiano L, Cosola D, Biloslavo A, de Manzini N. Is the ACS-NSQIP Risk Calculator Accurate in Predicting Adverse Postoperative Outcomes in the Emergency Setting? An Italian Single-center Preliminary Study. World J Surg. 2020 Nov;44(11):3710-3719. doi: 10.1007/s00268-020-05705-w. Epub 2020 Jul 24. PMID: 32710123; PMCID: PMC7527359.
25. Rivard C, Nahum R, Slagle E, Duininck M, Isaksson Vogel R, Teoh D. Evaluation of the performance of the ACS NSQIP surgical risk calculator in gynecologic oncology patients undergoing laparotomy. Gynecol Oncol. 2016 May;141(2):281-286. doi: 10.1016/j.ygyno.2016.02.015. Epub 2016 Feb 24. PMID: 26899020; PMCID: PMC5179222.
26. Schneider AL, Deig CR, Prasad KG, Nelson BG, Mantravadi AV, Brigance JS, Langer MP, McDonald MW, Johnstone PA, Moore MG. Ability of the National Surgical Quality Improvement Program Risk Calculator to Predict Complications Following Total Laryngectomy. JAMA Otolaryngol Head Neck Surg. 2016 Oct 1;142(10):972-979. doi: 10.1001/jamaoto.2016.1809. PMID: 27467967.
27. Prasad KG, Nelson BG, Deig CR, Schneider AL, Moore MG. ACS NSQIP Risk Calculator: An Accurate Predictor of Complications in Major Head and Neck Surgery? Otolaryngol Head Neck Surg. 2016 Nov;155(5):740-742. doi: 10.1177/0194599816655976. Epub 2016 Jun 21. PMID: 27329422.