HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Johan Nilsson Lars Thulin Samer A.M. Nashef Johan Brandt Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Nilsson, J. Articles by Brandt, J. Search for Related Content PubMed PubMed Citation Articles by Nilsson, J. Articles by Brandt, J. Related Collections Cardiac - other Related Article

J Thorac Cardiovasc Surg 2006;132:12-19
© 2006 The American Association for Thoracic Surgery

Surgery for Acquired Cardiovascular Disease

Risk factor identification and mortality prediction in cardiac surgery using artificial neural networks

^a Department of Cardiothoracic Surgery, Lund University, Lund, Sweden
^b Department of Theoretical Physics, Lund University, Lund, Sweden
^c Competence Centre for Clinical Research, Lund University Hospital, Lund, Sweden
^d Papworth Hospital, Cambridge, UK

Received for publication September 1, 2005; revisions received December 19, 2005; accepted for publication December 29, 2005.

^_* Address for reprints: Dr Johan Nilsson, Department of Cardiothoracic Surgery, Heart and Lung Center, Lund University Hospital, SE 221 85 LUND, Sweden. (Email: johan.nilsson{at}thorax.lu.se).

	Abstract

Top Abstract Introduction Methods Results Discussion Appendix E1 References References E1

 
OBJECTIVE: The artificial neural network model is a nonlinear technology useful for complex pattern recognition problems. This study aimed to develop a method to select risk variables and predict mortality after cardiac surgery by using artificial neural networks.

METHODS: Prospectively collected data from 18,362 patients undergoing cardiac surgery at 128 European institutions in 1995 (the European System for Cardiac Operative Risk Evaluation database) were used. Models to predict the operative mortality were constructed using artificial neural networks. For calibration a sixfold cross-validation technique was used, and for testing a fourfold cross-testing was performed. Risk variables were ranked and minimized in number by calibrated artificial neural networks. Mortality prediction with 95% confidence limits for each patient was obtained by the bootstrap technique. The area under the receiver operating characteristics curve was used as a quantitative measure of the ability to distinguish between survivors and nonsurvivors. Subgroup analysis of surgical operation categories was performed. The results were compared with those from logistic European System for Cardiac Operative Risk Evaluation analysis.

RESULTS: The operative mortality was 4.9%. Artificial neural networks selected 34 of the total 72 risk variables as relevant for mortality prediction. The receiver operating characteristics area for artificial neural networks (0.81) was larger than the logistic European System for Cardiac Operative Risk Evaluation model (0.79; P = .0001). For different surgical operation categories, there were no differences in the discriminatory power for the artificial neural networks (P = .15) but significant differences were found for the logistic European System for Cardiac Operative Risk Evaluation (P = .0072).

CONCLUSIONS: Risk factors in a ranked order contributing to the mortality prediction were identified. A minimal set of risk variables achieving a superior mortality prediction was defined. The artificial neural network model is applicable independent of the cardiac surgical procedure.

	Introduction

Top Abstract Introduction Methods Results Discussion Appendix E1 References References E1

 
Preoperative evaluation of a patient's surgical risk is an important component in cardiac surgery. Risk stratification can provide patients and their families with insight into the existent risk of complications and mortality and guide the selection of cases for surgery versus alternative, nonsurgical therapies. It can also predict the need for hospital care resources in cardiac surgery. ¹ During the last decades, several scoring systems to calculate the mortality risk before the surgery have been developed. ^2-5

Most risk scoring systems have been created using a biostatistical method based on a generalized linear model with assumptions of linear relationship. Artificial neural networks (ANNs) work in a nonlinear fashion, which may better describe the interaction between health risk factors. ANNs have been used in classification and diagnostic prediction of cancer ⁶ and electrocardiogram interpretation, ⁷ among others. Some studies in clinical medicine have demonstrated superiority of the classification or prediction by ANNs compared with other statistical models. ⁸ In the field of cardiac surgery, only a few studies using ANNs have been published, and the results have been ambiguous. ^9-14

To select risk variables for a model, significance testing (P values) is the most common methodology, but this does not assess the importance of the individual variable. ¹⁵ On the other hand, ANNs may be used for both variable selection and ranking of individual variables in order of importance. ¹⁵ For example, this methodology has been employed to select and minimize a large number of gene expression levels used in cancer classification, with excellent results. ¹⁶

This study aimed to systematically evaluate the accuracy and performance of ANNs to select and rank the most important risk factors for operative mortality in cardiac surgery by using high-performance computer clusters.

	Methods

Top Abstract Introduction Methods Results Discussion Appendix E1 References References E1

 
Database
The database used in the present study was that of the multinational European System for Cardiac Operative Risk Evaluation (EuroSCORE) cardiac surgical project. This was a prospective study to assess risk factors for operative mortality, defined as death within 30 days after the operation or within the same hospital admission, ¹⁷ and to construct a risk stratification system. ⁵ The database included 97 risk factors from all patients who underwent cardiac surgery in 128 centers from 8 European countries from September to December 1995. The data collection, quality checks, and validation have been described by Roques and colleagues. ¹⁷ A local database including risk factors for adult patients undergoing cardiac surgery at the Lund University Hospital between January 1996 and February 2001 was used to further evaluate the developed ANN risk model by blind testing.

Patients and Study Design
From the 97 original EuroSCORE variables, a subset of 72 variables was selected (Tables 1 and 2). This was done by excluding variables closely linked to other variables and data collected intraoperatively (ie, number of conduits and number of distal coronary anastomoses). Patients with a missing value in any mandatory variable (age, gender, or surgical procedure) or outcome (operative mortality) were excluded from analysis. Imputation was used to substitute missing values in the other variables with the statistical mode for categorical variables and the mean for continuous variables. ¹¹

View larger version (23K): [in this window] [in a new window]   Figure 1. Schematic illustration of the ANN training and analysis process. The cardiac database was randomly split into 4 groups (I). One of these groups was selected as a test set and excluded from further analysis. The remaining groups were used for the training (II). Following the sixfold cross-validation procedure, the training group was randomly partitioned into 6 new groups of equal size (III). One of these groups was reserved for validation and the rest for the actual training (IV). For each model the calibration were optimized with 200 iterations. The procedure was repeated and a new validation group was selected (VI). After 6 reselections all groups had been used for validation, and the training group was repartitioned into six new groups (III) and the entire procedure repeated. After six repartitions (VII) a new test group was selected and the full training process was repeated for the remaining patients (VIII). Thus, for each of the 4 test sets a complete model selection procedure was carried out (steps III-VII) and the final test result was taken as the average over the 4 test sets or by concatenating the 4 test sets into a complete set of test predictions and computing the ROC area for this entire list.

Statistical Analysis
Mean values (± standard deviation [SD]) were used to describe continuous variables, and frequencies were calculated for categorical variables. Logistic regression analysis was performed to obtain the coefficients for the risk variables included in the logistic model as described by Hosmer and Lemeshow. ²⁰

To compare the number of correctly classified patients by ANNs versus the logistic EuroSCORE, a proportion test was used. Effective odds ratio for the risk variables were determined as described by Lippmann and Shahian. ¹¹ The 95% confidence intervals (CIs) for both the odds ratio and the output from the ANNs were calculated using the bootstrap technique. ^11,21

Receiver operating characteristics (ROC) curves were used to describe the performance and predictive accuracy for the models. ²² The area (with 95% CI) under the curve was used as a quantitative measure of the ability of the risk predictor models to distinguish between survivors and nonsurvivors. To compare the areas under the resulting ROC curves, the nonparametric approach described by DeLong and coworkers ²³ was used.

Computer Cluster and Software
High-performance computing clusters were used to train and evaluate the ANNs. The ANN calibration and analyses were performed with MatLab 7 (2005), Neural Network Toolbox (MathWorks, Natick, Mass). Graphs and statistical analyses were performed using the Intercooled Stata version 9.0 (2005) statistical package (StataCorp LP, College Station, Tex).

	Results

Top Abstract Introduction Methods Results Discussion Appendix E1 References References E1

 
Patient Population
From the EuroSCORE database of 19,030 patients, 668 patients were excluded due to a missing value in any of the mandatory variables (age, gender, or surgical procedure) or outcome (operative mortality). Thus, 18,362 patients were included in the analysis. In 0.7% of the total data points missing values were imputed, as previously described.

The average age was 62.6 ± 10.7 years (range 17-89). The majority of patients were men (72%). Isolated coronary artery bypass grafting (CABG) was performed in 11,628 patients (63%), 4907 (27%) patients had a valve procedure with or without CABG surgery, and 1827 (10%) had miscellaneous procedures. The patient details are described in Table 1. The actual operative mortality was 4.9% (n = 891).

Architecture of Artificial Neural Networks
Approximately 42,500 different ANN models were validated. The architecture for the final validation ANN included 1 hidden layer with 14 nodes, 1 output node, and 6 individual members of the ensemble. This ANN architecture was used in the selection of risk factors utilized for the mortality prediction.

Selection of Risk Factors Utilized for Mortality Prediction
The importance ranking order of the risk variables for the ANN model is presented in Tables 1 and 2 and Figure 2, A. To optimize the model, an increasing number of the ranked variables was included in the model, starting with the top-ranked variable. The largest validation ROC area, 0.82 (95% CI: 0.80-0.83), was achieved when the 34 top-ranked risk variables were included (Figure 2, B).

View larger version (18K): [in this window] [in a new window]   Figure 2. A, The graph shows the difference (%) in the validation ROC area (y-axis) from the ANN model including 71 risk variables compared with a model including all 72 risk variables. The x-axis shows the excluded risk variable number (No.), in order of relevance (see Tables 1 and 2). B, The solid line shows the validation ROC area (y-axis) from the ANNs with different number of included risk variables (x-axis). Dashed lines indicate 95% confidence limits.

Performance and Predictive Accuracy for the Algorithms
The discriminatory power (ie, the area under the ROC curve) for operative mortality was significantly larger for the final ANNs, 0.81 (95% CI: 0.79-0.82) compared with the logistic EuroSCORE model, 0.79 (95% CI: 0.78-0.81; ² = 15.7; P = .0001; Figure 3). The final ANN ROC area was also significantly larger than the ROC area for a logistic model with the same 34 top-ranked risk variables, 0.80 (95% CI: 0.78-0.81; ² = 17.5, P < .0001).

View larger version (18K): [in this window] [in a new window]   Figure 3. The ROC curves from the test data set. The ANNs (solid line) and the logistic EuroSCORE (dashed line) risk stratification algorithms. The area under the curve for ANNs is larger compared with the logistic EuroSCORE. 2 = 15.7; P = .0001.

Effective Odds Ratio and Classification Confidence Limits for ANNs
The effective odds ratio for the 34 top-ranked risk variables is presented in Table 1. Bootstrap sampling was used to generate CIs for both the effective odds ratio (Table 1) and the ANN classification (Figure 4). Thus, an individual patient with a calculated mortality risk of 0.64 belongs with 95% certainty to the group of patients with a greater likelihood of not surviving the operation than surviving the surgery. For a patient with a calculated risk of 0.31, the opposite holds true.

View larger version (26K): [in this window] [in a new window]   Figure 4. Graph of predicted mortality risk (x-axis) (black dots) and the 95% confidence limits (y-axis) (gray dots) calculated by the ANNs for each individual patient.

	Discussion

Top Abstract Introduction Methods Results Discussion Appendix E1 References References E1

The search for an effective method for mortality prediction in cardiac surgery started in the 1980s. ²⁴ During the last decades, several risk score algorithms for cardiac surgery have been published, ^2,4,5,19 but it still remains difficult to risk stratify individual patients. ²⁵ Most risk scoring systems have been developed using a biostatistical method based on a generalized linear model. Different methods to improve the accuracy of risk algorithms have been suggested (eg, include more patients with higher risk, select and identify the most important risk factors, and the use of new algorithmic models such as machine learning techniques of which ANNs are an example). ^26-28

Only a few studies have investigated ANNs in the prediction of survival after cardiac surgery. ^9-14 Most of these are based on CABG-only patients ^9,11-14 and only 1 included all cardiac surgical procedures. ¹⁰ None of these studies found any considerable improvement over the traditional biostatistical methods. Orr ¹⁰ and Tu and colleagues ⁹ showed that an ANN could be used to estimate cardiac surgical mortality, but the performance was equivalent to that of logistic regression. These studies were made on smaller cohorts than the present (1477 and 4782 patients) and used few risk variables (7 and 11). Lippmann and Shahian ¹¹ obtained a similar result when ANNs were used on patients from the Society of Thoracic Surgeons database. Despite that 80,000 patients with 32 risk variables were included in the study, the ANNs showed a performance equivalent to the other prediction models. The authors concluded that no complex nonlinear relationship exists, at least not among the presented risk variables. Similar to the other studies, almost all variables were categorical, and the variable selection was performed in a classic way, by significance testing (P value). However, identifying a nonlinear relationship is more likely in continuous than categorical variables. Important risk variables for ANNs may also go unrecognized if traditional statistical variable selection is used.

One fundamental and controversial question is the number of variables optimally included in a risk model. In the present study a total of 72 variables (11 continuous) were used. No prior variable selection such as significance testing was used; instead, the ANNs ranked every variable in order of its importance for the mortality prediction. In a second step, the total number of variables was minimized to include only variables with a positive contribution to the outcome prediction. The largest ROC area was achieved when the 34 top-ranked variables were included in the model. If more variables were included, the discriminatory power decreased.

Five of the studies of ANNs in cardiac surgery used ROC analysis to describe the accuracy and the discrimination for the different models ^9,11-14 and 1 compared the number of correctly classified patients. ¹⁰ Even if comparison of ROC curves in a statistically valid fashion to evaluate models remains controversial, the ROC curve is currently the best-developed statistical tool for describing performance. ²² Importantly, the ROC curve for the ANN model is consistently above the logistic EuroSCORE ROC curve, making direct comparison possible. When applying a statistical model to clinical practice, cutoff values for sensitivity and specificity are valuable. ANNs performed significantly better than the logistic EuroSCORE at sensitivity cutoffs of 25%, 50%, and 75%. At a sensitivity of 75%, the ANNs classified 720 more survivors correctly than the logistic EuroSCORE model did.

The predictive accuracy of different risk scoring systems may be influenced by numerous factors, such as differences in variable definitions, management of incomplete data fields, surgical procedure selection criteria, and geographical differences in patient risk factors. The prevalence of risk factors in patients referred for heart surgery may also change over time.

The advantages of ANNs are that they do not require any a priori assumptions or knowledge of underlying frequency distributions, have the capacity to model complex nonlinear relationships, and are robust and tolerant of missing data and input errors. ¹¹

Earlier studies on risk analysis in cardiac surgery have mostly been developed and validated on isolated CABG-only patients ^2,4 or all cardiac surgery. ¹⁷ Recently the Northern New England Cardiovascular Disease Study Group presented a risk model for aortic valve surgery and another for mitral valve surgery. ²⁹ Analyses comparing risk score performance in different surgical procedures have been lacking. In the present study, the ANNs show a similar performance independent of the surgical procedure, unlike the logistic EuroSCORE model. This may be explained not only by a better risk factor selection but also by the capacity of ANNs to recognize complex nonlinear relationships.

Strengths of the present study are that the ANNs were developed on a large multi-institutional database from 8 European countries, that the patient data were quality-checked and validated by 2 independent operators before it was entered into the database, ¹⁷ that a large number (42,500) of combinations of parameters included in the ANNs architecture could be evaluated by using high-performance computer clusters, and that an independent blind test was performed on a second, external database. A limitation of the present study is that it was performed on data collected 10 years ago. However, a similar result was obtained in the blind test, where the surgical procedures were performed between 1996 and 2001. Hierarchical generalized linear modeling, which accounts for clustering of observations within providers, may improve the results of the logistic regression. ³⁰ This method may be particularly useful to rate provider performance.

The additive EuroSCORE algorithm ⁵ can be used at the bedside without a computer, and the logistic EuroSCORE ¹⁹ is available on the Internet (http://www.euroscore.org). The ANN model cannot compete with the additive model in simplicity, but it is feasible to make it available on a website.

	Appendix E1

Top Abstract Introduction Methods Results Discussion Appendix E1 References References E1

 
Calibration and Validation of the Artificial Neural Network Models
An ensemble approach was used, where several artificial neural networks (ANNs) were combined into a single prediction model. The individual members of the ensemble were standard multilayer perceptrons (MLPs) with 1 hidden layer and 1 output node that was used to encode the operative mortality. ¹ Each MLP was trained using conjugate gradient descent applied to a mean square error function. To avoid overtraining and improve the generalization performance, a weight decay regularization term was utilized. The output of the neural network ensemble was computed as the mean of the output of the individual members in the ensemble. The model selection (ie, the procedure to find the optimal set of model parameters and to select important risk variables) was performed using a sixfold cross-validation procedure (see Figure 1). The model selection procedure explored a large number of different parameter settings by using high-performance computer clusters, including the size of the MLPs, weight decay parameters, size of the ensemble, and risk variable selection (see below).

The final prediction models were tested on patients not previously exposed to the ANNs or the logistic EuroSCORE, by using a fourfold cross-testing technique. Thus, the patient material was randomly split into 4 groups. One of these groups was selected as a test set and excluded from further analysis. The remaining groups were used for calibration and validation. This procedure was performed 4 times with a new group selected each time as a test set (see Figure 1).

Selection of Risk Factors Utilized for Mortality Prediction
To select the most important risk variables and to minimize the number of variables included in the final model, a risk variable ranking was performed. A baseline receiver operating characteristics (ROC) area (see below) was created using all 72 variables. The ranking list was then obtained by measuring the change of the ROC area, as compared with the baseline, when a risk variable was excluded from the model. The highest-ranked variable corresponded to the largest decrease of the ROC area when it was excluded from the model. Each of the models lacking one of the risk variables was recalibrated prior to the ROC area assessment. To optimize the model an increasing number of the ranked variables was included in the model, starting with the top-ranked variable. In this procedure the ANNs were recalibrated after every second variable inclusion.

Effective Odds Ratio and Confidence Intervals for the ANN Output
The odds ratio for a specific risk variable in each patient was determined by changing the risk variable in the patient from "absent" to "present" and calculating the odds for the two conditions. By computing the geometric mean for the odds ratio from all patients, an effective odds ratio for the specific variable was obtained. ² The 95% confidence limit for both the odds ratio and the output from the ANNs were calculated using the bootstrap technique. ^2,3

From the original database, 1750 bootstrap training data sets were created by resampling with replacement. These bootstrap training sets were then used to calibrate new ANN models with the same architecture and parameters settings as for the final ANN risk prediction model. Each ANN model generated an odds ratio for each variable, resulting in 1750 odds ratios for each variable. Standard techniques ^2,3 were then used to extract the confidence limits from these sets of odds ratios. The confidence limits for the mortality risk of individual patients were calculated in the same way.

Computer Cluster
Three clusters for high-performance computing were used to train and evaluate the ANNs. Two Linux clusters hosted by Lunarc at Lund University, one with 210 AMD Opteron nodes and one with 184 Intel P4 nodes, and one Mac OS X cluster with 7 nodes were employed. The latter was also used for the statistical analysis.

See related editorial on page 8.

 

	Acknowledgments

 
We thank Per-Anders Wernberg for fruitful discussions and assistance with computer questions.

	Footnotes

 
Supported by grants from the Swedish Heart Lung Foundation, Anna Lisa and Sven-Eric Lundgrens foundation for medical research, and by computer resources from Lunarc at Lund University.

	References

Top Abstract Introduction Methods Results Discussion Appendix E1 References References E1

 

1. Nilsson J, Algotsson L, Hoglund P, Luhrs C, Brandt J. EuroSCORE predicts intensive care unit stay and costs of open heart surgery. Ann Thorac Surg 2004;78:1528-1534discussion 1534-5..[Abstract/Free Full Text]
   
2. Eagle KA, Guyton RA, Davidoff R, Ewy GA, Fonger J, Gardner TJ, et al. ACC/AHA guidelines for coronary artery bypass graft surgery: executive summary and recommendations: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines (Committee to revise the 1991 guidelines for coronary artery bypass graft surgery). Circulation 1999;100:1464-1480.[Free Full Text]
   
3. Shroyer AL, Coombs LP, Peterson ED, Eiken MC, DeLong ER, Chen A, et al. The Society of Thoracic Surgeons. 30-day operative mortality and morbidity risk models. Ann Thorac Surg 2003;75:1856-1864discussion 1864-5.[Abstract/Free Full Text]
   
4. Hannan EL, Kilburn Jr H, Racz M, Shields E, Chassin MR. Improving the outcomes of coronary artery bypass surgery in New York State. JAMA 1994;271:761-766.[Abstract/Free Full Text]
   
5. Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow A, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg 1999;16:9-13.[Abstract/Free Full Text]
   
6. Burke HB, Goodman PH, Rosen DB, Henson DE, Weinstein JN, Harrell Jr FE, et al. Artificial neural networks improve the accuracy of cancer survival prediction. Cancer 1997;79:857-862.[Medline]
   
7. Heden B, Ohlin H, Rittner R, Edenbrandt L. Acute myocardial infarction detected in the 12-lead ECG by artificial neural networks. Circulation 1997;96:1798-1802.[Abstract/Free Full Text]
   
8. Baxt WG. Application of artificial neural networks to clinical medicine. Lancet 1995;346:1135-1138.[Medline]
   
9. Tu JV, Weinstein MC, McNeil BJ, Naylor CD. Predicting mortality after coronary artery bypass surgery. what do artificial neural networks learn?. The Steering Committee of the Cardiac Care Network of Ontario. Med Decis Making 1998;18:229-235.
   
10. Orr RK. Use of a probabilistic neural network to estimate the risk of mortality after cardiac surgery. Med Decis Making 1997;17:178-185.[Abstract/Free Full Text]
    
11. Lippmann RP, Shahian DM. Coronary artery bypass risk prediction using neural networks. Ann Thorac Surg 1997;63:1635-1643.[Abstract/Free Full Text]
    
12. Ennett CM, Frize M. Weight-elimination neural networks applied to coronary surgery mortality prediction. IEEE Trans Inf Technol Biomed 2003;7:86-92.[Medline]
    
13. Buzatu DA, Taylor KK, Peret DC, Darsey JA, Lang NP. The determination of cardiac surgical risk using artificial neural networks. J Surg Res 2001;95:61-66.[Medline]
    
14. Chong CF, Li YC, Wang TL, Chang H. Stratification of adverse outcomes by preoperative risk factors in coronary artery bypass graft patients. an artificial neural network prediction model. Proc AMIA Symp 2003:160-164Nov.
    
15. Warren S. How to measure importance of inputs? 2000, SAS Institute Inc. June 23, 2000. Available at: ftp://ftp.sas.com/pub/neural/importance.html. Accessed June 23, 2005..
    
16. Khan J, Wei JS, Ringner M, Saal LH, Ladanyi M, Westermann F, et al. Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks. Nat Med 2001;7:673-679.[Medline]
    
17. Roques F, Nashef SA, Michel P, Gauducheau E, de Vincentiis C, Baudet E, et al. Risk factors and outcome in European cardiac surgery. analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg 1999;15:816-822discussion 822-3.[Abstract/Free Full Text]
    
18. Cross SS, Harrison RF, Kennedy RL. Introduction to neural networks. Lancet 1995;346:1075-1079.[Medline]
    
19. Michel P, Roques F, Nashef SA. Logistic or additive EuroSCORE for high-risk patients?. Eur J Cardiothorac Surg 2003;23:684-687.[Abstract/Free Full Text]
    
20. Hosmer DW, Lemeshow S. Model-building strategies and methods for logistic regression. In: Hosmer DW, Lemeshow S, editors. Applied logistic regression. 2 ed.. New York: John Wiley & Sons; 2000. pp. 91-134.
    
21. Wehrens R, Putter H, Buydens L. The bootstrap. a tutorial. Chemom Intell Lab Syst 2000;54:35-52.
    
22. Pepe MS. The receiver operating characteristic curve. In: Pepe MS, editor. The statistical evaluation of medical tests for classification and prediction. New York: Oxford University Press; 2003. pp. 92-94.
    
23. DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves. a nonparametric approach. Biometrics 1988;44:837-845.[Medline]
    
24. Parsonnet V, Dean D, Bernstein AD. A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease. Circulation 1989;79:I3-I12.[Medline]
    
25. Pinna-Pintor P, Bobbio M, Colangelo S, Veglia F, Giammaria M, Cuni D, et al. Inaccuracy of four coronary surgery risk-adjusted models to predict mortality in individual patients. Eur J Cardiothorac Surg 2002;21:199-204.[Abstract/Free Full Text]
    
26. Jones RH, Hannan EL, Hammermeister KE, Delong ER, O'Connor GR, Luepker RV, et al. Identification of preoperative variables needed for risk adjustment of short-term mortality after coronary artery bypass graft surgery. The Working Group Panel on the Cooperative CABG Database Project. J Am Coll Cardiol 1996;28:1478-1487.
    
27. Omar RZ, Ambler G, Royston P, Eliahoo J, Taylor KM. Cardiac surgery risk modeling for mortality. a review of current practice and suggestions for improvement. Ann Thorac Surg 2004;77:2232-2237.[Abstract/Free Full Text]
    
28. Wyse RK, Taylor KM. Using the STS and multinational cardiac surgical databases to establish risk-adjusted benchmarks for clinical outcomes. Heart Surg Forum 2002;5:258-264.[Medline]
    
29. Nowicki ER, Birkmeyer NJ, Weintraub RW, Leavitt BJ, Sanders JH, Dacey LJ, et al. Multivariable prediction of in-hospital mortality associated with aortic and mitral valve surgery in Northern New England. Ann Thorac Surg 2004;77:1966-1977.[Abstract/Free Full Text]
    
30. Shahian DM, Normand SL, Torchiana DF, Lewis SM, Pastore JO, Kuntz RE, et al. Cardiac surgery report cards. comprehensive review and statistical critique. Ann Thorac Surg 2001;72:2155-2168.[Abstract/Free Full Text]
    

	References E1

Top Abstract Introduction Methods Results Discussion Appendix E1 References References E1

1. Cross SS, Harrison RD, Kennedy RL. Introduction to neural networks. Lancet 1995;346:1075-1079.[Medline]
2. Lippmann RP, Shahian DM. Coronary artery bypass risk prediction using neural networks. Ann Thorac Surg 1997;63:1635-1643.[Abstract/Free Full Text]
3. Wehrens R, Putter H, Buydens L. The bootstrap. a tutorial. Chemom Intell Lab Syst 2000;54:35-52.

This article has been cited by other articles:

				A. A. Klein and S. A. M. Nashef Perception and Reporting of Cardiac Surgical Performance Seminars in Cardiothoracic and Vascular Anesthesia, September 1, 2008; 12(3): 184 - 190. [Abstract] [PDF]

				K. Karkouti, D. N. Wijeysundera, W. S. Beattie, and for the Reducing Bleeding in Cardiac Surgery (RBC) Risk Associated With Preoperative Anemia in Cardiac Surgery: A Multicenter Cohort Study Circulation, January 29, 2008; 117(4): 478 - 484. [Abstract] [Full Text] [PDF]

				T. B. Ferguson Jr The "scientific investment" by cardiac surgery J. Thorac. Cardiovasc. Surg., July 1, 2006; 132(1): 8 - 9. [Full Text] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Author home page(s): Johan Nilsson Lars Thulin Samer A.M. Nashef Johan Brandt Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Nilsson, J. Articles by Brandt, J. Search for Related Content PubMed PubMed Citation Articles by Nilsson, J. Articles by Brandt, J. Related Collections Cardiac - other Related Article