J Thorac Cardiovasc Surg 2004;127:174-178
© 2004 The American Association for Thoracic Surgery
Surgery for acquired cardiovascular disease |
Coronary artery bypass grafting: Are risk models developed from on-pump surgery valid for off-pump surgery?
YingXing Wu, MDa,*,
Gary L. Grunkemeier, PhDa,
John R. Handy, Jr, MDa
a Providence Health System, Portland, Ore, USA.
Received for publication June 19, 2003; revisions received August 13, 2003; accepted for publication August 21, 2003.
* Address for reprints: YingXing Wu, MD, 9205 SW Barnes, #33LL, Portland, OR 97225, USA
yingxing.wu{at}providence.org
 |
Abstract
|
|---|
OBJECTIVE: This study was undertaken to test whether risk models developed from on-pump coronary artery bypass grafting are valid for assessing the risk for off-pump coronary artery bypass grafting.
METHODS: From January 1997 through June 2002, a total of 12,845 patients underwent isolated coronary artery bypass grafting procedures in Providence Health System hospitals. Of these, 1782 operations (14%) were performed without cardiopulmonary bypass. An operative mortality risk model was derived from on-pump data with logistic regression. This model and two other external risk models developed from on-pump data were then applied to patients undergoing off-pump coronary artery bypass grafting to test the model adequacy.
RESULTS: Good model discrimination and calibration were obtained from all three models.
CONCLUSION: Operative mortality risk models developed from on-pump coronary artery bypass grafting can be used to assess the risk for off-pump coronary artery bypass grafting.
In a recent editorial on coronary artery bypass grafting (CABG), accompanying an article by Mack and colleagues1 that presented the results of off-pump CABG (OPCAB), Bonchek2 posed an interesting question: "Are the risk models in the Society of Thoracic Surgeons (STS) National Database, which are derived from experience with on-pump CABG, valid for assessing risk for patients undergoing OPCAB?" We tried to find the answer from an analysis of 5 years' experience with 12,845 isolated CABG surgeries in the Providence Heath System (PHS) collaborative cardiac database.
|
Methods
|
|---|
Patients
From January 1997 through June 2002, a total of 12,845 patients underwent isolated CABG procedures in 9 PHS hospitals. Of these, 1782 operations (14%) were performed without cardiopulmonary bypass. Table 1 compares the major features of the on-pump CABG and OPCAB groups. All nine PHS hospitals participating in the cardiac surgery programs collected the data according to the same definitions, and the data were merged into the PHS collaborative cardiac database (see Appendix Table 1 for details).
Statistical methods
To test whether risk models derived from on-pump CABG surgery are valid for assessing the risk of OPCAB, the PHS data were split into two subsets. The patients who underwent on-pump CABG were used as the training set for model development, and the OPCAB group was used as the test set for model validation. An operative mortality risk model was developed by logistic regression with the training set only. Model entry criteria were set at P < .20, and removal criteria were set at P 
.20. The resulting model was then applied to the test set to test the model adequacy for OPCAB. Two other risk models, the Northern New England (NNE) risk model (unpublished model, provided by courtesy of Dr Pamela Jenkins) and the logistic EuroSCORE model,3 (http://www.euroscore.org/logisticEuroSCORE.htm), both of which were developed from on-pump surgery only, were also applied to the patients undergoing OPCAB. We did not apply the Society of Thoracic Surgeons (STS) CABG risk model because not all the risk factors in STS CABG risk model were collected in the PHS collaborative cardiac database.
For both the training and test sets, the area under the receiver operating characteristic curve, also called the C-statistic,4 which evaluates the ability of a model to correctly classify dead and alive, was used to measure model discrimination. The Hosmer-Lemeshow test5 was used to measure model calibration, the agreement between estimated and actual mortality.
Applying external risk models (eg, NNE or EuroSCORE) may lead to overestimation or underestimation of the predicted risk, because perfect matching of risk factors from one system to another is unlikely. Unlike poor discrimination, however, poor calibration can be improved by secondary logistic regression (using the original logit as an independent risk factor)6. We calibrated the two external models so that the overall predicted operative mortality would be equal to the observed.7 The discrimination was unchanged by recalibration.
|
Results
|
|---|
The demographic and clinical characteristics were similar for the on-pump and OPCAB groups (Table 1). P values are not given because P < .05 does not necessarily mean the difference has clinical significance, especially when the sample sizes are large.
The risk model developed from the on-pump training set is described in Appendix Table 2. Twelve variables were found to be risk factors for operative death. The C-statistic for this model was 0.810, which indicated good discrimination, and the P value for the Hosmer-Lemeshow test was .937, which indicated good calibration. For the test set, the C-statistic was 0.781 and the Hosmer-Lemeshow test P value was .511, which also indicated good discrimination and calibration, respectively. Good model discrimination and calibration were also obtained when applying NNE and EuroSCORE risk models to the test set (Table 2). Because the PHS risk model derived from on-pump data works efficiently in predicting operative death for patients undergoing OPCAB, we combined the training set and the test set and produced a final PHS model (Appendix Table 2). We also applied the NNE and the EuroSCORE risk models to the entire set of patients. The C-statistic and the Hosmer-Lemeshow test showed that all the three models performed well (Table 2). We compared the final PHS-predicted risks with the calibrated NNE-predicted risks (Figure 1) and the calibrated EuroSCORE (Figure 2) for all patients undergoing CABG, on and off pump, and good agreement was found between PHS and the two external models.
View larger version (35K):
[in this window]
[in a new window]
|
Figure 1. Scattergram of calibrated NNE predicted risk versus PHS predicted risk. Small gray dots indicate patients alive; large black dots indicate patients dead. Horizontal and vertical axes are in logit scale. Diagonal line is 45° line of perfect agreement. Both gray and black dots are almost equally distributed around diagonal line, which indicates good agreement between calibrated NNE predicted risk and PHS predicted risk. Black dots get denser as logits get larger on both risk scales, which means larger predicted risk is associated with higher mortality.
|
|
View larger version (32K):
[in this window]
[in a new window]
|
Figure 2. Scattergram of calibrated logistic EuroSCORE vs. Providence Health System (PHS) predicted risk. Small gray dots indicate patients alive; large black dots indicate patients dead. Horizontal and vertical axes are in logit scale. Diagonal line is 45° line of perfect agreement. Both gray and black dots are almost equally distributed around diagonal line, which indicates good agreement between calibrated logistic EuroSCORE and PHS predicted risk. Black dots get denser as logits get larger on both risk scales, which means larger predicted risk is associated with higher mortality.
|
|
|
Discussion
|
|---|
The 1996 STS CABG risk model was developed from a cohort of patients who underwent on-pump CABG from 1990 through 1996.8 Bonchek2 asked whether the STS risk models for on-pump CABG were valid to assess the risk of OPCAB, because some variables unique to OPCAB that could affect mortality, morbidity, and the selection of patients for OPCAB were not being collected (eg, intramyocardial coronary arteries, diffuse coronary disease, small and poor-quality target vessels, aortic atherosclerosis, left ventricular hypertrophy). With regard to mortality, we tried to answer the question by using a simple model validation method, data splitting. The risk model for operative mortality derived from our PHS on-pump data performed well with our OPCAB data. This was confirmed when two external risk models that also had been derived from on-pump data were applied. It is true that not all the potential risk factors are always available for selection, and no risk model can identify all the risk factors. The PHS, NNE, and EuroSCORE risk models may have not considered all of the potentially important risk factors for OPCAB (and perhaps not even for on-pump CABG), but although none are ideal, they all worked well in assessing the operative mortality among patients undergoing OPCAB. Sergeant and associates9 used the EuroSCORE risk model10 to evaluate their OPCAB experience and obtained good model discrimination. The NNE risk model used in this study was based on on-pump data only, but the NNE Cardiovascular Disease Study Group has recently compare the risk-adjusted mortality and morbidity between on-pump CABG and OPCAB, and all the patients undergoing CABG were used to develop the logistic regression models11. Though the OPCAB group was excluded from the published 1996 STS risk model, the same set of risk factors for the STS mortality and morbidity risk models have been used to develop the risk models for all patients undergoing CABG; these models were thereafter used to compare risk-adjusted mortality and morbidity between on-pump CABG and OPCAB.12
This study has some technical limitations. We have not validated the comparability of these risk models within the STS Database. Furthermore, our database assigned patients to the on-pump or OPCAB groups solely on the basis of the operation they finally underwent and did not assess the incidence or influence of intraoperative conversion from OPCAB to on-pump CABG. We also did not assess the potential influence of differences in mortality among different surgeons. As pointed out in Bonchek's editorial,2, the report of OPCAB results by Mack and colleagues1 revealed marked differences in mortality among surgeons, and those with the lowest operative mortality rates performed the vast majority of OPCAB procedures. This phenomenon may have influenced the observation of Cleveland and colleagues12 that operative mortality in the STS Database was lower for OPCAB than for on-pump CABG. The influence of the surgeon's skill and experience on the on-pump versus OPCAB operative risk for any individual patient requires further study. Subject to these limitations, however, we conclude that risk model for operative mortality in the PHS Database that was developed from on-pump data can be used for assessing the risk for patients undergoing OPCAB.
Appendix
|
Acknowledgments
|
|---|
We are grateful to Elaine M. Olmstead, BA, and Pamela Jenkins, MD, for providing information on the Northern New England CABG risk model, to S. A. M. Nashef, FRCS, for providing information on the EuroSCORE, and to Lawrence I. Bonchek, MD, for generating the research question and for helpful improvements to the final manuscript. We also thank the Providence Health System hospitals for use of cardiac surgery data from the following institutions: Alaska, Providence Anchorage Medical Center; Washington, Providence Everett Medical Center, Providence St Peter Hospital, and Providence Yakima Medical Center; Oregon, Providence Portland Medical Center and Providence St Vincent Medical Center; and California, Providence St Joseph Medical Center, Providence Holy Cross Medical Center, and Little Company of Mary Hospital.
|
Footnotes
|
|---|
See appendixes on page 178.
|
References
|
|---|
- Mack M, Bachand D, Acuff T, Edgerton J, Prince S, Dewey T, et al. Improved outcomes in coronary artery bypass grafting with beating-heart techniques. J Thorac Cardiovasc Surg. 2002;124:598–607[Abstract/Free Full Text]
- Bonchek LI. Off-pump coronary bypass: is it for everyone? J Thorac Cardiovasc Surg. 2002;124:431–434[Free Full Text]
- Roques F, Nashef SA, Michel P, Gauducheau E, de Vincentiis C, Baudet E, et al. Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg. 1999;15:816–823[Abstract/Free Full Text]
- Grunkemeier GL, Jin R. Receiver operating characteristic curve analysis of clinical risk models. Ann Thorac Surg. 2001;72:323–326[Abstract/Free Full Text]
- Hosmer DW, Lemeshow S. A goodness-of-fit test for the multiple logistic regression model. Commun Stat. 1980;A10:1043–1069
- Harrell FE Jr, Lee KL, Califf RM, Pryor DB, Rosati RA. Regression modelling strategies for improved prognostic prediction. Stat Med. 1984;3:143–152[Medline]
- DeLong ER, Peterson ED, DeLong DM, Muhlbaier LH, Hackett S, Mark DB. Comparing risk-adjustment methods for provider profiling. Stat Med. 1997;16:2645–2664[Medline]
- Shroyer AL, Plomondon ME, Grover FL, Edwards FH. The 1996 coronary artery bypass risk model: the Society of Thoracic Surgeons Adult Cardiac National Database. Ann Thorac Surg. 1999;67:1205–1208[Abstract/Free Full Text]
- Sergeant P, de Worm E, Meyns B, Wouters P. The challenge of departmental quality control in the reengineering towards off-pump coronary artery bypass grafting. Eur J Cardiothorac Surg. 2001;20:538–543[Abstract/Free Full Text]
- Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg. 1999;16:9–13[Abstract/Free Full Text]
- Hernandez F, Cohn WE, Baribeau YR, Tryzelaar JF, Charlesworth DC, Clough RA, et al. In-hospital outcomes of off-pump versus on-pump coronary artery bypass procedures: a multicenter experience. Ann Thorac Surg. 2001;72:1528–1534[Abstract/Free Full Text]
- Cleveland JC Jr, Shroyer AL, Chen AY, Peterson E, Grover FL. Off-pump coronary artery bypass grafting decreases risk-adjusted mortality and morbidity. Ann Thorac Surg. 2001;72:1282–1289[Abstract/Free Full Text]
This article has been cited by other articles:
|
|
|
|
 
J. Brevig, J. McDonald, E. S. Zelinka, T. Gallagher, R. Jin, and G. L. Grunkemeier
Blood transfusion reduction in cardiac surgery: multidisciplinary approach at a community hospital.
Ann. Thorac. Surg.,
February 1, 2009;
87(2):
532 - 539.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
R. Jin, L. F. Hiratzka, G. L. Grunkemeier, A. Krause, and U. S. Page III
Aborted Off-Pump Coronary Artery Bypass Patients Have Much Worse Outcomes Than On-Pump or Successful Off-Pump Patients
Circulation,
August 30, 2005;
112(9_suppl):
I-332 - I-337.
[Abstract]
[Full Text]
[PDF]
|
|
|
|
|
|
|
R. Jin, G. L. Grunkemeier, A. P. Furnary, J. R. Handy Jr, and for the Providence Health System Cardiovascular St
Is Obesity a Risk Factor for Mortality in Coronary Artery Bypass Surgery?
Circulation,
June 28, 2005;
111(25):
3359 - 3365.
[Abstract]
[Full Text]
[PDF]
|
|
|
Top
Abstract
Methods
