J Thorac Cardiovasc Surg 2007;134:663-669
© 2007 The American Association for Thoracic Surgery
Surgery for Acquired Cardiovascular Disease |
Cumulative sum failure analysis for eight surgeons performing minimally invasive direct coronary artery bypass
David M. Holzhey, MD*,
Stephan Jacobs, MD,
Thomas Walther, MD, PhD,
Michael Mochalski, MD,
Friedrich W. Mohr, MD, PhD,
Volkmar Falk, MD, PhD
Department of Cardiac Surgery, Heart Center Leipzig, Leipzig, Germany.
Received for publication November 11, 2006; revisions received March 14, 2007; accepted for publication March 20, 2007.
* Address for reprints: David M. Holzhey, MD, Herzzentrum Leipzig, Strümpellstraße 39 04289 Leipzig, Germany. (Email: dholzhey{at}web.de).
 |
Abstract
|
|---|
Objective: Analysis of average and individual surgical performance for minimally invasive direct coronary artery bypass was used to enhance quality control for that operation.
Methods: A total of 1441 standard minimally invasive direct coronary artery bypass procedures performed from August 1996 to January 2006 were analyzed for mortality and 10 other major perioperative complications. Learning curves and assessment of perioperative outcome were calculated using descriptive statistics and cumulative sum observed minus expected failure analysis for 8 involved surgeons with a personal experience ranging from 27 to 443 procedures.
Results: The incidence of in-hospital mortality was 0.9% and compared favorably with the predicted mortality calculated by the logistic EuroSCORE (3.6%, P < .01). Cumulative sum analysis revealed that 2 surgeons crossed the 95% reassurance boundary after 50 operations and that 2 surgeons crossed the 95% reassurance boundary after 100 operations. There were significant differences between surgeons with regard to the learning curves and perioperative complications (3.6%–29.6%, P < .01). Two surgeons crossed the 95% alarm-line indicating unacceptably high failure rates.
Conclusions: Minimally invasive direct coronary artery bypass has become a procedure with low mortality and low complication rates, but results are case-load and surgeon dependent. Cumulative sum analysis is a valuable method allowing for a breakdown of complication rates over time displaying individual surgeons strengths.
Abbreviations and Acronyms CUSUM = cumulative sum; MIDCAB = minimally invasive direct coronary artery bypass
 |
Introduction
|
|---|
For several years, quality assurance has become increasingly important in cardiac surgery. De Leval and colleagues1
and Carthey and colleagues2
outlined the importance of analysis of human factors in cardiac surgery and individual failure analysis. Minimally invasive direct coronary artery bypass (MIDCAB) grafting has been performed since 1996 in a standardized way at our institution. With the high number of MIDCAB procedures performed under similar conditions, it is not only possible to describe the outcome and complication rate of the procedure but also possible to compare individual surgeons performances and learning curves.
The usefulness of the sequential probability cumulative sum (CUSUM) technique to analyze surgical performance has been shown in recent publications.3-6
It allows for detection of changes in perioperative mortality and morbidity during the patient care process. It provides almost real-time monitoring of surgical performance if updated after each procedure.3
CUSUM analysis acknowledges the importance of individual experience in monitoring performance and allows for easy charting of a learning curve with regard to the incidence of perioperative complications. The charts are intuitively readable, but care is needed to avoid misinterpretation.7
The CUSUM method is able to demonstrate changes in the patient care process as a whole regardless of where these changes originate. Comparing results between different surgeons often fails because of the case mix and the variety of variables influencing patient outcome. Several risk-adjusted methods have been suggested for this scenario. In this study, no risk adaptation was used for the reasons outlined in the discussion.
With the quantity of MIDCAB operations at our institution performed under a standardized protocol (equal patient origin and patient selection, standardized surgical technique, identical postoperative management and medication), we found the non–risk-adjusted methods most practical and sufficient to evaluate and compare individual surgical performance over time.
 |
Materials and Methods
|
|---|
From August 1996 to January 2006, 1441 patients underwent MIDCAB at our institution following a standardized protocol. The surgical technique has been described.8
Eight surgeons were involved in the MIDCAB program, achieving a different level of experience ranging from 27 to 443 operations. Five surgeons performed more than 100 operations. Written and electronic files of all patients were screened for demographic data, risk factors, intraoperative parameters, and postoperative short-term and long-term complications and outcome. All data were entered into a database, and standard descriptive statistical and CUSUM analyses were conducted using Microsoft Office Excel (Microsoft Corp, Redmond, Wash) and SPSS 10.0 (SPSS Inc, Chicago, Ill).
 |
CUSUM Analysis
|
|---|
Cumulative failure charts and their use have been described.3-6
In this study, non–risk-adjusted cumulative observed minus expected failure charts were used. The statistical principles were adapted from the comprehensive tutorial by Rogers and colleagues.3
CUSUM was defined as Sn = where Xi = 1 for a "failure" (intraoperative conversion, death, or any major complication as defined in Table 1) and as Xi = 0 for a complete "success" (none of the above complications). The target value p0 was set to 0.1, indicative of an "acceptable failure rate" of 10% according to previous publications4,5
and our own experiences with off-pump coronary artery bypass surgery.
CUSUM curves, together with control boundaries, were calculated and drawn according to the formulas shown in the Appendix. The crossing of an upper boundary was interpreted as an increase of the failure rate to an unacceptably high level of p1 = 0.2, whereby crossing the upper 80% boundary set off a mild alarm to the surgeon and crossing the upper 95% boundary gave reason for more thorough investigation. Crossing the lower 95% boundary led to the conclusion that the complication rate of the particular surgeon was equal to or below the accepted rate of p0. The curve moving in between the boundary lines indicated lack of statistic significance and triggered merely further monitoring (Figures E1, E2, and E3).

View larger version (11K):
[in this window]
[in a new window]
|
Figure E2. After the typical learning process and the lower boundary was crossed, the process seems to be completely in control. After approximately operation 150, an obvious accumulation of failures occurs. The surgeon has built up a credit; thus, the CUSUM curve only enters the inconclusive zone.
|
|

View larger version (11K):
[in this window]
[in a new window]
|
Figure E3. Several authors3 recommend resetting the zero line and the boundaries once the lower boundary is crossed. This way, the visually obvious increase in the failure rate becomes statistically evident.
|
|
Whereas crossing the boundary lines is equivalent to a significant result in statistical testing, the comparison between surgeons is merely visual.
 |
Results
|
|---|
Operation Times
Median operation times were between 100 and 138 minutes. One surgeon (surgeon E) reached a median operation time of 61 minutes. There was a wide range from 40 to 350 minutes, and the duration of a MIDCAB operation can hardly be predicted. It was not possible to construct a "typical" learning curve with regard to operating time for every surgeon. Some operators needed the same average time throughout their whole experience. For the 2 most experienced surgeons (surgeons B and F), the learning curve for the operation time was calculated using logarithmic regression and is shown in Figure 1.
The trend graphs confirm the presumed shape with a relatively steep slope at the beginning. The individual operation times are different, and the trend is only a rough approximation.
Overall Complication Rate
The incidence of in-hospital mortality was 0.9% and compared favorably with the predicted mortality calculated by the logistic EuroSCORE of 3.6% (P < .01). The logistic regression version of the EuroSCORE can be calculated using the same risk factors as those for the EuroSCORE value. For a given patient, the logistic EuroSCORE predicts the risk of perioperative mortality. (For further details, see http://euroscore.org.) The need for reintervention in 4.0% of the patients seems to be relatively high and may partly be explained by the fact that during the first years approximately half of the patients (709 patients) underwent routine postoperative angiography for quality control during the first postoperative days. Thereby, some asymptomatic patients with a less then optimal surgical outcome were detected, and, subsequently, repeat revascularization was performed.
Despite individual differences between surgeons, 3 stages of experience could be identified after analysis of average performance and individual learning curves (see below). After the "learning phase" (<50 operations) perioperative mortality decreased from 1.1% to 0.4% (P = .204). At the end of the "intermediate phase" (up to 100 operations), the overall complication rate and need for reintervention decreased from 12.7% to 7.1% (P < .001) and 5.1% to 2.8% (P < .001), respectively. The following "expert phase" (>100 operations) was characterized by few complications for most surgeons. The need for intraoperative conversion to sternotomy decreased from 2.7% to 0.2% (P < .001) after 150 operations. Figure 2
shows the average rates of mortality, need for intraoperative conversion, need for reintervention, and relative occurrence of any major complication with growing MIDCAB experience.
Individual Surgeons Statistics
Descriptive statistics are widely used to evaluate a surgeons individual performance. The total number of operated cases, the preoperatively mortality risk calculated with the EuroSCORE, and the frequency of complications for every surgeon are listed in Table 2. Although these statistics are easily understood, they reflect only part of the truth. Particularly for surgeons with little experience (surgeons D and H), the mere percentage is of questionable statistical value and may be an exaggeration in either direction.
CUSUM curves react sensitively to irregularities in the patient care process so as to be an early marker of inherent errors in the system. For correct interpretation of the graphs, it is important to note that they do not reflect the patients eventual outcome but merely depict periprocedural complications.
Surgeons with little experience
Surgeon D demonstrated an unacceptably high complication rate after 20 operations. This fact is also reflected in the absolute numbers (Table 2). He has left the program. Surgeon H was performing well, but his total experience was too small to allow for a valid statement. Surgeon C had an experience of approximately 60 operations and an average performance; statistics were still inconclusive (Figure 3).

View larger version (8K):
[in this window]
[in a new window]
|
Figure 3. CUSUM charts of surgeons with little experience (with 80% and 95% boundary lines). CUSUM, Cumulative sum.
|
|
Surgeons with intermediate experience
Surgeon E was facing a steep increase in his complication rate after 70 operations. On closer analysis the main problem was a high frequency of in-hospital reinterventions (surgical or interventional) for anastomosis stenosis. It is interesting that this surgeon required only two thirds of the average operation time. As the surgeon became aware of the problem, the results improved toward the end of the period under review. Surgeon A went through the "typical" learning process up to operation 150. After that, an upward slope of postoperative complications occurred mainly as the result of postoperative bleeding. The surgeon was made aware of the problem by being confronted with his chart and was able to draw conclusions for future operations. The development was similar for surgeon G. After his 100th operation, a steep increase in complications and break of the upper boundary after 150 operations were observed. The main problem was a high number of necessary conversions to sternotomy, partly because of improper patient selection (Figure 4).

View larger version (10K):
[in this window]
[in a new window]
|
Figure 4. CUSUM charts of surgeons with intermediate experience (with 80% and 95% boundary lines). CUSUM, Cumulative sum.
|
|
Surgeons with large experience
Approximately half of all MIDCAB operations at our institution were performed by surgeons B and F. They performed and still perform the operation on a regular basis, which results in an overall low complication rate and a stable performance over time (Figure 5).

View larger version (9K):
[in this window]
[in a new window]
|
Figure 5. CUSUM charts of surgeons with large experience (with 80% and 95% boundary lines). CUSUM, Cumulative sum.
|
|
Influence of the frequency of operations
Some experienced surgeons showed a marked increase in perioperative complications after having passed the initial learning phase. Subsequently, the influence of the frequency with which MIDCAB was performed was investigated. Examples are shown in Figure 6. The CUSUM failure value is plotted against the number of the operations. In addition, the average time between 2 operations in days (the reciprocal of the frequency of this operation) is shown by the solid line and the scale on the right. As can be seen by the density of the operations, surgeon F has regularly performed MIDCAB operations and thus stabilized his failure rate around the expected level of approximately 10%. Surgeon A, initially performing MIDCAB operations regularly, also has an initial learning curve. After that, he has a stable failure rate from operation 25 to operation 150. With decreasing frequency the failure rate increases again (Figure 6).

View larger version (19K):
[in this window]
[in a new window]
|
Figure 6. Influence of the frequency of operations. The same CUSUM curves as in Figures 4 and 5 are plotted against the average time in days (right y-axis) between 2 MIDCAB operations. Although this is not a statistically proven method, the influence of longer time intervals became obvious during the data analysis. CUSUM, Cumulative sum.
|
|
These figures underline the importance of regular specialized surgical practice and training.
 |
Discussion
|
|---|
The importance of quality control in cardiac surgery is widely accepted. Yet it remains a sensitive subject, because it means dealing with failures and imperfections. On the other hand, the thorough analysis of failures can be a powerful means toward improvement of overall performance. Despite the potential for improving the quality of surgical performance, the evaluation of human factors and analysis of surgical errors have found little echo in the cardiac surgical literature. De Leval and colleagues1
pioneered the monitoring of surgical performance in a comprehensive multicenter study in pediatric cardiac surgery. Carthey and colleagues2
published a research review on the same topic. Novick and colleagues4,5,9
recently used the CUSUM method for describing the performance of single surgeons in off-pump coronary artery bypass grafting. The utility of the CUSUM method has also been shown for comparing results in transplant surgery by Rogers and colleagues10
and Axelrod and colleagues.11
The correlation between surgical volume and quality has been investigated, particularly concerning postoperative mortality. Although some groups report data that strongly support the concentration of certain surgical procedures to high-volume centers and surgeons,12-14
others question this evidence because of their own experience.15-18
In this study not only the surgical caseload and the influence on mortality but also the effect of changing frequency of performing MIDCABs was evaluated.
The traditional way of surgical audits with retrospective analysis of outcome data and statistical testing is an appropriate method of confirming outlying performance when the difference has reached a magnitude of statistic significance.19
In contrast, with CUSUM failure analysis, sudden changes in postoperative patient outcome can be detected quickly, and surgeons can be made aware of a deterioration in their performance. CUSUM charts are easily calculated, and their interpretation allows for an "online" monitoring of surgical performance.
Limitations of this study include its retrospective nature in the beginning. From 2005 onward, data were entered prospectively. No risk adjustment was applied. The theoretic and practical advantages of using risk-adjusted curves have been well described.3,20
They are particularly useful whenever an appropriate risk model is available (eg, EuroSCORE for predicted perioperative mortality). In a recent publication, Novick and colleagues20
applied an institutional logistic regression model for adverse outcome to calculate risk-adjusted CUSUM curves. They found an advantage over the non–risk-adjusted curves, particularly in avoiding inappropriate alarm signals, although the clinical significance was only moderate. However, a risk-adjusted analysis can only be as good as the underlying risk prediction model. For retrospective data such as in the current study, we were reluctant to apply a similar risk prediction model because not all possible influencing factors were recorded.
The 3 major conclusions of this study are as follows: (1) Even for experienced surgeons, there is a learning curve for the MIDCAB procedure that levels after approximately 100 operations. (2) Thereafter, performance depends not only on the absolute number of the performed operations ("high volume history") but also on the frequency; that is, the level of skill will not automatically be maintained. (3) Technically demanding operations such as the MIDCAB procedure might not be opportune for every surgeon. These conclusions seem logical and self-evident; however, CUSUM analysis comparing the results of different surgeons provides scientific and statistic evidence for these widely acknowledged presumptions.
 |
Conclusions
|
|---|
MIDCAB has proven to be an excellent method for single-vessel coronary artery disease with a low overall complication rate, a high patency rate, and an excellent long-term survival.21
Still, there is room for improvement. Data are now being updated on a monthly basis and presented to the involved surgeons. Future analyses will show the value of this practice.
 |
Appendix
|
|---|
Adapted from Rogers and colleagues.3
CUSUM charts and boundary lines were constructed using 4 parameters:
p0 the acceptable complication rate (in this article p0 = 0.1)
p1 the unacceptable complication rate (in this article p1 = 0.2)
probability of false alarm, ie, error of assuming that the complication rate has increased to p1 when, in fact, it has not (in this article
= 0.05 and
= 0.2 for upper boundaries)
ß probability of false reassurance, ie, error of assuming that the complication rate has not increased when, in fact, it has (in this article ß = 0.05 and ß = 0.2 for lower boundaries)
If Xi indicates the outcome of the operation I, with Xi = 1 if a serious adverse event occurred and Xi = 0 if it did not, the CUSUM of adverse events was defined as
(1)
The upper boundary l1 (to detect an increase from p0 to p1) and the lower boundary l0 (to assume a failure rate equal or less than p0) with an odds ratio corresponding to an increase in event rate from p0 to p1 were calculated as
(2) l1 = i x (s–p0) + h1 and
(3) l0 = i x (s–p0) – h0
with
(4)
(5)
(6)
(7)
 |
References
|
|---|
- De Leval MR, Carthey J, Wright DJ, Farewell VT, Reason JT. Human factors and cardiac surgery: a multicenter study. J Thorac Cardiovasc Surg 2000;119:661-672.[Abstract/Free Full Text]
- Carthey J, de Leval MR, Reason JT. The human factor in cardiac surgery: errors and near misses in a high technology medical domain. Ann Thorac Surg 2001;72:300-305.[Abstract/Free Full Text]
- Rogers CA, Reeves BC, Caputo M, Ganesh JS, Bonser RS, Angelini GD. Control chart methods for monitoring cardiac surgical performance and their interpretation. J Thorac Cardiovasc Surg 2004;128:811-819.[Free Full Text]
- Novick RJ, Fox SA, Stitt LW, Kiaii BB, Swinamer SA, Rayman R, et al. Assessing the learning curve in off-pump coronary artery surgery via CUSUM failure analysis. Ann Thorac Surg 2002;73:358-362.
- Novick RJ, Stitt LW. The learning curve of an academic cardiac surgeon: use of the CUSUM method. J Card Surg 1999;14:312-320.[Medline]
- Novick RJ, Fox SA, Stitt LW, Swinamer SA, Lehnhardt KR, Rayman R, et al. Cumulative sum failure analysis of a policy change from on-pump to off-pump coronary artery bypass grafting. Ann Thorac Surg 2001;72:1016-1021.
- Blackstone EH. Monitoring surgical performance. J Thorac Cardiovasc Surg 2004;128:807-810.[Free Full Text]
- Diegeler A, Spyrantis N, Matin M, Falk V, Hambrecht R, Autschbach R, et al. The revival of surgical treatment for isolated proximal high grade LAD lesions by minimally invasive coronary artery bypass grafting. Eur J Cardiothorac Surg 2000;17:501-504.[Abstract/Free Full Text]
- Novick RJ, Fox SA, Kiaii BB, Stitt LW, Rayman R, Kodera K, et al. Analysis of the learning curve in telerobotic, beating heart coronary artery bypass grafting: a 90 patient experience. Ann Thorac Surg 2003;76:749.[Abstract/Free Full Text]
- Rogers CA, Ganesh JS, Banner NR, Bonser RS. Cumulative risk adjusted monitoring of 30-day mortality after cardiothoracic transplantation: UK experience. Eur J Cardiothorac Surg 2005;27:1022-1029.[Abstract/Free Full Text]
- Axelrod DA, Guidinger MK, Metzger RA, Wiesner RH, Webb RL, Merion RM. Transplant center quality assessment using a continuously updatable, risk-adjusted technique (CUSUM). Am J Transplant 2006;6:313-323.[Medline]
- Dibra A, Kastrati A, Schuhlen H, Schomig A. The relationship between hospital or operator volume and outcomes of coronary patients undergoing percutaneous coronary interventions. Z Kardiol 2005;94:231-238.[Medline]
- Hannan EL, Wu C, Ryan TJ, Bennett E, Culliford AT, Gold JP, et al. Do hospitals and surgeons with higher coronary artery bypass graft surgery volumes still have lower risk-adjusted mortality rates?. Circulation 2003:795-80119;108.
- Shahian DM, Normand SL. The volume-outcome relationship: from Luft to Leapfrog. Ann Thorac Surg 2003;75:1048-1058.[Abstract/Free Full Text]
- Welke KF, Barnett MJ, Sarrazin MS, Rosenthal GE. Limitations of hospital volume as a measure of quality of care for coronary artery bypass graft surgery. Ann Thorac Surg 2005;80:2114-2119.[Abstract/Free Full Text]
- Zacharias A, Schwann TA, Riordan CJ, Durham SJ, Shah A, Papadimos TJ, et al. Is hospital procedure volume a reliable marker of quality for coronary artery bypass surgery? A comparison of risk and propensity adjusted operative and midterm outcomes. Ann Thorac Surg 2005;79:1961-1969.[Abstract/Free Full Text]
- Rathore SS, Epstein AJ, Volpp KG, Krumholz HM. Hospital coronary artery bypass graft surgery volume and patient mortality, 1998-2000. Ann Surg 2004;239:110-117.[Medline]
- Papadimos TJ, Habib RH, Zacharias A, Schwann TA, Riordan CJ, Durham SJ, et al. Early efficacy of CABG care delivery in a low procedure-volume community hospital: operative and midterm results. BMC Surg 2005;5:10.[Medline]
- Kang N, Tsang VT, Gallivan S, Sherlaw-Johnson C, Cole TJ, Elliott MJ, et al. Quality assurance in congenital heart surgery. Eur J Cardiothorac Surg 2006;29:693-697.[Abstract/Free Full Text]
- Novick RJ, Fox S, Stitt LW, Forbes TL, Steiner S. Direct comparison of risk-adjusted and non-risk-adjusted CUSUM analyses of coronary artery bypass surgery outcomes. J Thorac Cardiovasc Surg 2006;132:386-391.[Abstract/Free Full Text]
- Holzhey DM, Jacobs S, Mochalski M, Walther T, Thiele H, Mohr FW, et al. Seven year follow-up after minimally invasive direct coronary artery bypass experience with more than 1300 patients. Ann Thorac Surg 2007;83:108-114.[Abstract/Free Full Text]
This article has been cited by other articles:

|
 |

|
 |
 
M. Murzi, A. G. Cerillo, S. Bevilacqua, D. Gilmanov, P. Farneti, and M. Glauber
Traversing the learning curve in minimally invasive heart valve surgery: a cumulative analysis of an individual surgeon's experience with a right minithoracotomy approach for aortic valve replacement
Eur J Cardiothorac Surg,
January 9, 2012;
(2012)
ezr230v1.
[Abstract]
[Full Text]
[PDF]
|
 |
|

|
 |

|
 |
 
V. Falk and D. P. Taggart
NICE guidance for off-pump CABG: turn off the pump
Heart,
November 1, 2011;
97(21):
1731 - 1733.
[Full Text]
[PDF]
|
 |
|

|
 |

|
 |
 
S. Hu, Q. Li, P. Gao, H. Xiong, Z. Zheng, L. Li, B. Xu, and R. Gao
Simultaneous Hybrid Revascularization Versus Off-Pump Coronary Artery Bypass for Multivessel Coronary Artery Disease
Ann. Thorac. Surg.,
February 1, 2011;
91(2):
432 - 438.
[Abstract]
[Full Text]
[PDF]
|
 |
|

|
 |

|
 |
 
T. Schachner, N. Bonaros, D. Wiedemann, F. Weidinger, G. Feuchtner, G. Friedrich, G. Laufer, and J. Bonatti
Training Surgeons to Perform Robotically Assisted Totally Endoscopic Coronary Surgery
Ann. Thorac. Surg.,
August 1, 2009;
88(2):
523 - 527.
[Abstract]
[Full Text]
[PDF]
|
 |
|

|
 |

|
 |
 
M. Hartrumpf, T. Claus, M. Erb, and J. M. Albes
Surgeon performance index: tool for assessment of individual surgical quality in total quality management
Eur J Cardiothorac Surg,
May 1, 2009;
35(5):
751 - 758.
[Abstract]
[Full Text]
[PDF]
|
 |
|