HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS		QUICK SEARCH:   [advanced] Author: Keyword(s): Year:  Vol:  Page:

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Austin, P. C. Search for Related Content PubMed PubMed Citation Articles by Austin, P. C.

J Thorac Cardiovasc Surg 2007;134:1128-1135
© 2007 The American Association for Thoracic Surgery

Surgery for Acquired Cardiovascular Disease

Propensity-score matching in the cardiovascular surgery literature from 2004 to 2006: A systematic review and suggestions for improvement

Institute for Clinical Evaluative Sciences, the Department of Public Health Sciences, University of Toronto, and the Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada.

Received for publication April 16, 2007; accepted for publication July 31, 2007.

^_* Address for reprints: Peter C. Austin, PhD, Institute for Clinical Evaluative Sciences, G1 06, 2075 Bayview Ave, Toronto, Ontario M4N 3M5, Canada. (Email: peter.austin{at}ices.on.ca).

	Abstract

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 
Objective: I conducted a systematic review of the use of propensity score matching in the cardiovascular surgery literature. I examined the adequacy of reporting and whether appropriate statistical methods were used.

Methods: I examined 60 articles published in the Annals of Thoracic Surgery, European Journal of Cardio-thoracic Surgery, Journal of Cardiovascular Surgery, and the Journal of Thoracic and Cardiovascular Surgery between January 1, 2004, and December 31, 2006.

Results: Thirty-one of the 60 studies did not provide adequate information on how the propensity score–matched pairs were formed. Eleven (18%) of studies did not report on whether matching on the propensity score balanced baseline characteristics between treated and untreated subjects in the matched sample. No studies used appropriate methods to compare baseline characteristics between treated and untreated subjects in the propensity score–matched sample. Eight (13%) of the 60 studies explicitly used statistical methods appropriate for the analysis of matched data when estimating the effect of treatment on the outcomes. Two studies used appropriate methods for some outcomes, but not for all outcomes. Thirty-nine (65%) studies explicitly used statistical methods that were inappropriate for matched-pairs data when estimating the effect of treatment on outcomes. Eleven studies did not report the statistical tests that were used to assess the statistical significance of the treatment effect.

Conclusions: Analysis of propensity score–matched samples tended to be poor in the cardiovascular surgery literature. Most statistical analyses ignored the matched nature of the sample. I provide suggestions for improving the reporting and analysis of studies that use propensity score matching.

	Introduction

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

There are three commonly used propensity score methods: covariate adjustment using the propensity score, stratification on the propensity score, and propensity score matching.⁷ Earlier studies have shown that propensity score matching results in the comparison of treated and untreated subjects who are more similar than does stratification on the propensity score.^6,7 However, the analysis of a propensity score–matched sample requires statistical methods appropriate for matched data.

A recently published survey found that statistical errors were present in a high proportion of articles published in two medical journals.⁸ A systematic review of articles that employed propensity-score matching and were published in the literature between 1996 and 2003 found that a high proportion of articles contained errors in the application of propensity-score matching.⁹ Propensity score matching is frequently used in the cardiovascular surgery literature. The objective of the current study was twofold: first, to systematically examine the use of propensity score matching in the cardiovascular surgery literature; second, to provide recommendations to cardiovascular surgery researchers on the implementation of propensity score matching.

	Statistical Methods for Propensity Score–Matched Samples

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 
There are 4 different steps in a propensity score–matched analysis. First, one must specify the propensity score model. Second, one must construct the propensity score–matched sample. Third, one must assess the degree to which matching on the propensity score has resulted in a matched sample in which the distribution of measured baseline variables is similar between treated and untreated subjects. Fourth, one must estimate the effect of the treatment or exposure on the outcomes under consideration. Each of these steps is described in the subsequent subsections.

	Specifying the Propensity Score Model

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 
I briefly provide some guidance on specifying the propensity score model. Rosenbaum and Rubin¹ demonstrated that the propensity score is a balancing score: conditioning on the propensity score results in treated and untreated subjects having similar distributions of baseline variables. Ho and associates¹⁰ refer to the propensity score tautology as the fact that one has correctly specified the propensity score model when matching on the propensity score results in a matched sample in which treated and untreated subjects have similar distributions of measured baseline variables. This reflects Rosenbaum and Rubin’s² use of an iterative process for specifying the propensity score. Recently, along with Grootendorst and Anderson, I⁶ published an article on variable selection for propensity score models. I demonstrated that including variables that are related to exposure but that are independent of the outcome in the propensity score model can result in the formation of fewer matched pairs. This can result in less precise estimates of the treatment effect. Including only the confounders of the treatment–outcome relationship, or all the variables associated with the outcome, resulted in the formation of a greater number of matched pairs and more precise estimates of treatment effect. Furthermore, it was shown that the receiver operating characteristic curve area (c-statistic) of the propensity score model did not provide any information about whether important confounders had been omitted. I propose the following steps in specifying the propensity score model: First, derive a list of measured baseline variables that are likely related to exposure and/or the outcome. The variables in this list can be selected from reviews of the literature, from prior studies, and from expert opinion. Importantly, the list should only include variables measured at baseline, before exposure. The list should not include the outcome, nor should it include variables in the causal pathway. Second, derive an initial propensity score model by including all variables in the list as main effects. Third, assess whether matching on the propensity score results in a matched sample in which measured baseline variables are balanced between treated and untreated subjects. Fourth, in the event of imbalance, modify the propensity score, possibly by using methods described in more detail by Rosenbaum and Rubin.² The third and fourth steps can be repeated iteratively until the baseline variables are balanced between treated and untreated subjects. If the final propensity score model contains variables that are associated with treatment but that are independent of the outcome, then one can examine whether, by dropping these variables, one can form a larger number of propensity score–matched pairs without increasing systematic differences in prognostic variables between treated and untreated subjects.

	Propensity Score Matching

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 
The propensity score is usually estimated by a logistic regression model in which treatment (yes/no; 1/0) is regressed on baseline characteristics. The estimated propensity score is then the predicted probability of exposure to the treatment from the logistic regression model. Once the propensity score has been estimated for each subject, treated and untreated subjects are matched on the propensity score. Typically, nearest neighbor matching within a specified caliper width is used. By this method, treated subjects are randomly sorted. Then, the first treated subject is matched to the untreated subject with the closest propensity score within a specified range (the caliper width). If no untreated subject has a propensity score that lies within a specified caliper width of the treated subject’s propensity score, then that treated subject is left unmatched and is not used in subsequent analyses. Matching without replacement is usually employed: once an untreated subject has been matched to a given treated subject, this untreated subject is no longer considered as a possible match for subsequent treated subjects. This process is then repeated until all possible matches have been formed. Because the propensity score is a probability, it takes on values that lie between 0 and 1. If two subjects have propensity scores of 0.12345678 and 0.12345123, then these subjects have propensity scores that match on the first five digits (0.12345). Another common approach is to attempt to match treated and untreated subjects on the first five digits of the propensity score. If no match is found for a specific treated subject, then matching is attempted on the first four digits. If no suitable untreated subject exists, then matches are attempted on the first three, first two, and finally, the first digit of the propensity score.¹¹ I refer to this method as 51 digit matching. A recent study found that matching on the logit of propensity score, using calipers of width 0.2 of the standard deviation of the logit of the propensity score, tended to have superior performance compared with other competing methods that are used in the medical literature. [Austin PC. The performance of different propensity-score matching methods used in the medical literature. Under review.]

The term "greedy matching" refers to any matching algorithm in which, at a specific step in the matching process, the nearest untreated subject is matched to the treated subject in question, even if that untreated subject would have been a better match for a subsequent treated subject.¹² The term "greedy" does not provide any information about the calipers that were used in the matching process. The alternative to using a "greedy" approach is to use an "optimal" matching strategy that makes matches so as to minimize a weighted average of the within-pair distance over all possible matches.¹² In the above, I have assumed matching without replacement. One can also use matching with replacement, in which an untreated subject can serve as a match for more than one treated subject. Thus, it is possible to have multiple matched pairs, each consisting of a different treated subject, but each consisting of the same untreated subject. When matching with replacement is used, statistical methods for estimating the treatment effect must take into account the lack of independence in outcomes for the same untreated subject that is contained in multiple matched sets. Throughout the remainder of the manuscript, I will assume that matching without replacement is being used.

	Statistical Methods for Assessing Balance

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 
The propensity score is a balancing score: matching on the true propensity score results in a matched sample in which the distribution of each baseline characteristic is similar between treated and untreated subjects. In reality, except in controlled experiments, one does not know the true propensity score, and it must be estimated from the data. Ho and colleagues¹⁰ refer to the propensity score tautology as the fact that one knows that one has properly specified the propensity score model when matching on the estimated propensity score balances baseline variables between treated and untreated subjects. Thus, an important component of any propensity score–matched analysis is comparing the balance in baseline variables between treated and untreated subjects.

There is clear consensus among statisticians as to the inappropriateness of using significance test to compare the distribution of baseline covariates between different arms of a randomized controlled trial.^13-19 Imai, King, and Stuart²⁰ have proposed two criteria for appropriate methods for comparing baseline variables between treated and untreated subjects in observational studies. First, because balance is a property of a sample, and not of a hypothetical super-population, the measure for assessing balance must be a property of the sample. Second, the method for assessing balance must not be influenced by the size of the sample. Both of these criteria rule out the use of significance testing for assessing balance in baseline variables between treated and untreated subjects. The second criterion is important, for if the method to assess balance were influenced by sample size, then the matched sample may appear to have better balance solely because of the reduction in sample size that results from matching. Several authors have proposed the use of standardized differences for assessing balance in observational studies.^5,6,21,22 The standardized difference is defined as follows:

where and are the mean of the variable among the treated and untreated subjects, respectively, while and are the sample standard deviation of covariate in the treated and untreated subjects, respectively. Ho and associates¹⁰ describe other possible measures to assess balance, including quantile–quantile plots.

	Estimating the Treatment Effect

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 
The need to account for the matched nature of the sample
The propensity score–matched sample was created by matching pairs of subjects with a similar propensity score. Therefore, treated and untreated subjects within the same matched pair have a similar propensity score. Thus, these treated and untreated subjects within the same matched pair have baseline variables that come from the same multivariate distribution.¹ Randomly chosen treated and untreated subjects are likely to differ systematically in their baseline variables. Hence, treated and untreated subjects within the same matched pair are, on average, more similar than two randomly selected treated and untreated subjects. Because outcomes are influenced by baseline characteristics (otherwise there would be no confounding), then outcomes are more similar within the matched pair than between randomly selected treated and untreated subjects. This within-pair homogeneity means that subjects within the same matched pair are not independent. Therefore, by construction, the propensity score–matched sample does not consist of independent observations. The need to account for matching in the statistical analyses is well described in the epidemiology literature.^23,24

Statistical methods for estimating the treatment effect
The final analytic step is to estimate the effect of treatment on the outcomes. This must be done in a manner that accounts for the matched nature of the propensity score–matched sample. The statistical significance of the effect of exposure on continuous outcomes can be assessed by a paired t test²⁵ or the Wilcoxon signed rank test.²⁶ Proportions can be compared by the McNemar test for correlated binary proportions, or extensions thereof for categorical variables with more than two levels.²⁷ Agresti and Min²⁸ describe methods for estimating relative risks and odds ratios in matched samples and for constructing appropriate confidence intervals. Kaplan–Meier survival curves can be compared by a test described by Klein and Moeschberger.²⁹ As a brief description of this test, let D₁ denote the number of matched pairs in which the treated subject experiences the event first, while D₂ denotes the number of matched pairs in which the untreated subject experiences the event first. The test statistic is as follows:

which has a standard normal distribution under the null hypothesis and when the number of matched pairs is large (the matched pairs in which the smaller of the two times is a censored observation makes no contribution to the test statistic). The log–rank test, which assumes independent strata, is not appropriate for comparing survival curves in matched samples.^29,30 The standard Cox regression model is not appropriate for matched-pairs data, as it assumes independent samples.³¹ However, for analyzing survival data, using a Cox proportional hazards models stratified on the matched pairs would be appropriate.³² Another approach would be to use a Cox proportional hazards model with robust standard errors that account for the clustering in matched pairs.³³ Similarly, conventional logistic regression would not be appropriate.³⁴ However, for analyzing binary outcomes, conditional logistic regression or logistic regression models estimated using generalized estimating equation methods take into account the matched nature of the data can be used.³⁵

	Survey of Propensity Score Matching in the Cardiovascular Surgery Literature

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 
Identification of Published Articles Using Propensity Score Matching
I used a search strategy similar to that of a recently published systematic review of propensity score methods in the medical literature.³⁶ I used both PubMed and the Science Citation Index to identify studies that used propensity score matching. I identified studies that included the keyword propensity, using a keyword search in PubMed. I restricted my search to articles published between January 1, 2004, and December 31, 2006, in the following cardiovascular surgery journals: Annals of Thoracic Surgery, European Journal of Cardio-thoracic Surgery, Journal of Cardiovascular Surgery, Journal of Heart and Lung Transplant, and the Journal of Thoracic and Cardiovascular Surgery. Using the Science Citation Index, I also searched for articles that cited one of the important papers on propensity score methods.^1,2,37-40 The combined search identified 115 articles. I then examined these 115 articles and selected only those that used propensity score matching. The combined search strategy resulted in the identification of 60 studies^E1-E60 that used propensity score matching in the following journals: Annals of Thoracic Surgery (31 articles), European Journal of Cardio-thoracic Surgery (9 articles), Journal of Cardiovascular Surgery (1 article), and the Journal of Thoracic and Cardiovascular Surgery (19 articles).

Abstraction of Analytic Methods in Propensity Score–Matched Samples
I abstracted the following information from each of the published articles:

1 The method by which propensity score–matched pairs were formed.

2 Whether the authors assessed the balance in baseline characteristics between treated and untreated subjects in the matched sample. When the authors compared balance of baseline variables between treated and untreated subjects, I examined the methods that the authors used.

3 The statistical methods used to assess the effect of treatment on the outcome, and whether this method was appropriate for matched-pairs data.

	Results of Systematic Review

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 
I critically examined 60 articles published in the cardiovascular surgery literature between 2004 and 2006 that used propensity score matching. I report our results separately for each of the three items that were abstracted.

	Propensity Score Matching

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 
Seventeen (28%) of the studies did not report the manner by which propensity score–matched pairs were formed. An additional 4 reported that greedy matching was used, and 10 stated that nearest neighbor matching was used. As noted earlier, the use of the term "greedy matching" does not provide any details about the required similarity of the propensity score between matched treated and untreated subjects. Similarly, the use of the term "nearest neighbor matching" is also uninformative, because it does not provide any details on the caliper widths that were used in the matching process. Taken to the extreme, if a caliper width was not used, then a matched untreated subject would be able to be found for each treated subject, because there is no specification that their propensity scores are required to be similar. In total, 31 (52%) of the studies did not provide adequate information about how the matching was done. This has important consequences in that it does not permit other researchers to reproduce the studies’ methods. Among studies that fully describe how matches were formed, 20 used 51 digit matching, 1 study matched on the logit of the propensity score using calipers of width 0.2 standard deviations of the logit of the propensity score, and other studies matched on the propensity score using the following calipers: 0.1 (1 study), 0.05 (2 studies), 0.02 (1 study), 0.015 (1 study), 0.01 (2 studies), and 0.001 (1 study).

	Assessing Balance Between Treated and Untreated Subjects

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 
Eleven (18%) studies did not report whether matching on the propensity score resulted in a matched sample in which the distribution of baseline characteristics was similar between treated and untreated subjects. One additional study reported that balance was achieved, but did not report a table comparing the distribution of baseline characteristics between treated and untreated subjects in the matched sample. The remaining 48 studies (80%) reported a table in which the distribution of baseline characteristics was compared between treated and untreated subjects.

Of the 49 studies that reported comparing the distribution of baseline characteristics between treated and untreated subjects, 47 studies used statistical significance testing, 1 study relied on visual comparison, and 1 study did not report the methods that were used. Of the 47 studies that used statistical significance testing, 1 study explicitly stated that correct statistical methods were used for matched-pairs data,^E28 35 (58%) studies explicitly used statistical hypothesis testing methods that did not incorporate the matched nature of the sample, and 1 study used appropriate statistical hypothesis tests for some variables but inappropriate statistical hypothesis tests for other variables. Ten studies did not report what statistical tests were used to compare the distribution of baseline characteristics between treated and untreated subjects—only significance levels were reported, but not the statistical tests used to obtain these significance levels. Importantly, none of the studies reported using appropriate methods for assessing balance in baseline variables between treated and untreated subjects. As discussed in the "Statistical Methods for Assessing Balance" section, statistical hypothesis testing (regardless of whether it accounts for the matched nature of the sample) is not appropriate for comparing baseline balance of measured covariates. No studies reported using standardized differences or other comparable methods.

	Estimating the Effect of Treatment or Exposure on the Outcome

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 
Eight (13%) of the 60 articles explicitly stated that methods appropriate for the analysis of matched data were used in estimating the treatment effect and its statistical significance. These studies used McNemar’s test,^E8 regression models estimated generalized estimating equation methods to account for the matched-pairs nature of the sample,^{E16,E36,E39,E46} Cox proportional hazards regression stratified on the matched pairs,^E42 and conditional logistic regression.^E43,E54 Two additional studies used methods appropriate for correlated data for some outcomes, but not for other outcomes.^E33,E49

Thirty-nine (65%) studies explicitly used inappropriate statistical methods for assessing the statistical significance of the effect of treatment on the outcomes. Common errors included using the log–rank test to compare Kaplan–Meier survival curves in the matched sample, using Cox proportional hazards models in the matched sample, using logistic regression in the matched sample, using ² tests to compare proportions in the matched sample, and using Wilcoxon rank sum tests or standard t tests to compare continuous variables in the matched sample. Eleven studies did not describe the statistical methods that were used to compare the outcome between treated and untreated subjects.^{E5,E12,E14,E15,E19,E20,E22,E24,E28,E44,E45} In general, these studies were comparing proportions, means, medians, or Kaplan–Meier survival curves between treated and untreated subjects.

	Discussion

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 
The objective of the current study was to critically examine the use of propensity score matching in the cardiovascular surgery literature. None of the 60 studies compared the distribution of baseline variables between treated and untreated subjects in the matched sample and explicitly documented using appropriate statistical methods to assess whether measured characteristics were balanced between treated and untreated subjects in the matched sample. Eight (13%) of the 60 studies explicitly used appropriate statistical methods for all analyses examining the impact of treatment on outcomes. I make the following recommendations for the design, analysis, and reporting of studies that use propensity score matching. I summarize my recommendations for the implementation of propensity score matching in Table E1.

Reporting the Balance of Baseline Variables Between Treated and Untreated Subjects in the Matched Sample
The CONSORT statement recommends that baseline demographic and clinical data be reported for each arm in a randomized controlled trial.⁴² In a randomized controlled trial, reporting baseline characteristics in each arm of the study allows the reader to assess whether there was potentially a breakdown in randomization, since randomization should, on average, result in similar distributions of baseline variables between the different arms of the study. Although observational studies are, by definition, nonrandomized, matching on the propensity score allows one to balance measured baseline variables between treated and untreated subjects. Describing the balance in measured variables between treated and untreated subjects in the matched sample allows both the researcher and the reader to assess whether the propensity model was adequately specified.

Appropriate statistical methods should be used to compare the distribution of the baseline covariates between treated and untreated subjects. Statistical methods for assessing balance in baseline variables should not be affected by sample size and should reflect the fact that balance is a property of a sample and not refer to a super-population.²⁰ I encourage researchers to use standardized differences to compare distributions between treated and untreated subjects.^6,7,21 Unlike P values, the standardized difference is not confounded with sample size, and thus balance in the initial sample can be compared with that in the matched sample. It can also be used to compare the relative balance of variables measured in different units.

Some studies that use propensity score matching compare characteristics of matched treated subjects with those of unmatched treated subjects. This comparison can provide useful information on differences between treated patients who were used in estimating the treatment effect and the treated patients who were excluded from these analyses. These comparisons can provide useful clinical information and information on generalizability of the results. However, I do not consider the decision concerning their inclusion or exclusion to be a statistical one. Furthermore, their inclusion or exclusion does not affect the quality of statistical methods for assessing balance in measured variables between treated and untreated subjects in the matched sample and for estimating the significance of the treatment effect.

Statistical Methods for Estimating the Effect of Treatment on Outcomes
Researchers should explicitly report that methods appropriate for the analysis of matched data were used. In the section titled "Statistical Methods for Estimating the Treatment Effect," appropriate methods are described for estimating the treatment effect in propensity score–matched samples. Some researchers have used conditional logistic regression for the analysis of propensity score–matched samples. Whereas conditional logistic regression is appropriate for matched-pairs data, it has been shown to result in biased estimation of odds ratios when used in propensity score–matched samples⁴³ since propensity score methods allow one to estimate marginal treatment effects and not conditional treatment effects.⁴⁴ A marginal treatment effect is the average effect at the population level, whereas the conditional treatment effect is the average effect at the subject level. The odds ratio is not collapsible, meaning that the marginal treatment effect is different from the conditional treatment effect.⁴⁵ However, risk differences are collapsible, whereas the relative risk is collapsible under certain circumstances.⁴⁵ The use of conditional logistic regression is thus discouraged. Because the matched sample has reduced or eliminated systematic differences in measured variables between treated and untreated subjects, regression adjustment should rarely be needed. Researchers are encouraged to report risk differences or relative risks, rather than odds ratios. Agresti and Min²⁸ describe statistical methods for estimating relative risks and associated confidence intervals in matched data. When researchers want to adjust for possible residual baseline differences between treated and untreated subjects, regression methods that account for the matched nature of the sample should be incorporated. Appropriate statistical methods for different types of outcomes are summarized in Table E2, which appears online only.

Limitations
There is a limitation to this current systematic review. First, the quality of articles employing propensity score matching was assessed using published articles in the cardiovascular surgery literature. It is possible that authors had provided greater details on the analyses and results, but that these were removed during the revision and editorial process. However, this limitation is tempered by the fact that 65% of the reviewed articles explicitly used statistical methods inappropriate for matched-pairs data when estimating the effect of treatment on the outcome. Furthermore, 47 (78%) of articles used statistical significance testing to compare differences in baseline variables between treated and untreated subjects in the matched sample, despite this not being appropriate. Most of the errors that were highlighted were errors of commission, rather than errors of omission.

	Conclusions

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 
In conclusion, propensity score matching tended to be poorly implemented in the cardiovascular surgery literature. The majority of studies ignored the matched nature of the propensity score–matched sample in the subsequent analyses. I have provided suggestions for improving the analysis of propensity score–matched samples and for improving the reporting of these analyses.

	Footnotes

 
The Institute for Clinical Evaluative Sciences (ICES) is supported in part by a grant from the Ontario Ministry of Health and Long Term Care. The opinions, results and conclusions are those of the author and no endorsement by the Ministry of Health and Long-Term Care or by the Institute for Clinical Evaluative Sciences is intended or should be inferred. Dr Austin is supported in part by a New Investigator award from the Canadian Institutes of Health Research (CIHR).

	References

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

 

1. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika 1983;70:41-55.[Abstract/Free Full Text]
   
2. Rosenbaum PR, Rubin DB. Reducing bias in observational studies using subclassification on the propensity score. J Am Stat Assoc 1984;79:516-524.
   
3. Grunkmeier GL, Payne N, Jin R, Handy Jr JR. Propensity score analysis of stroke after off-pump coronary artery bypass grafting. Ann Thorac Surg 2002;74:301-305.[Free Full Text]
   
4. Blackstone EH. Comparing apples and oranges. J Thorac Cardiovasc Surg. 2002;123:8-15.[Free Full Text]
   
5. Austin PC, Mamdani MM, Stukel TA, Anderson GM, Tu JV. The use of the propensity score for estimating treatment effects: administrative versus clinical data. Stat Med 2005;24:1563-1578.[Medline]
   
6. Austin PC, Grootendorst P, Anderson GM. A comparison of the ability of different propensity score models to balance measured variables between treated and untreated subjects: A Monte Carlo study. Stat Med 2007;26:734-753.[Medline]
   
7. Austin PC, Mamdani MM. A comparison of propensity score methods: a case-study estimating the effectiveness of post-AMI statin use. Stat Med 2006;25:2084-2106.[Medline]
   
8. Strasak AM, Zaman Q, Marinell G, Pfeiffer KP, Ulmer H. The use of statistics in medical research: a comparison of The New England Journal of Medicine and Nature Medicine . Am Stat 2007;61:47-55.
   
9. Austin PC. A critical appraisal of propensity-score matching in the medical literature from 1996 to 2003. Stat Med. Accepted.
   
10. Ho DE, Imai K, King G, Stuart EA. Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political Anal. In press.
    
11. Parson LS. Proceedings of the Twenty-sixth Annual SAS Users Group International Conference. Reducing bias in a propensity score matched-pair sample using greedy matching techniques. Cary (NC): SAS Institute Inc; 2001. pp. 214-216.
    
12. Rosenbaum PR. Observational studies. Springer-Verlag: New York; 1995.
    
13. Altman DT, Dore CJ. Randomisation and baseline comparisons in clinical trials. Lancet 1990;335:149-153.[Medline]
    
14. Altman DG. Comparability of randomised groups. Statistician 1985;34:125-136.
    
15. Senn S. Testing for baseline balance in clinical trials. Stat Med 1994;13:1715-1726.[Medline]
    
16. Senn S. Baseline comparisons in randomized clinical trials. Stat Med 1991;10:1157-1160.[Medline]
    
17. Senn SJ. Covariate imbalance and random allocation in clinical trials. Stat Med 1989;8:467-475.[Medline]
    
18. Begg CB. Significance tests of covariate imbalance in clinical trials. Control Clin Trials 1990;11:223-225.[Medline]
    
19. Rothman KJ. Epidemiologic methods in clinical trials. Cancer 1977;39:1771-1775.[Medline]
    
20. Imai K, King G, Stuart EA. Misunderstandings among experimentalists and observationalists: balance test fallacies in causal inference. Technical report. Princeton University. Available from: http://imai.princeton.edu/research/balance.html.
    
21. Flury BK, Riedwyl H. Standard distance in univariate and multivariate analysis. Am Stat 1986;40:249-251.
    
22. Normand ST, Landrum MB, Guadagnoli E, Ayanian JZ, Ryan TJ, Cleary PD, et al. Validating recommendations for coronary angiography following acute myocardial infarction in the elderly: a matched analysis using propensity scores. J Clin Epidemiol 2001;54:387-398.[Medline]
    
23. Rothman KJ, Greenland S. Modern epidemiology. Philadelphia: Lippincott Williams & Wilkins; 1998.
    
24. Breslow NE, Day NE. Statistical methods in cancer research. The analysis of case–control studies. Vol I. Lyon: International Agency for Research on Cancer; 1980.
    
25. Snedecor GW, Cochran WG. Statistical methods. 8th ed.. Ames (IA): Iowa State University Press; 1989.
    
26. Conover WJ. Practical nonparametric statistics. 3rd ed.. New York: John Wiley; 1999.
    
27. Fleiss JL, Levin B, Paik MC. Statistical methods for rates and proportions. 3rd ed.. New York: John Wiley; 2003.
    
28. Agresti A, Min Y. Effects and non-effects of paired identical observations in comparing proportions with binary matched-pairs data. Stat Med 2004;23:65-75.[Medline]
    
29. Klein JP, Moeschberger ML. Survival analysis: techniques for censored and truncated data. New York: Springer-Verlag; 1997.
    
30. Harrington D. Encyclopedia of biostatisics. In: Armitage P, Colton T, editors. Linear rank tests in survival analysis. 2nd ed.. New York: John Wiley; 2005. pp. 2802-2812.
    
31. Cox DR, Oakes K. Analysis of survival data. London: Chapman & Hall; 1984.
    
32. Therneau TM, Grambsch PM. Modeling survival data: extending the Cox model. New York: Springer-Verlag; 2000.
    
33. Lin DY, Wei LJ. The robust inference for the proportional hazards model. J Am Stat Assoc 1989;84:1074-1078.
    
34. Cox DR, Snell EJ. Analysis of binary data. London: Chapman & Hall; 1989.
    
35. Diggle PJ, Liang KY, Zeger SL. Analysis of longitudinal data. Oxford: Oxford University Press; 1994.
    
36. Stürmer T, Joshi M, Glynn RJ, Avorn J, Rothman KJ, Schneeweiss S. A review of the application of propensity score methods yielded increasing use, advantages in specific settings, but not substantially different estimates compared with conventional multivariable methods. J Clin Epidemiol 2006;59:437-447.[Medline]
    
37. Rosenbaum PR, Rubin DB. The bias due to incomplete matching. Biometrics 1985;41:103-116.[Medline]
    
38. D’Agostino Jr RB. Propensity score methods for bias reduction in the comparison of a treatment to a non-randomized control group. Stat Med 1998;17:2265-2281.[Medline]
    
39. Joffe MM, Rosenbaum PR. Invited commentary: Propensity scores. Am J Med 1999;150:327-333.
    
40. Rubin DB. Estimating causal treatment effects from large datasets using propensity scores. Ann Intern Med 1997;127:757-763.[Medline]
    
41. Cochran WG, Rubin DB. Controlling bias in observational studies: a review. Sankhya Series A 1973;35:417-446.
    
42. Moher D, Schulz KF, Altman D, CONSORT Group The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001;285:1987-1991.[Medline]
    
43. Austin PC. The performance of different propensity score methods for estimating marginal odds ratios. Stat Med 2007;26:3078-3094.[Medline]
    
44. Austin PC, Grootendorst P, Normand SLT, Anderson GM. Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: a Monte Carlo study. Stat Med 2007;26:754-768.[Medline]
    
45. Greenland S. Interpretation and choice of effect measures in epidemiologic analyses. Am J Epidemiol 1987;125:761-768.[Free Full Text]
    

	E-References (systematic review articles)

Top Abstract Introduction Statistical Methods for... Specifying the Propensity Score... Propensity Score Matching Statistical Methods for... Estimating the Treatment Effect Survey of Propensity Score... Results of Systematic Review Propensity Score Matching Assessing Balance Between... Estimating the Effect of... Discussion Conclusions References E-References (systematic review...

1. Rice TW, Khuntia D, Rybicki LA, Adelstein DJ, Vogelbaum MA, Mason DP, et al. Brain metastases from esophageal cancer: a phenomenon of adjuvant therapy?. Ann Thorac Surg 2006;82:2042-20492049. e1-2.[Abstract/Free Full Text]
2. Koch CG, Li L, Van Wagoner DR, Duncan AI, Gillinov AM, Blackstone EH. Red cell transfusion is associated with an increased risk for postoperative atrial fibrillation. Ann Thorac Surg 2006;82:1747-1756.[Abstract/Free Full Text]
3. Kunihara T, Schmidt K, Glombitza P, Dzindzibadze V, Lausberg H, Schafers HJ. Root replacement using stentless valves in the small aortic root: a propensity score analysis. Ann Thorac Surg 2006;82:1379-1384.[Abstract/Free Full Text]
4. Mishra M, Malhotra R, Karlekar A, Mishra Y, Trehan N. Propensity case-matched analysis of off-pump versus on-pump coronary artery bypass grafting in patients with atheromatous aorta. Ann Thorac Surg 2006;82:608-614.[Abstract/Free Full Text]
5. DeCamp MM, Blackstone EH, Naunheim KS, Krasna MJ, Wood DE, Meli YM, et al. NETT Research Group Patient and surgical factors influencing air leak after lung volume reduction surgery: lessons learned from the National Emphysema Treatment Trial. Ann Thorac Surg 2006;82:197-206.[Abstract/Free Full Text]
6. Di Mauro M, Di Giammarco G, Vitolla G, Contini M, Iaco AL, Bivona A, et al. Impact of no-to-moderate mitral regurgitation on late results after isolated coronary artery bypass grafting in patients with ischemic cardiomyopathy. Ann Thorac Surg 2006;81:2128-2134.[Abstract/Free Full Text]
7. Brunelli A, Xiume’ F, Al Refai M, Salati M, Marasco R, Sabbatini A. Gemcitabine-cisplatin chemotherapy before lung resection: a case-matched analysis of early outcome. Ann Thorac Surg 2006;81:1963-1968.[Abstract/Free Full Text]
8. Baskett RJ, Cafferty FH, Powell SJ, Kinsman R, Keogh BE, Nashef SA. Total arterial revascularization is safe: multicenter ten-year analysis of 71,470 coronary procedures. Ann Thorac Surg 2006;81:1243-1248.[Abstract/Free Full Text]
9. Olsson C, Thelin S. Antegrade cerebral perfusion with a simplified technique: unilateral versus bilateral perfusion. Ann Thorac Surg 2006;81:868-874.[Abstract/Free Full Text]
10. Toumpoulis IK, Anagnostopoulos CE, Balaram S, Swistel DG, Ashton Jr RC, DeRose Jr JJ. Does bilateral internal thoracic artery grafting increase long-term survival of diabetic patients?. Ann Thorac Surg 2006;81:599-606.[Abstract/Free Full Text]
11. Chukwuemeka A, Weisel A, Maganti M, Nette AF, Wijeysundera DN, Beattie WS, et al. Renal dysfunction in high-risk patients after on-pump and off-pump coronary artery bypass surgery: a propensity score analysis. Ann Thorac Surg 2005;80:2148-2153.[Abstract/Free Full Text]
12. Sabik 3rd JF, Blackstone EH, Houghtaling PL, Walts PA, Lytle BW. Is reoperation still a risk factor in coronary artery bypass surgery?. Ann Thorac Surg 2005;80:1719-1727.[Abstract/Free Full Text]
13. Calafiore AM, Di Mauro M, Di Giammarco G, Teodori G, Iaco AL, Mazzei V, et al. Single versus bilateral internal mammary artery for isolated first myocardial revascularization in multivessel disease: long-term clinical results in medically treated diabetic patients. Ann Thorac Surg 2005;80:888-895.[Abstract/Free Full Text]
14. Gillinov AM, Blackstone EH, Rajeswaran J, Mawad M, McCarthy PM, Sabik 3rd JF, et al. Ischemic versus degenerative mitral regurgitation: does etiology affect survival?. Ann Thorac Surg 2005;80:811-819.[Abstract/Free Full Text]
15. Lawton JS, Barner HB, Bailey MS, Guthrie TJ, Moazami N, Pasque MK, et al. Radial artery grafts in women: utilization and results. Ann Thorac Surg 2005;80:559-563.[Abstract/Free Full Text]
16. Louagie YA, Jamart J, Gruslin A. Do coronary bypass graft flows differ between on-pump and off-pump operations?. Ann Thorac Surg 2005;79:2004-2012.[Abstract/Free Full Text]
17. Habib RH, Zacharias A, Schwann TA, Riordan CJ, Durham SJ, Shah A. Effects of obesity and small body size on operative and long-term outcomes of coronary artery bypass surgery: a propensity-matched analysis. Ann Thorac Surg 2005;79:1976-1986.[Abstract/Free Full Text]
18. van Domburg RT, Takkenberg JJ, Noordzij LJ, Saia F, van Herwerden LA, Serruys PW, et al. Late outcome after stenting or coronary artery bypass surgery for the treatment of multivessel disease: a single-center matched-propensity controlled cohort study. Ann Thorac Surg 2005;79:1563-1569.[Abstract/Free Full Text]
19. Okereke I, Murthy SC, Alster JM, Blackstone EH, Rice TW. Characterization and importance of air leak after lobectomy. Ann Thorac Surg 2005;79:1167-1173.[Abstract/Free Full Text]
20. Lam BK, Gillinov AM, Blackstone EH, Rajeswaran J, Yuh B, Bhudia SK, et al. Importance of moderate ischemic mitral regurgitation. Ann Thorac Surg 2005;79:462-470.[Abstract/Free Full Text]
21. Di Mauro M, Iaco AL, Contini M, Teodori G, Vitolla G, Pano M, et al. Reoperative coronary artery bypass grafting: analysis of early and late outcomes. Ann Thorac Surg 2005;79:81-87.[Abstract/Free Full Text]
22. Lytle BW, Blackstone EH, Sabik JF, Houghtaling P, Loop FD, Cosgrove DM. The effect of bilateral internal thoracic artery grafting on survival during 20 postoperative years. Ann Thorac Surg 2004;78:2005-2012.[Abstract/Free Full Text]
23. Svensson LG, Blackstone EH, Rajeswaran J, Sabik 3rd JF, Lytle BW, Gonzalez-Stawinski G, et al. Does the arterial cannulation site for circulatory arrest influence stroke risk?. Ann Thorac Surg 2004;78:1274-1284.[Abstract/Free Full Text]
24. Baskett RJ, Kalavrouziotis D, Buth KJ, Hirsch GM, Sullivan JA. Training residents in mitral valve surgery. Ann Thorac Surg 2004;78:1236-1240.[Abstract/Free Full Text]
25. Diodato MD, Moon MR, Pasque MK, Barner HB, Moazami N, Lawton JS, et al. Repair of ischemic mitral regurgitation does not increase mortality or improve long-term survival in patients undergoing coronary artery revascularization: a propensity analysis. Ann Thorac Surg 2004;78:794-799.[Abstract/Free Full Text]
26. Karthik S, Grayson AD, McCarron EE, Pullan DM, Desmond MJ. Reexploration for bleeding after coronary artery bypass surgery: risk factors, outcomes, and the effect of time delay. Ann Thorac Surg 2004;78:527-534.[Abstract/Free Full Text]
27. Quader MA, McCarthy PM, Gillinov AM, Alster JM, Cosgrove 3rd DM, Lytle BW, et al. Does preoperative atrial fibrillation reduce survival after coronary artery bypass grafting?. Ann Thorac Surg 2004;77:1514-1522.[Abstract/Free Full Text]
28. Sabik JF, Nemeh H, Lytle BW, Blackstone EH, Gillinov AM, Rajeswaran J, et al. Cannulation of the axillary artery with a side graft reduces morbidity. Ann Thorac Surg 2004;77:1315-1320.[Abstract/Free Full Text]
29. Coselli JS, LeMaire SA, Conklin LD, Adams GJ. Left heart bypass during descending thoracic aortic aneurysm repair does not reduce the incidence of paraplegia. Ann Thorac Surg 2004;77:1298-1303.[Abstract/Free Full Text]
30. Guller U, Anstrom KJ, Holman WL, Allman RM, Sansom M, Peterson ED. Outcomes of early extubation after bypass surgery in the elderly. Ann Thorac Surg 2004;77:781-788.[Abstract/Free Full Text]
31. Khan JH, Lambert AM, Habib JH, Broce M, Emmett MS, Davis EA. Abdominal complications after heart surgery. Ann Thorac Surg 2006;82:1796-1801.[Abstract/Free Full Text]
32. Reddy SL, Grayson AD, Oo AY, Pullan, MD, Poonacha T, Fabri BM. Does off-pump surgery offer benefit in high respiratory risk patients?. A respiratory risk stratified analysis in a propensity-matched cohort. Eur J Cardiothorac Surg 2006;30:126-131.[Free Full Text]
33. Reeves BC, Ascione R, Caputo M, Angelini GD. Morbidity and mortality following acute conversion from off-pump to on-pump coronary surgery. Eur J Cardiothorac Surg 2006;29:941-947.[Free Full Text]
34. Pai KR, Ramnarine IR, Grayson AD, Mediratta NK. The effect of chronic steroid therapy on outcomes following cardiac surgery: a propensity-matched analysis. Eur J Cardiothorac Surg 2005;28:138-142.[Abstract/Free Full Text]
35. Ali IS, Buth KJ. Preoperative statin use and in-hospital outcomes following heart surgery in patients with unstable angina. Eur J Cardiothorac Surg 2005;27:1051-1056.[Abstract/Free Full Text]
36. Frankel TL, Stamou SC, Lowery RC, Kapetanakis EI, Hill PC, Haile E, et al. Risk factors for hemorrhage-related reexploration and blood transfusion after conventional versus coronary revascularization without cardiopulmonary bypass. Eur J Cardiothorac Surg 2005;27:494-500.[Abstract/Free Full Text]
37. Brunelli A, Sabbatini A, Xiume’ F, Borri A, Salati M, Marasco RD, et al. Inability to perform maximal stair climbing test before lung resection: a propensity score analysis on early outcome. Eur J Cardiothorac Surg 2005;27:367-372.[Abstract/Free Full Text]
38. Pandey R, Grayson AD, Pullan DM, Fabri BM, Dihmis WC. Total arterial revascularisation: effect of avoiding cardiopulmonary bypass on in-hospital mortality and morbidity in a propensity-matched cohort. Eur J Cardiothorac Surg 2005;27:94-98.[Abstract/Free Full Text]
39. Stamou SC, Jablonski KA, Garcia JM, Boyce SW, Bafi AS, Corso PJ. Operative mortality after conventional versus coronary revascularization without cardiopulmonary bypass. Eur J Cardiothorac Surg 2004;26:549-553.[Abstract/Free Full Text]
40. Calafiore AM, Di Giammarco G, Teodori G, Di Mauro M, Iaco AL, Bivona A, et al. Late results of first myocardial revascularization in multiple vessel disease: single versus bilateral internal mammary artery with or without saphenous vein grafts. Eur J Cardiothorac Surg 2004;26:542-548.[Abstract/Free Full Text]
41. Ben-Gal Y, Mohr R, Uretzky G, Medalion B, Hendler A, Hansson N, et al. Drug-eluting stents versus arterial myocardial revascularization in patients with diabetes mellitus. J Thorac Cardiovasc Surg 2006;132:861-866.[Abstract/Free Full Text]
42. Guru V, Fremes SE, Tu JV. How many arterial grafts are enough? A population-based study of midterm outcomes. J Thorac Cardiovasc Surg 2006;131:1021-1028.[Abstract/Free Full Text]
43. Clark LL, Ikonomidis JS, Crawford Jr FA, Crumbley 3rd A, Kratz JM, Stroud MR, et al. Preoperative statin treatment is associated with reduced postoperative mortality and morbidity in patients undergoing cardiac surgery: an 8-year retrospective cohort study. J Thorac Cardiovasc Surg 2006;131:679-685.[Abstract/Free Full Text]
44. Svensson LG, Mumtaz MA, Blackstone EH, Feng J, Banbury MK, Sabik 3rd JF, et al. Does use of a right internal thoracic artery increase deep wound infection and risk after previous use of a left internal thoracic artery?. J Thorac Cardiovasc Surg 2006;131:609-613.[Abstract/Free Full Text]
45. Smedira NG, Blackstone EH, Roselli EE, Laffey CC, Cosgrove DM. Are allografts the biologic valve of choice for aortic valve replacement in nonelderly patients?. Comparison of explantation for structural valve deterioration of allograft and pericardial prostheses. J Thorac Cardiovasc Surg 2006;131:558-564e4.[Abstract/Free Full Text]
46. Stamou SC, Hill PC, Haile E, Prince S, Mack MJ, Corso PJ. Clinical outcomes of nonelective coronary revascularization with and without cardiopulmonary bypass. J Thorac Cardiovasc Surg 2006;131:28-33.[Abstract/Free Full Text]
47. Rajakaruna C, Rogers CA, Angelini GD, Ascione R. Risk factors for and economic implications of prolonged ventilation after cardiac surgery. J Thorac Cardiovasc Surg 2005;130:1270-1277.[Abstract/Free Full Text]
48. Kunihara T, Grun T, Aicher D, Langer F, Adam O, Wendler O, et al. Hypothermic circulatory arrest is not a risk factor for neurologic morbidity in aortic surgery: a propensity score analysis. J Thorac Cardiovasc Surg 2005;130:712-718.[Abstract/Free Full Text]
49. Roselli EE, Murthy SC, Rice TW, Houghtaling PL, Pierce CD, Karchmer DP, et al. Atrial fibrillation complicating lung cancer resection. J Thorac Cardiovasc Surg 2005;130:438-444.[Abstract/Free Full Text]
50. Calafiore AM, Di Giammarco G, Teodori G, Iaco AL, Pano M, Contini M, et al. Bilateral internal thoracic artery grafting with and without cardiopulmonary bypass: six-year clinical outcome. J Thorac Cardiovasc Surg 2005;130:340-345.[Abstract/Free Full Text]
51. Ercan S, Rice TW, Murthy SC, Rybicki LA, Blackstone EH. Does esophagogastric anastomotic technique influence the outcome of patients with esophageal cancer?. J Thorac Cardiovasc Surg 2005;129:623-631.[Abstract/Free Full Text]
52. Kumpati GS, Cook DJ, Blackstone EH, Rajeswaran J, Abdo AS, Young JB, et al. HLA sensitization in ventricular assist device recipients: does type of device make a difference?. J Thorac Cardiovasc Surg 2004;127:1800-1807.[Abstract/Free Full Text]
53. Kim K, Rice TW, Murthy SC, DeCamp MM, Pierce CD, Karchmer DP, et al. Combined bronchoscopy, mediastinoscopy, and thoracotomy for lung cancer: who benefits?. J Thorac Cardiovasc Surg 2004;127:850-856.[Abstract/Free Full Text]
54. Wijeysundera DN, Beattie WS, Rao V, Ivanov J, Karkouti K. Calcium antagonists are associated with reduced mortality after cardiac surgery: a propensity analysis. J Thorac Cardiovasc Surg 2004;127:755-762.[Abstract/Free Full Text]
55. Sharony R, Grossi EA, Saunders PC, Galloway AC, Applebaum R, Ribakove GH, et al. Propensity case-matched analysis of off-pump coronary artery bypass grafting in patients with atheromatous aortic disease. J Thorac Cardiovasc Surg 2004;127:406-413.[Abstract/Free Full Text]
56. Mack MJ, Pfister A, Bachand D, Emery R, Magee MJ, Connolly M, et al. Comparison of coronary bypass surgery with and without cardiopulmonary bypass in patients with multivessel disease. J Thorac Cardiovasc Surg 2004;127:167-173.[Abstract/Free Full Text]
57. Legare JF, Buth KJ, Sullivan JA, Hirsch GM. Composite arterial grafts versus conventional grafting for coronary artery bypass grafting. J Thorac Cardiovasc Surg 2004;127:160-166.[Abstract/Free Full Text]
58. Sabik JF, Blackstone EH, Lytle BW, Houghtaling PL, Gillinov AM, Cosgrove DM. Equivalent midterm outcomes after off-pump and on-pump coronary surgery. J Thorac Cardiovasc Surg 2004;127:142-148.[Abstract/Free Full Text]
59. Koch CG, Khandwala F, Cywinski JB, Ishwaran H, Estafanous FG, Loop FD, et al. Heath-related quality of life after coronary artery bypass grafting: a gender analysis using the Duke Activity Status Index. J Thorac Cardiovasc Surg 2004;128:284-295.[Abstract/Free Full Text]
60. Kaw R, Golish J, Ghamande S, Burgess R, Foldvary N, Walker E. Incremental risk of obstructive sleep apnea on cardiac surgical outcomes. J Cardiovasc Surg 2006;47:683-689.[Medline]

This article has been cited by other articles:

				P. A. Hayward, C. H. Yap, W. Y. Shi, B. F. Buxton, D. T. Dinh, C. M. Reid, G. C. Shardey, and J. A. Smith Does the addition of a radial artery graft improve survival after higher risk coronary artery bypass grafting? A propensity-score analysis of a multicentre database Eur J Cardiothorac Surg, March 18, 2013; (2013) ezt116v1. [Abstract] [Full Text] [PDF]

				F. F. Viana, W. Y. Shi, P. A. Hayward, M. E. Larobina, F. Liskaser, and G. Matalanis Custodiol versus blood cardioplegia in complex cardiac operations: an Australian experience Eur J Cardiothorac Surg, March 1, 2013; 43(3): 526 - 531. [Abstract] [Full Text] [PDF]

				L. E. Wee, G. C.-H. Koh, W. X. Yeo, R. T. Chin, J. Wong, and B. Seow Screening for cardiovascular disease risk factors in an urban low-income setting at baseline and post intervention: a prospective intervention study European Journal of Cardiovascular Prevention & Rehabilitation, February 1, 2013; 20(1): 176 - 188. [Abstract] [Full Text] [PDF]

				J. K. Mortensen, H. Larsson, S. P. Johnsen, and G. Andersen Post Stroke Use of Selective Serotonin Reuptake Inhibitors and Clinical Outcome Among Patients With Ischemic Stroke: A Nationwide Propensity Score-matched Follow-up Study Stroke, February 1, 2013; 44(2): 420 - 426. [Abstract] [Full Text] [PDF]

				A. Saxena, W. Y. Shi, S. Bappayya, D. T. Dinh, J. A. Smith, C. M. Reid, G. C. Shardey, and A. E. Newcomb Postoperative Atrial Fibrillation After Isolated Aortic Valve Replacement: A Cause for Concern? Ann. Thorac. Surg., January 1, 2013; 95(1): 133 - 140. [Abstract] [Full Text] [PDF]

				M. A. Quail, A. Frigiola, A. Giardini, V. Muthurangu, M. Hughes, P. Lurz, S. Khambadkone, J. E. Deanfield, V. Tsang, and A. M. Taylor Impact of Pulmonary Valve Replacement in Tetralogy of Fallot With Pulmonary Regurgitation: A Comparison of Intervention and Nonintervention Ann. Thorac. Surg., November 1, 2012; 94(5): 1619 - 1626. [Abstract] [Full Text] [PDF]

				A. D'Onofrio, A. Messina, R. Lorusso, O. R. Alfieri, M. Fusari, P. Rubino, M. Rinaldi, R. Di Bartolomeo, M. Glauber, G. Troise, et al. Sutureless aortic valve replacement as an alternative treatment for patients belonging to the "gray zone" between transcatheter aortic valve implantation and conventional surgery: A propensity-matched, multicenter analysis J. Thorac. Cardiovasc. Surg., November 1, 2012; 144(5): 1010 - 1018. [Abstract] [Full Text] [PDF]

				A. Hogh, J. S. Lindholt, H. Nielsen, L. P. Jensen, and S. P. Johnsen Use of Angiotensin-Converting Enzyme Inhibitors and Cardiovascular Outcomes Following Primary Vascular Surgery: A Nationwide Propensity Score Matched Follow-up Study Vascular and Endovascular Surgery, October 1, 2012; 46(7): 515 - 523. [Abstract] [PDF]

				J. Borgermann, K. Hakim, A. Renner, A. Parsa, A. Aboud, T. Becker, M. Masshoff, A. Zittermann, J. F. Gummert, and O. Kuss Clampless Off-Pump Versus Conventional Coronary Artery Revascularization: A Propensity Score Analysis of 788 Patients Circulation, September 11, 2012; 126(11_suppl_1): S176 - S182. [Abstract] [Full Text] [PDF]

				G. S. Collins and Y. Le Manach Comparing treatment effects between propensity scores and randomized controlled trials: improving conduct and reporting Eur. Heart J., August 1, 2012; 33(15): 1867 - 1869. [Full Text] [PDF]

				F. Baudo, P. Collins, A. Huth-Kuhne, H. Levesque, P. Marco, L. Nemes, F. Pellegrini, L. Tengborn, P. Knoebl, and on behalf of the EACH2 registry contributors Management of bleeding in acquired hemophilia A: results from the European Acquired Haemophilia (EACH2) Registry Blood, July 5, 2012; 120(1): 39 - 46. [Abstract] [Full Text] [PDF]

				T. A. MacKenzie, J. R. Brown, D. S. Likosky, Y. Wu, and G. L. Grunkemeier Review of Case-Mix Corrected Survival Curves Ann. Thorac. Surg., May 1, 2012; 93(5): 1416 - 1425. [Abstract] [Full Text] [PDF]

				D. Goksedef, G. Panagopoulos, N. Nassiri, R. L. Levine, P. G. Hountis, and K. A. Plestis Intraoperative use of recombinant activated factor VII during complex aortic surgery J. Thorac. Cardiovasc. Surg., May 1, 2012; 143(5): 1198 - 1204. [Abstract] [Full Text] [PDF]

				P.-C. Chen, Y.-C. Lee, and R. N. Kuo Differences in patient reports on the quality of care in a diabetes pay-for-performance program between 1 year enrolled and newly enrolled patients Int. J. Qual. Health Care, April 1, 2012; 24(2): 189 - 196. [Abstract] [Full Text] [PDF]

				M. Noda, Y. Okada, S. Maeda, T. Sado, A. Sakurada, Y. Hoshikawa, C. Endo, and T. Kondo Is there a benefit of awake thoracoscopic surgery in patients with secondary spontaneous pneumothorax? J. Thorac. Cardiovasc. Surg., March 1, 2012; 143(3): 613 - 616. [Abstract] [Full Text] [PDF]

				C. Wu, F. T. Camacho, A. T. Culliford, J. P. Gold, A. S. Wechsler, R. S. D. Higgins, S. J. Lahey, C. R. Smith, D. Jordan, and E. L. Hannan A Comparison of Long-Term Mortality for Off-Pump and On-Pump Coronary Artery Bypass Graft Surgery Circ Cardiovasc Qual Outcomes, January 1, 2012; 5(1): 76 - 84. [Abstract] [Full Text] [PDF]

				C. Wu, S. Zhao, A. S. Wechsler, S. Lahey, G. Walford, A. T. Culliford, J. P. Gold, C. R. Smith, D. R. Holmes Jr, S. B. King III, et al. Long-Term Mortality of Coronary Artery Bypass Grafting and Bare-Metal Stenting Ann. Thorac. Surg., December 1, 2011; 92(6): 2132 - 2138. [Abstract] [Full Text] [PDF]

				F. Webster, G. Saposnik, M. K. Kapral, J. Fang, C. O'Callaghan, and V. Hachinski Organized Outpatient Care: Stroke Prevention Clinic Referrals Are Associated With Reduced Mortality After Transient Ischemic Attack and Ischemic Stroke Stroke, November 1, 2011; 42(11): 3176 - 3182. [Abstract] [Full Text] [PDF]

				C. Wu, A.-M. Dyer, S. B. King III, G. Walford, D. R. Holmes Jr, N. J. Stamato, F. J. Venditti, S. K. Sharma, I. Fergus, A. K. Jacobs, et al. Impact of Incomplete Revascularization on Long-Term Mortality After Coronary Stenting Circ Cardiovasc Interv, October 1, 2011; 4(5): 413 - 421. [Abstract] [Full Text] [PDF]

				W. Y. Shi, P. A. Hayward, C.-H. Yap, D. T. Dinh, C. M. Reid, G. C. Shardey, and J. A. Smith Training in mitral valve surgery need not affect early outcomes and midterm survival: a multicentre analysis Eur J Cardiothorac Surg, October 1, 2011; 40(4): 826 - 833. [Abstract] [Full Text] [PDF]

				E. Ruttmann, N. Fischler, A. Sakic, O. Chevtchik, H. Alber, R. Schistek, H. Ulmer, and M. Grimm Second Internal Thoracic Artery Versus Radial Artery in Coronary Artery Bypass Grafting: A Long-Term, Propensity Score-Matched Follow-Up Study Circulation, September 20, 2011; 124(12): 1321 - 1329. [Abstract] [Full Text] [PDF]

				R. R. Quinn, J. E. Hux, M. J. Oliver, P. C. Austin, M. Tonelli, and A. Laupacis Selection Bias Explains Apparent Differential Mortality between Dialysis Modalities J. Am. Soc. Nephrol., August 1, 2011; 22(8): 1534 - 1542. [Abstract] [Full Text] [PDF]

				D. N. Wijeysundera, W. S. Beattie, K. Karkouti, M. D. Neuman, P. C. Austin, and A. Laupacis Association of echocardiography before major elective non-cardiac surgery with postoperative survival and length of hospital stay: population based cohort study BMJ, June 30, 2011; 342(jun30_1): d3695 - d3695. [Abstract] [Full Text] [PDF]

				V. A. Ferraris, D. L. Davenport, S. P. Saha, A. Bernard, P. C. Austin, and J. B. Zwischenberger Intraoperative Transfusion of Small Amounts of Blood Heralds Worse Postoperative Outcome in Patients Having Noncardiac Thoracic Operations Ann. Thorac. Surg., June 1, 2011; 91(6): 1674 - 1680. [Abstract] [Full Text] [PDF]

				S. R. JOHNSON, J. T. GRANTON, G. A. TOMLINSON, H. A. GROSBEIN, G. A. HAWKER, and B. M. FELDMAN Effect of Warfarin on Survival in Scleroderma-associated Pulmonary Arterial Hypertension (SSc-PAH) and Idiopathic PAH. Belief Elicitation for Bayesian Priors J Rheumatol, March 1, 2011; 38(3): 462 - 469. [Abstract] [Full Text] [PDF]

				D. M. Holzhey, W. Shi, M. A. Borger, J. Seeburger, J. Garbade, B. Pfannmuller, and F. W. Mohr Minimally Invasive Versus Sternotomy Approach for Mitral Valve Surgery in Patients Greater Than 70 Years Old: A Propensity-Matched Comparison Ann. Thorac. Surg., February 1, 2011; 91(2): 401 - 405. [Abstract] [Full Text] [PDF]

				M. M. Mokhles, H. Kortke, U. Stierle, O. Wagner, E. I. Charitos, A. J. J. C. Bogers, J. Gummert, H.-H. Sievers, and J. J. M. Takkenberg Survival Comparison of the Ross Procedure and Mechanical Valve Replacement With Optimal Self-Management Anticoagulation Therapy: Propensity-Matched Cohort Study Circulation, January 4, 2011; 123(1): 31 - 38. [Abstract] [Full Text] [PDF]

				P. C. Austin Statistical Criteria for Selecting the Optimal Number of Untreated Subjects Matched to Each Treated Subject When Using Many-to-One Matching on the Propensity Score Am. J. Epidemiol., November 1, 2010; 172(9): 1092 - 1097. [Abstract] [Full Text] [PDF]

				O. Kuss, B. von Salviati, and J. Borgermann Off-pump versus on-pump coronary artery bypass grafting: A systematic review and meta-analysis of propensity score analyses J. Thorac. Cardiovasc. Surg., October 1, 2010; 140(4): 829 - 835.e13. [Abstract] [Full Text] [PDF]

				S. Bramer, A. H. M. van Straten, M. A. Soliman Hamad, E. Berreklouw, E. J. Martens, and J. G. Maessen The impact of preoperative atrial fibrillation on early and late mortality after coronary artery bypass grafting Eur J Cardiothorac Surg, September 1, 2010; 38(3): 373 - 379. [Abstract] [Full Text] [PDF]

				A. Piccardo, D. Rusinaru, B. Petitprez, P. Marticho, I. Vaida, C. Tribouilloy, and T. Caus Thrombocytopenia After Aortic Valve Replacement With Freedom Solo Bioprosthesis: A Propensity Study Ann. Thorac. Surg., May 1, 2010; 89(5): 1425 - 1430. [Abstract] [Full Text] [PDF]

				A. D'Onofrio, D. Cugola, I. Bolgan, L. Menicanti, A. Fabbri, and M. Di Donato Surgical ventricular reconstruction with different myocardial protection strategies. A propensity matched analysis Interact CardioVasc Thorac Surg, April 1, 2010; 10(4): 530 - 534. [Abstract] [Full Text] [PDF]

				D. N Wijeysundera, W S. Beattie, P. C Austin, J. E Hux, and A. Laupacis Non-invasive cardiac stress testing before elective major non-cardiac surgery: population based cohort study BMJ, January 28, 2010; 340(jan28_3): b5526 - b5526. [Abstract] [Full Text] [PDF]

				E. L. Hannan, Y. Zhong, M. Racz, A. K. Jacobs, G. Walford, K. Cozzens, D. R. Holmes, R. H. Jones, M. Hibberd, D. Doran, et al. Outcomes for Patients With ST-Elevation Myocardial Infarction in Hospitals With and Without Onsite Coronary Artery Bypass Graft Surgery: The New York State Experience Circ Cardiovasc Interv, December 1, 2009; 2(6): 519 - 527. [Abstract] [Full Text] [PDF]

				P. C. Austin The Relative Ability of Different Propensity Score Methods to Balance Measured Covariates Between Treated and Untreated Subjects in Observational Studies Med Decis Making, November 1, 2009; 29(6): 661 - 677. [Abstract] [PDF]

				C.-J. Jakobsen, F. Sondergaard, V. E. Hjortdal, and S. P. Johnsen Use of aprotinin in cardiac surgery: effectiveness and safety in a population-based study Eur J Cardiothorac Surg, November 1, 2009; 36(5): 863 - 868. [Abstract] [Full Text] [PDF]

				S. G. Raja, H. Siddiqui, C. D. Ilsley, and M. Amrani In-Hospital Outcomes of Off-Pump Multivessel Total Arterial and Conventional Coronary Artery Bypass Grafting: Single Surgeon, Single Center Experience Ann. Thorac. Surg., July 1, 2009; 88(1): 47 - 52. [Abstract] [Full Text] [PDF]

				G. Rocco, E. A. Rendina, F. Venuta, M. R. Mueller, S. Halezeroglu, H. Dienemann, D. V. Raemdonck, and H. J. Hansen The use of sealants in modern thoracic surgery: a survey Interact CardioVasc Thorac Surg, July 1, 2009; 9(1): 1 - 3. [Full Text] [PDF]

				M. Chiu, D. T. Ko, P. C. Austin, E. A. Cohen, J. L. Velianou, R. Goeree, G. Blackhouse, and J. V. Tu Paclitaxel Versus Sirolimus Stents in Diabetic and Nondiabetic Patients Circ Cardiovasc Qual Outcomes, March 1, 2009; 2(2): 96 - 107. [Abstract] [Full Text] [PDF]

				G. N. Vaccarino, J. Thierer, M. Albertal, M. Vrancic, F. Piccinini, M. Benzadon, H. Raich, and D. O. Navia Impact of preoperative clopidogrel in off pump coronary artery bypass surgery: a propensity score analysis. J. Thorac. Cardiovasc. Surg., February 1, 2009; 137(2): 309 - 313. [Abstract] [Full Text] [PDF]

				P. C. Austin Primer on Statistical Interpretation or Methods Report Card on Propensity-Score Matching in the Cardiology Literature From 2004 to 2006: A Systematic Review Circ Cardiovasc Qual Outcomes, September 1, 2008; 1(1): 62 - 67. [Abstract] [Full Text] [PDF]

This Article Abstract Full Text (PDF) Alert me when this article is cited Alert me if a correction is posted Citation Map Services Email this article to a friend Similar articles in this journal Similar articles in PubMed Alert me to new issues of the journal Add to Personal Folders Download to citation manager Permission Requests Citing Articles Citing Articles via HighWire Citing Articles via Google Scholar Google Scholar Articles by Austin, P. C. Search for Related Content PubMed PubMed Citation Articles by Austin, P. C.