| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
J Thorac Cardiovasc Surg 2006;132:249-251
© 2006 The American Association for Thoracic Surgery
Statistics for the Rest of Us |
Departments of Medicine and Epidemiology and Biostatistics, Lerner Medical College of Case Western Reserve University, and the Department of Cardiovascular Medicine, The Cleveland Clinic Foundation, Cleveland, Ohio
Received for publication November 21, 2005; accepted for publication March 3, 2006. * Address for reprints: Michael S. Lauer, MD, FACC, FAHA, The Cleveland Clinic Foundation, 9500 Euclid Ave/Desk JJ40, Cleveland, OH 44122 (Email: Lauerm{at}ccf.org).
Clinical trials are the medical community's diagnostic tests. When a physician sends a patient for a conventional diagnostic test, he or she is asking whether a certain disease is likely to be present. Similarly, when the medical community designs and executes a clinical trial, it wants to know whether a certain treatment is likely to work. And just like a diagnostic test, a clinical trial has the potential for yielding an incorrect, erroneous result.
The traditional approach to reporting probability of error in clinical trials includes P values and statements of power. The P value is the probability that a positive trial result is just the result of chance, given an assumed truth of the null hypothesis, which is that the treatment does not work. Power refers to the ability of a trial to show that a treatment works, given that in truth it actually does. Although presenting P values and statements of power has become lore in reporting clinical trials, careful reflection shows that clinicians might not gain as much value from them as they might think.
When a clinician reads a diagnostic test report that is "positive," the next question is, "How likely is it that my patient has the disease?" That is, what is the likelihood that this positive report is in fact correct? As described by Bayesian theory, the likelihood of a correct result given a positive test result, or positive predictive value, is related not only to the sensitivity and specificity of the test but also to the pretest likelihood of disease being present.
1
Similarly, when a clinical trial report is presented to the medical community suggesting that a treatment works, the critical question that clinicians should ask is, "Is the trial likely to be reporting a correct result?" What is the positive predictive value of the trial? To estimate this, we need to know not only the P value and the power but also something about the pretrial likelihood of a positive result, as recently argued by Ioannidis
2
and O'brien and Castelloe.
3
Bayesian thinking applied to clinical trials is shown schematically in Table 1
(taken from Ioannides
2
), which shows an example of an experimental treatment in which the pretrial likelihood of success is considered to be low (ie, 10%). This might be because we are testing a new and innovative treatment for which there is little prior experience. We imagine a universe in which 1000 trials are performed, one of which happens to be the one reported in the journal resting between our hands. Each trial is small and has a power of only 50% because funding is limited and single centers have few eligible patients. Traditionally, a trial result would be considered positive if the P value is .05 or less. Given the pretrial expectation that 100 of 1000 trial results will be positive and given a power of 50%, one would expect 50 positive trial results, which is shown in the upper left hand box of Table 1. Conversely, among the 900 predicted negative trial results supporting the null hypothesis, we would expect 45 positive trial results, given a P value of .05. Thus of 1000 trials in this imaginary universe, we predict 95 positive trial results, of which 50 are true-positive results and 45 are false-positive results. The positive predictive value for any of the positive trial results turns out to be only 52%.
|
Although this hypothetical example might seem far-fetched, in fact, most medical literature is plagued by problems with low pretest probability and underpowered samples. As summarized by Ioannidis,
2
much of what is published in the medical literature might actually be false.
And the problem worsens.
A reported positive trial result might be incorrect not only because of a low pretrial likelihood but also because of bias. Bias refers to a fundamental problem in the way the trial is designed, analyzed, or reported that results in its result being reported as positive when in fact it is not. There are many well-known sources of bias. As mentioned by Tiruvoipati and colleagues,
4
bias can result from a lack of unpredictable allocation, inadequate concealment of allocation from investigators, inadequate blinding, inadequate sample size, and failure to use the intent-to-treat method for analyzing results.
Yet another problem arises from performing multiple trials of the same treatment.
2
Although it seems to be scientifically intuitive that one wants to reproduce results, performance of multiple trials by multiple groups leads to an increased risk of false-positive findings because of a standard, multiple testing problem.
5,6
When one puts together low pretest likelihood, bias, and multiple trials, one is left with a high likelihood of false-positive trial results (Figures 1 and 2).
|
|
Trials should have high power and little bias. I therefore agree with Tiruvoipati and colleague's call
4
for widespread adoption of the CONSORT statement
8
for reporting trials in cardiac surgery.
Tiruvoipati and colleagues
4
are to be congratulated for their systematic and critical evaluation of the cardiac surgery literature and for their challenge to its journal editors. Routine incorporation of the CONSORT guidelines, as exemplified by 5 recently published cardiac surgery trials,
9-13
will improve the transparency of reporting and decrease the likelihood of bias,
14
thereby increasing the predictive value of one of the medical community's most precious diagnostic tests, the randomized trial.
| See related articles on pages 229, 233, 241, 243, 245, and 247.
|
 |
References |
|---|
 Top References |
|---|
Related Articles
This article has been cited by other articles:
|
|
 |
|
  G. L. Grunkemeier, Y. Wu, and A. P. Furnary What is the value of a p value? Ann. Thorac. Surg., May 1, 2009; 87(5): 1337 - 1343. [Abstract] [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
value is the P value for significance, in this case .05. Also, in this case power is high at 90%. Bias is represented by µ, with values of 0 and 0.1 corresponding to little bias (in design, analysis, and reporting) and 0.4 corresponding to a substantial degree of bias (intermediate curve, µ = 0.2). Note the dramatic decrease in posttrial probability as bias increases and pretrial probability decreases. This type of scenario might be common for small trials of highly innovative treatments.