| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
|||||||
|
|||||||||
J Thorac Cardiovasc Surg 2006;132:233-240
© 2006 The American Association for Thoracic Surgery
Statistics for the Rest of Us |
a Department of Cardiac Surgery and ECMO, Glenfield Hospital, Leicester, Leicester, United Kingdom
b Academic Unit of Surgical Oncology, University of Sheffield, Sheffield, United Kingdom
c London School of Hygiene and Tropical Medicine, University of London, London, United Kingdom
Received for publication June 27, 2005; revisions received September 13, 2005; accepted for publication October 28, 2005. * Address for reprints: Ravindranath Tiruvoipati, FRCSEd, Department of ECMO, Glenfield Hospital, Groby Rd, Leicester, LE3 9QP, UK (Email: rtiruvoipati{at}yahoo.co.uk).
 |
Abstract |
|---|
 Top Abstract IntroductionMethodsResultsDiscussionReferences |
|---|
METHODS: Randomized controlled trials of cardiothoracic surgery published in principal cardiothoracic and 4 general medical journals in 2003 were included. The quality of reporting of randomized controlled trials was assessed by using allocation concealment, the Jadad score, and a Consolidated Standards for Reporting of Trials checklist devised for the purpose. A questionnaire survey of authors reporting randomized controlled trials in principal cardiothoracic journals in 2003 was conducted.
RESULTS: The overall reporting quality of the 64 randomized controlled trials included in the analysis was suboptimal as assessed by the 3 methods adopted. Most of the authors (63.5%) were not aware of the Consolidated Standards for Reporting of Trials statement; however, awareness was not associated with reporting quality. More than 65% of the authors responded that conducting randomized controlled trials in surgical specialties was difficult, and the main difficulties were blinding and obtaining a large-enough sample size to detect statistically significant differences. Fifty-four percent of the authors responded that endorsement of the Consolidated Standards for Reporting of Trials statement by the cardiothoracic journals may improve the reporting quality.
CONCLUSIONS: The quality of reporting randomized controlled trials in cardiothoracic surgery is suboptimal. Endorsement of the Consolidated Standards for Reporting of Trials statement by the cardiothoracic journals may improve the quality of reporting.
|
Introduction |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Well-designed, properly conducted, and clearly reported randomized controlled trials (RCTs) provide the best evidence for health care professionals to make appropriate clinical decisions about the relative effectiveness of different interventions.
1
High-quality reports are likely to improve the interpretation of RCTs and minimize biased conclusions.
2
There is, however, strong evidence to indicate that the quality of the reporting of RCTs in the medical literature is less than optimal.
3,4
To overcome such problems in reporting, the Consolidated Standards for Reporting of Trials (CONSORT) Group developed the CONSORT statement
5
in 1996, followed by a revised version
3
in 2001, which was accompanied by a longer document providing further elaboration and explanation.
1
The CONSORT statement has been widely supported and has been translated into several languages to facilitate awareness and dissemination.
6,7
The CONSORT statement was endorsed by several journals.
8
However, the principal cardiothoracic journals (Annals of Thoracic Surgery, European Journal of Cardiothoracic Surgery, Journal of Thoracic and Cardiovascular Surgery, and Asian Cardiovascular & Thoracic Annals) have not yet formally endorsed it.
Several studies have evaluated the reporting quality of RCTs published in medical and surgical journals,
3,9,10
and most have concluded that the reporting of trials may be further improved. One of the recent studies addressing the reporting of RCTs in cardiothoracic surgery during 1998 and 1999 concluded that the reporting is poor by the standards of the CONSORT statement.
11
This may be because of the lack of awareness of the CONSORT statement among the authors reporting the trials.
At least 25 scales have been described for the evaluation of quality of reporting RCTs,
12
among which the Jadad score is commonly used and has been validated by using established methodologic procedures.
13
The reporting of allocation concealment has been used to assess the reporting quality
14
of the RCTs.
The aims of this study were to evaluate the following:
|
Methods |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
|
Part of the methodology used to assess the quality of the RCT reports was similar to that described by Moher and colleagues.
15
The reporting of allocation concealment was assessed as adequate, inadequate, or unclear as described by Schulz and colleagues.
16
The Jadad scale was used, which contains 2 questions each for randomization and masking and 1 question evaluating the reporting of withdrawals and dropouts. Each question entailed yes or no response options. In total, 5 points were awarded, with a score of 3 or more indicating superior quality.
13
Allocation concealment and Jadad score were assessed by one observer (R.T.).
A checklist of 32 questions was then prepared from the 22 items of the revised CONSORT guidelines
1
published in 2001. The score for each item ranged from 1 to 3 (1, no description; 2, inadequate description; 3, adequate description). The CONSORT guidelines were studied in depth by the reviewers, and the definition of each checklist item was discussed. The checklist was then piloted independently by 3 observers (R.T., S.P.B., and D.E.) on 4 RCTs (published before 2003) to address any ambiguous questions and questions that might yield unusable data. After this, a final checklist involving 30 questions (Table 1
) was accepted for characterization of RCTs, excluding the first item (title and abstract, because the authors have to follow the instructions of the journal in preparing the abstract) of the CONSORT guidelines. Each article included in the study was then assessed for every item on the checklist and scored independently by 2 observers (R.T. and S.P.B.) to arrive at a consensus-modified CONSORT score. Discrepancies between the 2 observers were resolved by the third observer (D.E.). The scores for the 30 items were added, and a percentage score was calculated for each trial (some items were not applicable and did not merit any score).
|
The data were collected on an Excel spreadsheet (Microsoft Corp, Redmond, Wash) and exported to SPSS (version 12.0 for Windows; SPSS Inc, Chicago, Ill) for analysis. Initial presentations were descriptive, which included the median (interquartile range) Jadad scores, modified CONSORT scores, and percentages of the RCT reports reporting allocation concealment adequately of the individual journals and the specific characteristics of the studies. The association between study characteristics (number of authors, single center or multicenter, statistician/epidemiologist involvement, and funding source) and the awareness of authors about the CONSORT statement with the modified CONSORT score (measured as greater or less than the median) was studied. The responses of the authors on the views of conducting and reporting of RCTs were presented as percentages. Nonparametric tests were used for all analyses.
|
Results |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Table 2 shows the different journals along with their impact factor for 2002, number of articles analyzed (n = 64), endorsement of the CONSORT statement in their "Instructions to Authors" section, median Jadad score, and median modified CONSORT score, calculated as percentage and percentage of articles reporting allocation concealment. The agreement of the pair of observers who independently assessed the RCTs by using the CONSORT checklist was good (intraclass correlation coefficient, 0.85; 95% confidence interval, 0.76-0.90; P < .001). The 3 quality-assessment tools used in this study (Jadad score, allocation concealment, and modified CONSORT score) correlated moderately with each other (Jadad and allocation concealment: Spearman rho, 0.57; P < .001; Jadad and modified CONSORT: Spearman rho, 0.56; P < .001; allocation concealment and modified CONSORT: Spearman rho, 0.47; P < .001).
|
|
2 test with Yates correction: 2 = 0.117; P = .732), or involvement of a statistician/epidemiologist in the study (Fisher exact test: P = .091). Multicenter studies spanning across continents were associated with a higher-than-average quality of reporting (Kruskal-Wallis test: P = .005). Interpretation of this finding should be made with caution because of the small number (n = 3) of studies included in this group. A sensitivity analysis was conducted to include the 3 studies that had originally been excluded because their design and methods were reported before 2001. The overall median modified CONSORT Score (65.5; 61.7-71.3), median Jadad Score (2; 1-3), and percentage of articles reporting adequate allocate concealment (23.8) for the 67 studies did not differ significantly from the analysis of the original 64 included studies.
Authors of 52 RCTs (69% of the 75 RCTs in principal cardiothoracic journals) responded to the survey about the CONSORT statement by the cardiothoracic journals (Table 4). Nineteen (37%) were aware of CONSORT; half of the responders thought that cardiothoracic journals should endorse CONSORT, and the opinion of 28 (54%) was that this may improve reporting quality, although only 3 thought this would have meant reporting their trial differently. Three quarters of the respondents thought RCTs were difficult to conduct in surgical specialties.
|
|
Discussion |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
It is estimated that the number of RCTs is increasing by a factor of 10 every year.
18
The results of RCTs provide the most reliable evidence about which treatments are best for patients. It is therefore important that RCTs be reported in a high-quality manner so readers have a clear view on why the study was conducted, how it was conducted, and how it was analyzed. This would be helpful not only in the immediate appraisal of trials, but also in the long term, when further analyses are performed, such as in systematic reviews and meta-analyses. Our study shows that the reporting quality of RCTs in cardiothoracic surgery is suboptimal. Among the study characteristics assessed, only multicenter RCTs spanning across continents were found to have an above-average quality of reporting. Although the number of multicontinental reports (n = 3) was small, we speculate that this association (if real) may be due to the involvement of a wide range of experts at the inception and in the oversight of such large studies.
The main purpose of randomization is to eliminate selection bias, which is a crucial component for high-quality RCTs. In addition, randomization aims to create groups that are comparable in terms of any known or unknown potential confounding factors.
19
Proper randomization includes 2 important elements: generation of an unpredictable allocation sequence and concealment of this sequence from the investigators enrolling participants into the study. Unlike blinding, which may not be possible to practice at all times (especially with surgical trials), allocation concealment can always be successfully implemented.
4,16
The reporting of the generation of unpredictable allocation sequences was adequate in 20.3% (Table 1) and allocation concealment was adequate in only 22% of the reports (Table 2) in our study. Trials that have not reported adequate allocation concealment have been found to be associated with exaggerated treatment effects.
16
The reporting of blinding of participants, treatment providers, and assessors was adequate in 21.1%, 33.3%, and 37.5% of the reports, respectively. Most of the studies reported as "double-blinded" trials did not provide details on who was blinded. Similar exclusion of details on the double blinding was reported by Schulz and associates.
20
Omitting such details can create confusion and skepticism in interpreting the results. Studies have shown that nonrandomized trials and RCTs that do not incorporate blinding are more likely to show advantages of a new intervention over the standard treatment.
21
RCTs that have not reported blinding have been shown to exaggerate treatment effects.
16
One of the important recommendations of the CONSORT statement is to produce a flow diagram that describes the flow of participants through all the stages of trial. This was shown to improve the quality of reporting.
22
In our study, a flow diagram was shown in only 1.6% of the reports.
Prestudy sample size calculations based on a clearly defined outcome are considered essential for both scientific and ethical reasons. In our study, reporting of the prestudy sample size calculation was adequate in 20.3% of the reports. Studies with small sample sizes are often inadequately powered to detect small but clinically significant differences between interventions.
23
Intention-to-treat analyses are usually favored over per-protocol analyses because they avoid bias associated with nonrandom loss of participants.
24
Although such analyses may underestimate the real benefit of treatments for which noncompliance may be an issue, they address the effectiveness question, which is the more pragmatic approach to the application of the interventions in clinical practice. Intention-to-treat analyses were reported in 68.3% of the RCTs in this study.
The reporting of details that would aid in assessing the external validity was poor in most of the RCTs and systematic reviews. Trials lacking the details on external validity can lead to underuse of treatments that are effective.
25
Details that could be used to assess the external validity in our study were presented in 20.3% of the trials.
The need for improving the overall reporting of RCTs has been highlighted by several other studies.
4,10,11,26
Although the reporting of RCTs can give some clues on the quality of the methodology, poor reporting does not necessarily mean that the trials were poorly designed and executed. Well-conducted trials may be reported badly, and studies with poor methodologies can be reported in such as way as to hide important deficiencies.
14,27
The potential difficulties associated with performing RCTs in surgical practice include the difficulty in successfully blinding patients, investigators, and assessors; the variability of surgical techniques and operator skills; and the learning curve (where the technical performance of a new technique improves over time
28
), which influences the efficacy of many interventions under study. In addition, lack of training on the conduct of clinical trials during residencies and fellowships may also be an important element. These may be the reasons why RCTs in surgical specialties are comparatively fewer than in the medical specialties.
Nevertheless, although careful statistical techniques can help to reduce biases in nonrandomized studies, they cannot deal with the major problems of selection bias.
29
We therefore believe that well-conducted and properly reported RCTs are essential for improving the overall quality of surgical care.
The 2 elements that could potentially improve the reporting of RCTs in cardiothoracic surgery may be awareness of the CONSORT statement among authors and, more important than awareness, endorsement of the CONSORT statement by the cardiothoracic surgical journals. Although the dissemination of the CONSORT statement in several languages may have improved its awareness, our study shows that more than 60% of the authors who reported their studies in the principal cardiothoracic journals were not aware of it. This may be because of the publication of the CONSORT statement in general medical journals rather than the specialist surgical journals. Awareness of the statement, however, may not improve the quality of reporting on its own. In our study, awareness of the CONSORT statement among the authors was not significantly associated with improved quality of reporting. There is, however, evidence to suggest that the quality of reporting of RCTs has improved when the journals have endorsed the statement as compared with when the CONSORT statement was not endorsed.
15,22
The use of the CONSORT statement was also shown to positively influence the manner in which the RCTs were conducted.
3
The endorsement of the CONSORT statement by the journals may also help in improving the awareness of CONSORT statement because the authors would have to follow the "Instructions for Authors" of the journal in which they would like to publish their study.
In our study, more than 40% of authors replied that they may have reported their trial differently, and approximately 6% of authors said that they would have definitely reported their trial differently if the cardiothoracic journals had endorsed the CONSORT statement. As shown in Table 4, more than 50% of the authors believe that endorsement of the statement by cardiothoracic journals may further improve the reporting of RCTs in cardiothoracic surgery. Whether this belief of improvement in the reporting quality of RCTs in cardiothoracic surgery is true remains to be assessed in future research if the cardiothoracic journals endorse the CONSORT statement.
Our survey shows that most of the authors (65.4%) believe that conducting high-quality RCTs is more difficult in surgical specialties than in medical specialties, with the main difficulties including blinding of treatment providers and participants (74.3%) and obtaining a sufficient sample size to be able to draw conclusions (66.6%). These findings are similar to the views of Anyanwu and Treasure.
11
However, this should not deter surgical researchers in conducting RCTs to investigate the problems that could be answered best by this design.
Because of restrictions on resources and to preserve blinding of the reviewers to the source journals of the articles, the initial screening of articles was not performed separately by 2 people. This blinding may not, however, have been completely successful, because the reviewers may have previously encountered some of the articles assessed and could have been prejudiced by preexisting knowledge on the subject. In addition, the CONSORT checklist contained items whose scoring was subjective and dependent on reviewers' perceptions and domain knowledge. The assessment of the item of "allocation concealment" as part of the modified CONSORT checklist was slightly at variance with the assessment described by Schulz and associates
16
in 1995, which explains the difference in the scoring as shown in Tables 1 and 2. Finally, this study was restricted to cardiothoracic surgery (reflecting the interests of the authors and limitations in our time and other resources). The conclusions are not necessarily generalizable to trials in other areas of surgery or to medical trials.
In conclusion, the results of our study suggest that the reporting quality of RCTs in cardiothoracic surgery is suboptimal. Although studies spanning across continents (where numbers were small) were associated with above-average reporting quality, other study characteristics were not associated with reporting quality. Most of the authors were not aware of the CONSORT statement.
Because the results of RCTs will have a significant effect on clinical decision making at all levels, it is important to improve the quality of reporting. This may be achieved by the endorsement of the CONSORT statement by cardiothoracic journals, as well as increasing the awareness of the CONSORT statement among cardiothoracic surgical researchers.
| See related articles on pages 229, 241, 243, 245, 247, and 249.
|
|
References |
|---|
|
TopAbstractIntroductionMethodsResultsDiscussionReferences |
|---|
Related Articles
This article has been cited by other articles:
|
|
 |
|
  K. Moghissi Editorial comment: Are randomised trials needed in the era of rapidly evolving technologies? Eur. J. Cardiothorac. Surg., March 1, 2009; 35(3): 478 - 479. [Full Text] [PDF]
|
|
|
|
 |
|
 D. C. Bauer Randomized Trial Reporting in General Endocrine Journals: The Good, the Bad, and the Ugly J. Clin. Endocrinol. Metab., October 1, 2008; 93(10): 3733 - 3734. [Full Text] [PDF]
|
|
|
|
|
|
L. P. Rios, A. Odueyungbo, M. O. Moitri, M. O. Rahman, and L. Thabane Quality of Reporting of Randomized Controlled Trials in General Endocrinology Literature J. Clin. Endocrinol. Metab., October 1, 2008; 93(10): 3810 - 3816. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 S. A. Olivo, L. G. Macedo, I. C. Gadotti, J. Fuentes, T. Stanton, and D. J Magee Scales to Assess the Quality of Randomized Controlled Trials: A Systematic Review Physical Therapy, February 1, 2008; 88(2): 156 - 175. [Abstract] [Full Text] [PDF]
|
|
|
|
 |
|
 F. D. Rubens and H. Nathan Lessons learned on the path to a healthier brain: dispelling the myths and challenging the hypotheses Perfusion, May 1, 2007; 22(3): 153 - 160. [Abstract] [PDF]
|
|
|
|
 |
|
 R. M. Sade "Surgical research or comic opera" redux. Ann. Thorac. Surg., October 1, 2006; 82(4): 1173 - 1174. [Full Text] [PDF]
|
|
|
|
 |
|
 E. H. Blackstone CONSORT and beyond. J. Thorac. Cardiovasc. Surg., August 1, 2006; 132(2): 229 - 232. [Full Text] [PDF]
|
|
|
|
|
|
C. M. Balch Randomized clinical trials in surgery: Why do we need them? J. Thorac. Cardiovasc. Surg., August 1, 2006; 132(2): 241 - 242. [Full Text] [PDF]
|
|
|
|
|
|
N. F. Kassell and A. S. Dumont Randomized controlled trials in surgery: Comic opera no more? J. Thorac. Cardiovasc. Surg., August 1, 2006; 132(2): 243 - 244. [Full Text] [PDF]
|
|
|
|
|
|
R. M. Sade Reports of clinical trials: Ethical aspects. J. Thorac. Cardiovasc. Surg., August 1, 2006; 132(2): 245 - 246. [Full Text] [PDF]
|
|
|
|
|
|
S. Piantadosi Improving the quality of clinical trials in surgery. J. Thorac. Cardiovasc. Surg., August 1, 2006; 132(2): 247 - 248. [Full Text] [PDF]
|
|
|
|
|
|
M. S. Lauer Believability of clinical trials: A diagnostic testing perspective. J. Thorac. Cardiovasc. Surg., August 1, 2006; 132(2): 249 - 251. [Full Text] [PDF]
|
|
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
| ANN THORAC SURG | ASIAN CARDIOVASC THORAC ANN | EUR J CARDIOTHORAC SURG |
| J THORAC CARDIOVASC SURG | ICVTS | ALL CTSNet JOURNALS |
