- Research article
- Open Access
- Open Peer Review
How useful are systematic reviews for informing palliative care practice? Survey of 25 Cochrane systematic reviews
BMC Palliative Care volume 7, Article number: 13 (2008)
In contemporary medical research, randomised controlled trials are seen as the gold standard for establishing treatment effects where it is ethical and practical to conduct them. In palliative care such trials are often impractical, unethical, or extremely difficult, with multiple methodological problems. We review the utility of Cochrane reviews in informing palliative care practice.
Published reviews in palliative care registered with the Cochrane Pain, Palliative and Supportive Care Group as of December 2007 were obtained from the Cochrane Database of Systematic Reviews, issue 1, 2008. We reviewed the quality and quantity of primary studies available for each review, assessed the quality of the review process, and judged the strength of the evidence presented. There was no prior intention to perform any statistical analyses.
25 published systematic reviews were identified. Numbers of included trials ranged from none to 54. Within each review, included trials were heterogeneous with respect to patients, interventions, and outcomes, and the number of patients contributing to any single analysis was generally much lower than the total included in the review. A variety of tools were used to assess trial quality; seven reviews did not use this information to exclude low quality studies, weight analyses, or perform sensitivity analysis for effect of low quality. Authors indicated that there were frequently major problems with the primary studies, individually or in aggregate. Our judgment was that the reviewing process was generally good in these reviews, and that conclusions were limited by the number, size, quality and validity of the primary studies.
We judged the evidence about 23 of the 25 interventions to be weak. Two reviews had stronger evidence, but with limitations due to methodological heterogeneity or definition of outcomes. No review provided strong evidence of no effect.
Cochrane reviews in palliative care are well performed, but fail to provide good evidence for clinical practice because the primary studies are few in number, small, clinically heterogeneous, and of poor quality and external validity. They are useful in highlighting the weakness of the evidence base and problems in performing trials in palliative care.
In contemporary medical research, randomised controlled trials are seen as the gold standard for establishing treatment effects where it is ethical and practical to conduct them. In palliative care, randomised controlled trials may be impractical, unethical, or extremely difficult, with multiple methodological problems. The fact and nature of these issues with palliative care trials has been frequently commented upon [1–3]. Frequently encountered problems include recruitment and attrition, insufficient numbers of patients for any comparison, clinical heterogeneity between patients (condition palliated, comorbidity), heterogeneity in treatments (intervention, dose, duration), different outcomes reported, and use of non-standard scales. A palliative care Outcomes Working Group has recently made recommendations on outcomes they consider to be important in this context and how they might be sought in clinical trials .
Trials that have been done in palliative care are often small, diverse in nature and outcomes, and with high attrition rates, making meta-analysis, and even qualitative systematic review, impractical, unsatisfactory, or both. Moreover, some aspects of palliative care are difficult to capture, given the nature of palliative care as a person-centred approach, in which individual packages of care are often the norm .
With this background, the value of systematic reviews of randomized trials in palliative care might be questioned. One side of the argument would be that without a sufficiency of trials satisfying criteria of quality, validity, and size  systematic reviews are worthless. Another would see systematic reviews as a necessary first step to obtaining more evidence; despite their limitations, they at least tell us what we don't know, and may indicate how to improve.
This review set out to examine a subset of Cochrane reviews published under the auspices of the Pain, Palliative, and Supportive Care Review Group, to ascertain the number of successfully completed palliative care systematic reviews from this source over the last nine years, to assess their quality and the strength of the evidence presented to guide clinical practice.
A list of published reviews relating to palliative care and registered with the Cochrane Pain, Palliative and Supportive Care Group was obtained from the Review Group Coordinator as of December 2007. Copies of each review were obtained from the Cochrane Database of Systematic Reviews using the most recent upload, issue 1, 2008.
The following information was extracted from each review:
Number of studies included
Number of patients included
Trial design (randomized, observational)
Measures of quality and/or validity used
Whether exclusions due to poor quality were made, or a sensitivity analysis presented
Whether a pooled analysis was done
Original authors' conclusion on efficacy
Original authors' conclusion on strength of evidence
Original authors' implications for future research.
Two reviewers (GH, SD) independently carried out data extraction, using a standard form, and assessed the quality of each review using the Oxman & Guyatt Index of Scientific Quality . To determine the strength of the evidence presented, for each review we assessed the quality of the included studies, based on randomization and blinding since these characteristics are known to affect potential bias , and the number of patients available for any analysis, because small numbers are prone to random error [9, 10]. Any discrepancies were resolved by consensus.
There was no prior intention to perform any statistical analyses. What was intended was an evaluation of this set of systematic reviews in palliative care based on the quantity and quality of primary studies available, and the quality of the review process itself, in order to determine their utility for informing clinical practice.
Details of the 25 published systematic reviews [11–35] are in Additional file 1, together with the conclusions of the original authors. The first of these Cochrane reviews was published in 1999, and the most recent in 2007. Sixteen of the reviews concerned drug interventions for pain or other reasons, three involved radiotherapy, three complementary therapy, and one each for a mineral supplement, supportive care, and pleurodesis. Only five of the studies were published before 2003, and the rate of publication was five per year since 2004.
Number of trials and patients
The numbers of included trials ranged from none to 54. Thirteen had fewer than five controlled trials, and 16 had fewer than 10 trials. Three reviews had between 11 and 20 trials, and six more than 20 trials. Six reviews had information on fewer than 100 patients in total in controlled trials, fourteen had fewer than 500, while eight had between 1000 and 5000, and one more than 6,000 patients (see Additional file 1: Included reviews). Within each review, included trials were frequently heterogeneous, with differing interventions (drug, dose, route, technique) and reported outcomes, so that the number of patients contributing to any single analysis was nearly always much lower than the total number of patients included in the review.
All the reviews sought randomised controlled trials for inclusion. Five reviews [22, 25, 26, 30, 34] sought uncontrolled studies, but only two analysed these in the absence of randomised trials [22, 26]. Two reviews [28, 31] found no studies that met their inclusion criteria.
Types of patients
Eighteen reviews included trials involving only cancer patients. In most cases the type of cancer or site of the primary cancer was not restricted. One review included only AIDS patients , two included mixed diagnoses of cancer, lung disease, cardiac failure, cystic fibrosis, and elderly patients [20, 33], and one included patients with cancer or unspecified "terminal illness" .
Original authors' assessment of quality of included studies
All reviews with included studies assessed their quality, with the exception of Ballantyne  and Quigley , who found no randomised trials and included mainly retrospective studies, audits, or case reports, and uncontrolled prospective cohort studies. A number of scales were used. Most (18/23) used the Oxford Quality Score , and of these, three [16, 29, 30] additionally used the Oxford Pain Validity Scale , two [14, 21] used Rinck , one  used Detsky , and another  used both Juni  and Delphi . Shaw  graded trials according to criteria in the Cochrane Handbook , Feuer  according to Mann , and Ezzo et al  used their own set of five questions. Seventeen reviews also assessed allocation concealment using Cochrane criteria  in at least some of the included trials. Eight of the reviews that assessed trial quality did not use the information to exclude low quality studies, weight analyses, or perform sensitivity analysis for effect of low quality [11–15, 23–25]. For details of quality scores of included studies see Additional file 2 (Adequacy of included studies), and of the quality scoring tools used see Additional file 3 (Quality and validity tools).
The original authors themselves indicated that there were frequently major problems with the primary studies, individually or in aggregate. These included low numbers (either in total or available for pooled analysis) in 18 cases, the lack of useful outcomes in 10, methodological heterogeneity in eight, design problems in five, and clinical heterogeneity in two. For example, one review stated that we " ... need more larger studies with standardised outcomes of clinical relevance and clearer definitions of best supportive care" , while another stated that "Trials were too ...... short term for results to be meaningful" and that "Clinically relevant questions to address include which compounds are most beneficial, optimal dose and administration route, when prophylactic therapy ... should be started ..." .
Reviewers' assessment of quality of reviews
The methods used in these 25 reviews appeared to be sound. We attempted to use the Oxman & Guyatt Index of Scientific Quality , which asks questions about review methods. All the reviews had effective search strategies, and all looked at methodological quality in some way. However, deficiencies in the primary studies made judgment about assessment of validity and combining data close to impossible, as it was for the original authors. For instance, many reviews made no attempt to combine studies in a pooled analysis because of clinical heterogeneity and diverse interventions and outcomes, a decision that we felt to be correct.
We also felt that an overall Oxman & Guyatt score for these reviews was inappropriate because it attempts to measure flaws in the reviewing process. Our judgment was that the reviewing process was generally good in these reviews. Limited amounts and quality of data limited conclusions about efficacy or harm, most importantly lack of patient numbers, poor/inconsistent reporting, frequent use of non-standard outcome measures, and excluding outcomes which lack clinical relevance, for example patient satisfaction and long-term morbidity.
In our assessment of the strength of the evidence presented, we found that of the 25 reviews:
4 included randomized trials, with a mix of blind and open designs. Of these:
○ Wong  included mostly double blind studies, with 3600 patients, but using different drugs, doses, and routes of administration;
○ Nicholson  had 460 patients and 6/9 trials were double blind, but with different doses, and routes of methadone administration, and different comparators;
○ Dewey  had 60 patients and 4/5 trials were double blind, but they were insufficiently rigorous to be confident of any effect;
5 included randomised trials with only double blind design. Of these:
○ Roque  had only 325 patients in 4 trials using different drugs, and doses, in single or multiple dose schedules, and for different duration;
○ Jennings  had 292 patients in 3 trials, but with different drugs and doses.
Two reviews [20, 35] were considered to have the strongest evidence, although even for these reviews there were limitations with methodological heterogeneity or definition of outcomes. No review provided strong evidence of no effect. Even reviews with relatively large numbers of trials and patients could not provide strong evidence because of inappropriate comparison or trial design  or methodological heterogeneity .
This systematic review of systematic reviews in palliative care was to question the utility of systematic reviews for informing clinical practice in this area of medicine. It found that 25 reviews were published in the Cochrane Database of Systematic Reviews over nine years, a rate of about 2.7 per year overall, though almost double that rate occurred in the three years to 2007. Despite a respectable level of productivity from this prestigious source, 22/25 reviews could produce only weak evidence of the benefits of any intervention, and even of the two where the evidence was considered to be strong there were caveats.
The review processes themselves appeared adequate. Deficiencies lay in the primary studies, which were either missing or scant, or were characterized by heterogeneity in the methods, interventions, patients, and outcomes, which made an overall assessment of benefit or harm impossible. These deficiencies are similar to those identified previously [1–3, 5]. The authors of the reviews commonly commented on these deficiencies, and others. The biggest single issue was that of inadequate trials or inadequate patient numbers in high quality trials. In making even this point, the reviews and the reviewers make an important contribution.
It is likely that these observations are general to systematic reviews in palliative care. We limited our investigation to reviews from the Cochrane Database published through the auspices of the Palliative Care group, but we would expect such reviews to be no worse, and perhaps better, than non-Cochrane reviews [45, 46]. The restriction to Cochrane reviews should not limit any generalisability of these findings, especially as this reasonably sized body of reviews consistently makes the same, or very similar, points.
These findings are not a surprise. The dearth of good quality primary studies in the field of palliative care is widely accepted, and those trials that have been done are often known to have weaknesses [1–3, 5]. Together, these factors underline the limitations of the knowledge base upon which palliative care has to draw. Whether new guidance about outcomes to be measured in palliative care trials would make a difference  remains to be seen, but given the difficulties in design and conduct of palliative care trials, rapid change in the corpus of evidence is unlikely.
The challenge for palliative care is the lack of evidence that is available to support it and the inordinate difficulties in obtaining evidence, for example difficulties with recruitment and attrition in an ill and vulnerable population. This has led to calls for a different framework for examining evidence . Part of the problem is that nearly all randomised controlled trials examine single interventions, while in clinical practice that intervention will often form a small part of a much larger overall package of care . Randomised trials of overall packages of care with small or incremental differences between them are unlikely to be able to measure small improvements, and an evaluation of systematic reviews of palliative care services  highlighted similar problems to those of palliative care interventions. High patient losses also make interpretation of randomised trials difficult. It is important that palliative care research moves away from dependence on randomized trials, and explores alternative study designs to identify the most effective treatments and packages of care for its patients.
There may be alternatives. Nearly all the Cochrane reviews included only randomised trials, and the small number of reviews that did consider non-randomised studies found them to have many of the same problems as randomised trials, with an additional increased risk of bias. We know, from other areas of medicine, that high quality, well-formulated, and impeccably conducted large observational studies, can provide equivalent results to those obtained from randomised controlled trials [6, 49, 50]. To overcome the play of chance these good quality studies need to be large, and to minimise bias they need to be both prospective and inclusive (i.e. a whole population, or all patients attending a clinic in a defined time). Registry studies are studies based on information from registers that systematically record information from all individuals in a defined population. They can be entire populations, as in the death register in the UK, or all patients with a specific characteristic (eg twins) or condition (eg breast cancer) within a defined population. At least one large registry-based programme for continuous quality improvement aimed at cancer pain is ongoing in Italy . An extensive search for observational studies in palliative care has been undertaken, with the aim of identifying good quality observational studies and aspects of their design that make them reliable and useful (Hadley et al., manuscript in preparation). The proven limitation of controlled trials in palliative care may make registry studies a more acceptable option in future.
Cochrane reviews in palliative care are well performed, but fail to provide good evidence to guide clinical practice because the primary studies are few in number, small, clinically heterogeneous, and of poor quality and external validity. These reviews do, however, tell us how limited the evidence base is, and highlight common deficiencies in primary studies. There are well-documented problems with conducting valid randomised trials in this area, and it may be that for some questions more, and more clinically relevant, information can be obtained from other types of primary study, such as large registry studies.
Rinck GC, Bos van den GA, Kleijnen J, de Haes HJ, Schade E, Veenhof CH: Methodologic issues in effectiveness research on palliative cancer care: a systematic review. J Clin Oncol. 1997, 15: 1697-1707.
Grande GE, Todd CJ: Why are trials in palliative care so difficult?. Palliat Med. 2000, 14: 69-74.
Mazzocato C, Sweeney C, Bruera E: Clinical research in palliative care: choice of trial design. Palliat Med. 2001, 15: 261-264.
Mularski RA, Rosenfeld K, Joel Coons S, Dueck A, Cella D, Feuer DJ, Lipscomb J, Karpeh MS, Mosich T, Sloan JA, Krouse RS: Measuring Outcomes in Randomized Prospective Trials in Palliative Care. J Pain Symptom Manage. 2007, 34: S7-S19.
Higginson IJ: Evidence based palliative care. There is some evidence-and there needs to be more. BMJ. 1999, 319: 462-463.
Moore RA, McQuay HJ: Bandolier's little book of making sense of medical evidence. 2006, Oxford: Oxford University Press
Oxman AD, Guyatt GH: Validation of an index of the quality of review articles. J Clin Epidemiol. 1991, 44: 1271-1278.
Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, McQuay HJ: Assessing the quality of reports of randomized clinical trials: is blinding necessary?. Control Clin Trials. 1996, 17: 1-12.
Counsell CE, Clarke MJ, Slattery J, Sandercock PA: The miracle of DICE therapy for acute stroke: fact or fictional product of subgroup analysis?. BMJ. 1994, 309: 1677-1681.
Clarke M, Halsey J: DICE 2: a further investigation of the effects of chance in life, death and subgroup analyses. Int J Clin Pract. 2001, 55: 240-242.
Nicholson AB: Methadone for cancer pain. Cochrane Database Syst Rev. 2004, CD003971-2
Wiffen PJ, Edwards JE, Barden J, McQuay HJM: Oral morphine for cancer pain. Cochrane Database Syst Rev. 2003, 4: CD003868.
Dewey A, Baughan C, Dean T, Higgins B, Johnson I: Eicosapentaenoic acid (EPA, an omega-3 fatty acid from fish oils) for the treatment of cancer cachexia. Cochrane Database Syst Rev. 2007, 1: CD004597.
Miles CL, Fellowes D, Goodman ML, Wilkinson S: Laxatives for the management of constipation in palliative care patients. Cochrane Database Syst Rev. 2006, 4: CD003448.
Dennert G, Horneber M: Selenium for alleviating the side effects of chemotherapy, radiotherapy and surgery in cancer patients. Cochrane Database Syst Rev. 2006, 3: CD005037.
Martinez-Zapata MJ, Roqué M, Alonso-Coello P, Català E: Calcitonin for metastatic bone pain. Cochrane Database Syst Rev. 2006, 3: CD003223.
Tsao MN, Lloyd N, Wong R, Chow E, Rakovitch E, Laperriere N: Whole brain radiotherapy for the treatment of multiple brain metastases. Cochrane Database Syst Rev. 2006, 3: CD003869.
Ezzo JM, Richardson MA, Vickers A, Allen C, Dibble SL, Issell BF, Lao L, Pearl M, Ramirez G, Roscoe JA, Shen J, Shivnan JC, Streitberger K, Treish I, Zhang G: Acupuncture-point stimulation for chemotherapy-induced nausea or vomiting. Cochrane Database Syst Rev. 2006, 2: CD002285.
Zeppetella G, Ribeiro MDC: Opioids for the management of breakthrough (episodic) pain in cancer patients. Cochrane Database Syst Rev. 2006, 1: CD004311.
Berenstein EG, Ortiz Z: Megestrol acetate for the treatment of anorexia-cachexia syndrome. Cochrane Database Syst Rev. 2005, 2: CD004310.
Ahmed N, Ahmedzai SH, Vora V, Hillam S, Paz S: Supportive care for patients with gastrointestinal cancer. Cochrane Database Syst Rev. 2004, 3: CD003445.
Ballantyne JC, Carwood CM: Comparative efficacy of epidural, subarachnoid, and intracerebroventricular opioids in patients with pain due to cancer. Cochrane Database Syst Rev. 2005, CD005178-1
McNicol E, Strassels SA, Goudas L, Lau J, Carr DB: NSAIDS or paracetamol, alone or combined with opioids, for cancer pain. Cochrane Database Syst Rev. 2005, 2: CD005180.
Shaw P, Agarwal R: Pleurodesis for malignant pleural effusions. Cochrane Database Syst Rev. 2004, 1: CD002916.
Fellowes D, Barnes K, Wilkinson S: Aromatherapy and massage for symptom relief in patients with cancer. Cochrane Database Syst Rev. 2004, CD002287-2
Quigley C: Opioid switching to improve pain relief and drug tolerability. Cochrane Database Syst Rev. 2004, 3: CD004847.
Jackson KC, Lipman AG: Drug therapy for delirium in terminally ill patients. Cochrane Database Syst Rev. 2004, 2: CD004770.
Jackson KC, Lipman AG: Drug therapy for anxiety in palliative care. Cochrane Database Syst Rev. 2004, 1: CD004596.
Roqué M, Martinez-Zapata MJ, Alonso-Coello P, Català E, Garcia JL, Ferrandiz M: Radioisotopes for metastatic bone pain. Cochrane Database Syst Rev. 2003, 4: CD003347.
Bell R, Eccleston C, Kalso E: Ketamine as an adjuvant to opioids for cancer pain. Cochrane Database Syst Rev. 2003, 1: CD003351.
Hirst A, Sloan R: Benzodiazepines and related drugs for insomnia in palliative care. Cochrane Database Syst Rev. 2001, CD003346-4
Wong R, Wiffen PJ: Bisphosphonates for the relief of pain secondary to bone metastases. Cochrane Database Syst Rev. 2002, 2: CD002068.
Jennings AL, Davies AN, Higgins JPT, Broadley K: Opioids for the palliation of breathlessness in terminal illness. Cochrane Database Syst Rev. 2001, 3: CD002066.
Feuer DJ, Broadley KE: Corticosteroids for the resolution of malignant bowel obstruction in advanced gynaecological and gastrointestinal cancer. Cochrane Database Syst Rev. 2000, CD001219-2
McQuay HJ, Collins SL, Carroll D, Moore RA: Radiotherapy for the palliation of painful bone metastases. Cochrane Database Syst Rev. 2000, CD001793-2
Smith LA, Oldman AD, McQuay HJ, Moore RA: Teasing apart quality and validity in systematic reviews: an example from acupuncture trials in chronic neck and back pain. Pain. 2000, 86: 119-132.
Detsky AS, Naylor CD, O'Rourke K, McGeer AJ, L'Abbe KA: Incorporating variations in the quality of individual randomized trials into meta-analysis. J Clin Epidemiol. 1992, 45: 255-265.
Juni P, Altman DG, Eggert M: Assessing the methodological quality of randomised controlled trials. Systematic Reviews in Health Care. Edited by: Egger M, Smith GD, Altman DG. 2001, London: BMJ Publishing Group, 87-109. 2
Verhagen AP, de Vet HC, de Bie RA, Kessels AG, Boers M, Bouter LM, Knipschild PG: The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus. J Clin Epidemiol. 1998, 51: 1235-1241.
Mulrow CD, Oxman AD: Libraray of Examples. "Quality scale assessment". The Cochrane Library. Section 6:113. Edited by: Mulrow CD, Oxman AD. 1997, Oxford: Update Software
Mann T: Clinical Guidelines: using clinical guidelines to improve patient care within the HNS. 1996, Leeds: NHS Executive
Assessment of study quality. Cochrane handbook for systematic reviews of interventions 4.2.6 [updated September 2006]; Section 6:81. Edited by: Higgins JPT, Green S. 2006, Chichester UK: John Wiley & Sons Ltd
Carroll D, Tramèr M, McQuay H, Nye B, Moore A: Randomization is important in studies with pain outcomes: systematic review of transcutaneous electrical nerve stimulation in acute postoperative pain. Br J Anaesth. 1996, 77: 798-803.
Derry CJ, Derry S, McQuay HJ, Moore RA: Systematic review of systematic reviews of acupuncture published 1996–2005. Clin Med. 2006, 6: 381-386.
Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, Moher D: Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA. 1998, 280: 278-280.
Sheikh L, Johnston S, Thangaratinam S, Kilby MD, Khan KS: A review of the methodological features of systematic reviews in maternal medicine. BMC Med. 2007, 5: 10.
Aoun SM, Kristjanson LJ: Challenging the framework for evidence in palliative care research. Palliat Med. 2005, 19: 461-465.
Goodwin DM, Higginson IJ, Edwards AG, Finlay IG, Cook AM, Hood K, Douglas HR, Normand CE: An evaluation of systematic reviews of palliative care services. J Palliat Care. 2002, 18: 77-83.
K Benson, AJ Hartz: A comparison of observational studies and randomised controlled trials. NEJM. 2000, 342: 1878-1886.
Geborek P, Crnkic M, Petersson IF, Saxne T, South Swedish Arthritis Treatment Group: Etanercept, infliximab, and leflunomide in established rheumatoid arthritis: clinical experience using a structured follow up programme in southern Sweden. Ann Rheum Dis. 2002, 61: 793-798.
Apolone G, Mangano S, Compagnoni A, Negri E, Mosconi P, Mannino S, Villa M, Zuccaro P, Cancer Pain Outcome Research Study Group (CPOR SG): A multidisciplinary project to improve the quality of cancer pain management in Italy: background, methods, and preliminary results. J Ambul Care Manage. 2006, 29: 332-341.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-684X/7/13/prepub
We thank Andrew Moore and Henry McQuay for the initial concept and for helpful comments on the manuscript.
No specific funding was obtained for this work. Pain Research is supported in part by the Oxford Pain Research Trust. Authors have an absolute right to publish the results of their research, irrespective of any conclusions reached. No aspect of financial support influenced design and conduct of the study; collection, management, analysis, and interpretation of the data; or preparation, review, or approval of the manuscript.
BW and SD have received research support from charities, government and industry sources at various times, but no such support was received for this work. No author has any direct stock holding in any pharmaceutical company.
GH and SD were involved with planning the study, data extraction, analysis, and preparation of the manuscript. BW was involved with planning the study and preparation of the manuscript. All authors read and approved the final manuscript.
Gina Hadley and Sheena Derry contributed equally to this work.
Electronic supplementary material
About this article
Cite this article
Wee, B., Hadley, G. & Derry, S. How useful are systematic reviews for informing palliative care practice? Survey of 25 Cochrane systematic reviews. BMC Palliat Care 7, 13 (2008) doi:10.1186/1472-684X-7-13
- Palliative Care
- Primary Study
- Cochrane Review
- Cochrane Database
- Methodological Heterogeneity