This article has Open Peer Review reports available.
Improving palliative care in selected settings in England using quality indicators: a realist evaluation
© The Author(s). 2016
Received: 29 September 2015
Accepted: 27 July 2016
Published: 2 August 2016
There is a gap between readily available evidence of best practice and its use in everyday palliative care. The IMPACT study evaluated the potential of facilitated use of Quality Indicators as tools to improve palliative care in different settings in England.
1) Modelling palliative care services and selecting a set of Quality Indicators to form the core of an intervention, 2) Case studies of intervention using the Quality Indicator set supported by an expert in service change in selected settings (general practice, community palliative care teams, care homes, hospital wards, in-patient hospices) with a before-and-after evaluation, and 3) realist evaluation of processes and outcomes across settings. Participants in each setting were supported to identify no more than three Quality Indicators to work on over an eight-month period in 2013/2014.
General practices could not be recruited to the study. Care homes were recruited but not retained. Hospital wards were recruited and retained, and using the Quality Indicator (QI) set achieved some of their desired changes. Hospices and community palliative care teams were able to use the QI set to achieve almost all their desired changes, and develop plans for quality improvements. Improvements included: increasing the utility of electronic medical records, writing a manual for end of life care, establishing working relationships with a hospice; standardising information transfer between settings, holding regular multi-disciplinary team meetings, exploration of family carers’ views and experiences; developing referral criteria, and improvement of information transfer at patient discharge to home or to hospital.
Realist evaluation suggested that: 1) uptake and use of QIs are determined by organisational orientation towards continuous improvement; 2) the perceived value of a QI package was not powerful enough for GPs and care homes to commit to or sustain involvement; 3) the QI set may have been to narrow in focus, or more specialist than generalist; and 4) the greater the settings’ ‘top-down’ engagement with this change project, the more problematic was its implementation.
Whilst use of QIs may facilitate improvements in specialist palliative care services, different QI sets may be needed for generalist care settings.
The science of clinical decision-making is well developed but the science of the delivery of care is relatively new and unfamiliar to most health professionals . As a consequence there are gaps between available evidence of best practice and its application in everyday care, across all disciplines within health care . This includes palliative care [3, 4], which aims to “improve the quality of life of patients and families who face life-threatening illness, by providing pain and symptom relief, spiritual and psychosocial support from diagnosis to the end of life and bereavement” .
The ageing of the population and the longer survival of people with life-threatening conditions have resulted in rising demand for palliative care from patients with multiple and complex problems [6, 7]. For example, in Europe within the next decade the incidence and prevalence of cancers will increase by about 20 % whilst the prevalence of dementia is expected to double before 2050 [8, 9]. Meeting the palliative care needs of this growing population is a big task, since the aim is to optimise the quality of life of people who have complex, incurable and life-threatening health problems by addressing their physical, emotional, psychosocial and spiritual needs.
Palliative care for people with cancer is relatively well developed, in terms of its conceptual framework and evidence base . The evidence base to guide practice with those dying with complex non-cancer conditions (for example, dementia) is less well developed, although now evolving . The organisation of palliative care has been described extensively [12–14], but it is not clear how best to improve its quality, especially for non-cancer conditions.
Quality Indicators (QIs) can be used to assess where care can be improved. Campbell et al, , following Danabedian , describe QIs as evidence-based, explicitly defined and measurable items that evaluate and describe the structure, processes and outcomes of health care, and that can indicate problems in achievement of good quality care. They can be used to assess and feed back to professionals their actual performance against benchmarks, as a starting point for quality improvement . QIs have been used effectively to assess and improve hospital care , primary care , and dementia care . Several international studies have also developed QIs to improve the structures and process needed for the delivery of good quality palliative cancer and dementia care [21–23]. However, these indicator sets have not yet been widely applied in everyday clinical practice. This study developed and implemented a QI package to improve the organisation of palliative care in five European countries . This paper describes the experiences of using a QI set to improve palliative care in different settings in one of the five countries, England.
The IMPACT project (IMplementation of Quality Indicators in PAlliative Care sTudy, 2011 to 2015) aimed to evaluate the potential of QIs as tools to improve palliative care for people with cancer or dementia in five European sites (England, Germany, Italy, Norway, and The Netherlands). We adopted Batalden and Davidoff’s definition of quality improvement  as “combined and unceasing efforts… to make changes that will lead to better patient outcomes (health), better system performance (care) and better professional development (learning)”.
Evaluating the effect of QIs required research methods that were able to respond to the many factors that shape palliative care in general and its organisational processes in particular. Controlling these aspects in an international study with strict deadlines made a randomised controlled trial (RCT) impractical [26, 27]. RCTs of services are usually focused on one setting and often in one country, while this study explored and initiated QIs in several settings in multiple European countries with varying health care systems. A before-and-after (pre-test, post-test) design was used instead.
Tasks of the IMPACT project
Modelling the organisation of palliative care in Europe 
Identifying barriers and facilitators to successful interventions in focus groups and individual interviews 28]
Identifying quality improvement strategies to improve the organisation of palliative care 
Development of a QI set to assess the provision of palliative care 
Pre-test: assessment of the organisation of palliative care, using QIs
Implementation of strategies to improve the organisation of palliative care, tailored to national and setting-specific barriers and facilitators 
Post-test: assessment of the organisation of palliative care, using QIs
Evaluation and dissemination
Process evaluation of the pilot
Development of toolkit and manual about how to implement changes in the organisation of palliative care
Other scientific output (papers, presentations, etc.)
This study invited health and care professionals to participate in a quality improvement process, and those who participated were asked to give their written consent. Medical or care records were assessed by the health care professionals in the settings themselves. No patient data were recorded or used in a way that would allow the identity of patients to be recognised. The research team had an agreed protocol for responding to any potentially poor practice that they encountered.
Theory and modelling
A qualitative description of palliative care models in Europe informed both the development of strategies to improve the organisation of palliative care using QIs. The QIs were derived from existing indicator sets and selected using a modified RAND Delphi procedure  involving a mix of clinicians and researchers active in palliative care .
The final set of 32 QIs used in the intervention study referred to: access to palliative care (specialist services, out of hours care, continuity of care), infrastructure of palliative care, assessment tools used by practitioners, personnel providing palliative care services (disciplines, teamwork, sharing information), documentation of clinical data, quality of care and its measurement, and education about palliative care.
Additional file 1 provides an overview of the quality indicators.
QI 1: Is a specialist palliative care team available 24 h/day, 7 days a week (24/7)?
QI 2: Is specialist palliative care advice available 24/7?
QI 3: Are bereaved relatives and/or professionals offered support during the bereavement process?
Full details of the development of the complete QI set are reported in van Riet Paap et al . The present paper focuses on the application of the QI set in different settings in England, and the evaluation of the project.
The strategies to improve the organisation of palliative care were identified in an integrative review  that drew on both empirical and theoretical literature on implementation strategies . Subsequently, interviews and focus group interviews with health and care professionals were used to adjust the strategies to national and setting-specific circumstances, in order to develop a toolkit of country- and setting-specific strategies to improve the organisation of palliative care.
We know that using feedback alone results in small to moderate effects on service performance . In addition to individualised feedback, this study also used other strategies tailored to influence setting-specific barriers and facilitators, to promote change. These included educational interventions (didactic, experiential, mentoring), process mapping, feedback on performance using audit data, multi-disciplinary case discussions and multi-component approaches . Many of these strategies have been tested before [1, 39], but not yet in the complex multidimensional field of palliative care. Quality Indicators and improvement strategies were developed for four types of settings (hospitals, care homes, hospices and community palliative care teams); no strategies were developed for general practice because of the failure to recruit interested practices. Settings which opted to participate did so on the understanding that the Quality Indicator set would be used as a formative rather than summative tool; that is, the purpose of using QIs was not to reach a judgement about the quality of the service provided, but to allow participants to identify, reflect on and work on weaknesses in their service, as they perceived them.
Setting and participants
In the pre-test audit phase the QI set was discussed with professionals working in each setting to profile the palliative care services provided in each setting, and subsequently to identify aspects in their service that required improvement. At the launch meeting (see Fig. 1) a project lead for the site was selected. The project lead arranged the pre-test audit of the site’s palliative care services using the QI set. A web-based data registration system (Lime survey: www.limesurvey.org/) was used to capture answers about the use of QIs, allowing early comparison of data between settings and providing rapid feedback to participants. With feedback from the research team based on their responses, professionals in each site were then encouraged to set improvement objectives formulated in a SMART way (Specific, Measurable, Acceptable, Realistic and Timely) . A consultant with expertise in service development helped each site to start and finish structured improvement projects . In the intervention phase each site’s professionals then chose specific strategies, tailored to their situation, to improve the organisation of palliative care. After the intervention period (post-test phase), the palliative care service was re-assessed with the use of the same QI set.
Evaluation and dissemination
A process evaluation was performed to study barriers and facilitators of the implementation process during the intervention period . This process evaluation included an activity report which both the health and care professionals involved in the study and the consultant and researcher working with the site were asked to complete. These reports captured details of what was going to plan, what was not, and any barriers and facilitators facing the teams. To derive a deeper understanding of the barriers mentioned in these activity reports, semi-structured interviews were conducted among the professionals who participated in the implementation process at the end of the implementation period. In addition, field notes were kept by all researchers and consultants after each encounter at the implementation site, whether face-to-face, by telephone or by email.
To get a better understanding of the changes in the organisation of palliative care and the experiences of the health care professionals involved in the intervention, the qualitative data were analysed thematically . A realist evaluation approach was used to synthesise the data from interviews, pre and post audits and field notes, in order to obtain a comprehensive picture of the contexts within which palliative care professionals worked, the mechanisms which promoted or impeded changes, and the consequent outcomes for each setting (without describing any outcome as primary or secondary, a priori). Realist evaluation works to the immediate priorities of empirical research, responds to the research brief, deals with the substantive issue and contributes to policy development, rather than aiming for methodological purity .
Recruitment and retention of settings in the English arm of the IMPACT study
General Practitioners (GPs)
Community Palliative Care teams (PC)
No. of invitations to participate
20 + (Palliative care champions, academic GPs, GP registrars)
Negotiations with care home chain (3 meetings) resulting in 3 Expressions of Interest
2 recruited through a specialist Dementias & neurodegenerative diseases research network (DeNDRoN)
2 recruited through a Clinical Research Network (CLRN)
2 recruited through CLRN
With managers not staff
With staff and team leader
With staff and management
With staff and management
• Workshops with staff on End of Life Care
• Standardise use of pain scales
• Optimising IT use
• Ring-fencing beds
See Table 3
See Table 3
Content of follow up
Both Care Homes dropped out
Both sites completed follow up QI profile and interviews
Both sites completed follow up QI profile and interviews
Both sites completed follow up QI profile and part in interviews
Continued quality improvement…
Care homes were recruited but not retained in the study. Three care homes were recruited from one commercial chain, with the support of their senior management, and two agreed to take part in the study. In one care home the manager spoke frankly about her difficulties in engaging the staff in palliative care and welcomed the opportunity to have staff workshops run by IMPACT consultants, but had moved to a different care home before these workshops could begin. In the second care home the manager explained at the launch meeting that it was the home’s policy to avoid providing palliative care to residents, preferring to transfer dying residents to other homes or to hospital. This manager agreed that wider staff use of pain scales would be an appropriate change for the care home to explore, but thereafter did not respond to further contacts with the research team.
Overall the participant sites performed relatively well against the Quality Indicator set. All had access to palliative care expertise outside normal office hours and at weekends, although not necessarily to all disciplines involved in multi-disciplinary palliative care teams. Regular meetings of multi-disciplinary teams occurred at all sites, as did training activities for team members. Medical records were not accessible across organisational boundaries, so those providing end of life care in a hospital setting would not be able to access records held in the nearby but organisationally separate hospice or community palliative care team. All sites provided bereavement support (albeit in different forms) to relatives, and most also offered bereavement support for staff. Opioids and co-analgesics could be obtained when needed. Communication about transfer of patients to other settings was systematised in all sites, using mixtures of written, electronic (email or fax) and verbal (telephone, face-to-face) methods. All sites had access to necessary equipment (drug administration pumps, anti-decubitus mattresses, stoma care supplies etc.) when needed. Single rooms for dying patients were available in all sites, except of course in community services supporting patients in their own homes. Use of standardised instruments (for example, for pain assessment, or assessment of emotional needs) varied considerably between sites. Similar variation was found in the systematic capture of relatives’ experiences of the services, and in use of palliative care guidelines.
Changes planned in the implementation study in each site
Changes planned and achieved✓
Care Home 1
More systematic use of pain scales
Care Home 2
Workshops with staff on End of Life Care
Hospital ward 1
Protect beds from other departments
Improve utility of electronic records ✓
Hospital ward 2
Writing an EOLC manual✓
Establish links with hospice✓
Establish rules for referral to other services (e.g. hospice) ✓
Community PC team 1
Implement validated pain assessment scales
Standardise transfer of information about discharge to GPs✓
Hold regular Multi-disciplinary team meetings✓
Community PC team 2
Documenting conversations with patients
Explorations of family carer experiences✓
Improve transfer of information to next setting✓
Implement validated pain assessment scores
Standardise transfer of information on discharge and assessment by home care teams✓
Develop criteria for referral of patients to other services (e.g. hospital) ✓
Documenting conversations with patients
Explorations of family carer experiences✓
Improve transfer of information on discharge to next setting✓
Realist evaluation: conclusions on the impact of the context and the mechanisms on achieving outcomes
Systemic organisational change in general practice (implementation of the Health & Social Care Act 2012)
GPs avoid additional tasks that increase work burden or are perceived outwith remit
No engagement with the project despite potential for general practice as a discipline and patient gain
Top-down decision to involve care homes, during period of care home chain re-organisation
Care home managers avoid additional imposed tasks
No lasting engagement
Top-down decision to involve hospital ward, and period of major staff turnover
Ward staff minimise involvement, momentum lost when key staff leave
Selection of indicators for change in order to minimise workload and maximise gain
Specialist palliative care schemes in hospices & communities
Competing demands in hospices & community teams, with funding challenges, periods of staff absence; but strong audit & research culture
Value of indicators appreciated, as supplement to existing audit & research culture
Changes made, effects seen, though extended implementation timescale required
The novel feature of the IMPACT project was the use of a QI set as part of a complex intervention to facilitate improvement in palliative care across a range of settings. The IMPACT QI set was comparable to the EUROPALL Quality Indicators  but more organisational and less clinical than those reviewed by Lorenz and colleagues .
The stepwise implementation used in this study enabled us to tailor the intervention at a setting and country level and so to translate the intervention into everyday clinical practice . It followed the Batalden and Davidoff model of change, beginning with the generalisable scientific evidence that was applied in known contexts to give measureable performance improvements . The realist evaluation focussed on developing explanatory models, using multiple data sources and methods, and sought to investigate contexts, mechanisms and outcomes in specific configurations .
Uptake and implementation of changes in organisation and practice identified using the QI set varied across settings, with maximal impact on in-patient hospices and community palliative care teams, and no impact in general practice or care homes. Although claims have been made that palliative care is the “territory”  of generalists the failure to engage general practices in the study is not surprising. The context (major recent changes in the British National Health Service across healthcare delviery and in general practice in particular, lack of funding for GPs to carry out palliative care) added to the known difficulties that GPs have in caring for dying patients , the challenge of maintaining palliative care skills , and the persistent taboo about discussing death .
Two objectives were not reached by any of the hospices and community paliative care teams: documentation of discussions with family carers; and routine use of validated pain assessment tools. Our impression is that the former had low priority compared with other communication tasks, such as transfering information if the patient moved settings. Pain management, of which formalised assessment tools are an important part, remains a problem in palliative care , and our participants’ inability to achieve their intended goal reflects this.
uptake and use of QIs was determined by a history of organisational orientation towards continuous improvement, described by Flottorp and colleagues as “capacity for organisational change” ;
the perceived value of the QI package offered was not powerful enough for GPs and care homes to commit to or sustain involvement, given their contexts;
the QI set may have been too specialist (in palliative care terms) in its focus and not useful enough for non-specialists (in care homes and in general practice) and;
the greater the settings’ ‘top-down’ engagement with this change project, the more problematic was its implementation.
These conclusions are consistent with Berwick’s view that any complex multi-component intervention is essentially a process of social change that is in turn influenced by leadership, changing environments, details of the implementation and organisational history . It is also consistent with the argument that mechanisms are usually hidden and sensitive to variations in context .
The findings of this study should be treated cautiously. Althought case studies yield important lessons about service changes in complex organisational environments, and are therefore very appropriate to palliative care , their generalisability may by limited by their close attention to contexts. Before-and-after studies tend to over-estimate effect sizes , although this probably applies more to quantitative outcomes than to organisational processes, and realist evaluation has methodological problems that may limit its potential . More involvement of generalists in the development of the QI set, with public and patient involvement too, might have resulted in a QI set that was more applicable to general practice and care homes.
Nevertheless, the findings of this study can contribute to the further evolution of palliative care, as well as indicating where further research and development efforts are needed. For example, findings from the IMPACT project informed the European Declaration on Palliative Care in 2014 (www.palliativecare2020.eu/declaration/). In our view a single, generic QI set is unlikely to serve the needs of different settings, and further work is needed to find levers for change in general practice and care homes. Even in hospices and community palliative care teams the implementation of a QI set seemed to depend on the pre-existing research and development culture of the organisation. A range of methods for engaging services with less developed cultures is needed , especially if palliative care expertise is to be made available in England to those with non-cancer conditions, as will be recommended by the forthcoming NICE guidance (http://www.nice.org.uk/guidance/indevelopment/gid-cgwave0694)
EUROPALL, European Palliative care study; IMPACT, IMplementation of Quality Indicators in PAlliative Care sTudy; INTERDEM, interventions in dementia research group; MRC, Medical Research Council; NRES, National Research Ethics Service; QI, quality indicator; RCT, Randomised controlled trial; SMART, specific, measurable, acceptable, realistic and timely.
The views and opinions expressed here are those of the authors and do not necessarily reflect those of the European Union, the NIHR, the NHS or the English Department of Health. We thank all those who took part in the IMPACT project in England for their support, and members of the wider project team, including Jane Wilcock, who assisted with the research and evaluated the process and likelihood of sustainability of the adopted measures.
This project was funded by the European Union’s Seventh Framework Programme FP7/2007-2013 under grant agreement n°258883. Additional support was given by the Clinical Research Networks of the UK National Institute for Health Research (NIHR).
Availability of data and material
Contact the corresponding author.
SI, SA & JM helped develop the project and gain funding; MV-D & YE co-ordinated the IMPACT project; ND, SI & PC contributed to data gathering and analysis at the English sites. All authors have contributed to drafting this paper. All authors read and approved the final manuscript.
The authors declare that there have no competing interests.
Consent for publication
Ethics approval and consent to participate
Ethics approval for the study, including obtaining participants’ informed consent, was granted by NRES Committee London (London Bridge) Reference 13/LO/0359.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Marshall M, Mountford J. Developing a science of improvement. J Royal Soc Med. 2013;106:45–50.View ArticleGoogle Scholar
- Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients’ care. Lancet. 2003;362:1225–30.View ArticlePubMedGoogle Scholar
- Sachs GA, Shega JW, Cox-Hayley D. Barriers to excellent end-of-life care for patients with dementia. J Gen Intern Med. 2004;19:1057–63.View ArticlePubMedPubMed CentralGoogle Scholar
- Vissers KC, van den Brand MW, Jacobs J, Groot M, Veldhoven C, Verhagen C, Hasselaar J, Engels Y. Palliative medicine update: a multidisciplinary approach. Pain Pract. 2013;13:576–88.View ArticlePubMedGoogle Scholar
- WHO definition Palliative Care [URL: http://www.who.int/cancer/palliative/en/] Accessed 29/07/15.
- Ouwens M, Wollersheim H, Hermens R, Hulscher M, Grol R. Integrated care programmes for chronically ill patients: a review of systematic reviews. Int J Qual Health Care. 2005;17:141–6.View ArticlePubMedGoogle Scholar
- Lynn J, Adamsom D. Living Well at the End of Life: Adapting Health Care to Serious Chronic Illness in Old Age. Rand Health White Paper WP137 edition. 2003.Google Scholar
- van den Beuken-van Everdingen MH, de Rijke JM, Kessels AG, Schouten HC, van Kleef M, Patijn J. High prevalence of pain in patients with cancer in a large population-based study in The Netherlands. Pain. 2007;132:312–20.View ArticlePubMedGoogle Scholar
- EuroCode. Dementia in Europe Yearbook 2008. Luxembourg: Alzheimer Europe; 2008.Google Scholar
- Ahmedzai S, Costa A, Blengini C, Bosch A, San-Ortiz J, Ventafridda V, et al. A new international framework for palliative care. Eur J Cancer. 2004;40:2192–200.View ArticlePubMedGoogle Scholar
- van der Steen J. Dying with dementia: what we know after more than a decade of research. J Alzheimer’s Disease. 2010;22:37–55.Google Scholar
- Radbruch L, Payne S. White paper on standards and norms for hospice and palliative care in Europe: part 1. Eur J Palliat Care. 2009;16:278–89.Google Scholar
- Radbruch L, Payne S. White paper on standards and norms for hospice and palliative care in Europe: part 2. Eur J Palliat Care. 2010;17:22–33.Google Scholar
- Centeno C, Clark D, Lynch T, Racafort J, Praill D, De Lima L, Greenwood A, Flores LA, Brasch S, Giordano A. Facts and indicators on palliative care development in 52 countries of the WHO European region: results of an EAPC Task Force. Palliat Med. 2007;21:463–71.View ArticlePubMedGoogle Scholar
- Campbell SM, Braspenning J, Hutchinson A, Marshall MN. Research methods used in developing and applying quality indicators in primary care. BMJ. 2003;326:816–9.View ArticlePubMedPubMed CentralGoogle Scholar
- Donabedian A. The quality of care: How can it be assessed? JAMA. 1988;121(11):1145–50.Google Scholar
- Grol R, Wensing M, Eccles MP. Improving patient care: the implementation of change in clinical practice. Edingburg: Elsevier; 2005.Google Scholar
- de Vos M, Graafmans W, Kooistra M, Meijboom B, Van Der Voort P, Westert G. Using quality indicators to improve hospital care: a review of the literature. Int J Qual Health Care. 2009;21:119–29.View ArticlePubMedGoogle Scholar
- Engels Y, Dautzenberg M, Campbell S, Broge B, Boffin N, Marshall M, Elwyn G, Vodopivec-Jamsek V, Gerlach FM, Samuelson M, Grol R. Testing a European set of indicators for the evaluation of the management of primary care practices. Fam Pract. 2006;23:137–47.View ArticlePubMedGoogle Scholar
- Draskovic I, Vernooij-Dassen M, Verhey F, Scheltens P, Rikkert MO. Development of quality indicators for memory clinics. Int J Geriatr Psychiatry. 2008;23:119–28.View ArticlePubMedGoogle Scholar
- Pasman HR, Brandt HE, Deliens L, Francke AL. Quality indicators for palliative care: a systematic review. J Pain Symptom Manage. 2009;38:145–56.View ArticlePubMedGoogle Scholar
- Claessen SJ, Francke AL, Belarbi HE, Pasman HR, van der Putten MJ, Deliens L. A new set of quality indicators for palliative care: process and results of the development trajectory. J Pain Symptom Manage. 2011;42:169–82.View ArticlePubMedGoogle Scholar
- Woitha K, Van Beek K, Ahmed N, Hasselaar J, Mollard JM, Colombet I, Radbruch L, Vissers K, Engels Y. Development of a set of process and structure indicators for palliative care: the Europall project. BMC Health Serv Res. 2012;12:381.View ArticlePubMedPubMed CentralGoogle Scholar
- van Riet PJ, Vernooij-Dassen M, Brouwer F, Meiland F, Iliffe S, Davies N, Leppert W, Jaspers B, Mariani E, Sommerbakk R, Vissers K, Engels Y, on behalf of the IMPACT team. Improving the organization of palliative care: identification of barriers and facilitators in five European countries. Implement Sci. 2014;9:130. doi:10.1186/s13012-014-0130-z.View ArticleGoogle Scholar
- Batalden PB, Davidoff F. What is ‘quality improvement’ and how can it transform healthcare? Qual Saf Health Care. 2007;16:2–3.View ArticlePubMedPubMed CentralGoogle Scholar
- Carlson MD, Morrison RS. User’s guide to research in palliative care: why is a new series needed? J Palliat Med. 2008;11:1258–61.View ArticlePubMedPubMed CentralGoogle Scholar
- McKee M, Britton A, Black N, McPherson K, Sanderson C, Bain C. Methods in health services research. Interpreting the evidence: choosing between randomised and non-randomised studies. BMJ. 1999;319:312–5.View ArticlePubMedPubMed CentralGoogle Scholar
- Campbell NC, Murray E, Darbyshire J, Emery J, Farmer A, Griffiths F, Guthrie B, Lester H, Wilson P, Kinmonth AL. Designing and evaluating complex interventions to improve health care. BMJ. 2007;334:455–9.View ArticlePubMedPubMed CentralGoogle Scholar
- Evans C, Stone K, Manthorpe J, Higginson I. MRC Guidance on Developing and Evaluating Complex Interventions: Application to Research on Palliative and End of Life Care, SSCR Methods Review 15, NIHR School for Social Care Research, London. 2014. [URL: http://www.sscr.nihr.ac.uk/PDF/MR/MR15.pdf] Accessed 29/07/15.
- Iliffe S, Davies N, Vernooij-Dassen M, van Riet PJ, Sommerbakk R, Mariani E, Jaspers B, Radbruch L, Manthorpe J, Maio L, Engels Y, for the IMPACT research team. Modelling the landscape of palliative care for people with dementia: a European mixed methods study. BMC Palliat Care. 2013;12(1):30.View ArticlePubMedPubMed CentralGoogle Scholar
- Davies N, Maio L, Vedavanam K, Manthorpe J, Vernooij-Dassen M, Iliffe S, for the IMPACT research team. Barriers to the provision of high quality palliative care for people with dementia in England: A qualitative study. Health Soc Care Community. 2014;22(4):386–94.View ArticlePubMedGoogle Scholar
- van Riet PJ, Engels Y, Iliffe S, Radbruch L, Kaasa S, Chattat R, Vissers K, Vernooij-Dassen M. Improving the organisation of palliative care by implementing quality indicators and national and setting-specific interventions: study protocol of the IMPACT project. Prog Palliat Care. 2014;22(4):201–5.View ArticleGoogle Scholar
- van Riet PJ, Vernooij-Dassen M, Dröes RM, Radbruch L, Vissers K, Engels Y, IMPACT research team. Consensus on quality indicators to assess the organisation of palliative cancer and dementia care applicable across national healthcare systems and selected by international experts. BMC Health Serv Res. 2014;14:396.View ArticleGoogle Scholar
- Davies N, Maio L, Van-Riet-Paap J, Mariani E, Jaspers B, Sommerbakk R, Gramatico D, Manthorpe J, Ahmedzai S, Vernooij-Dassen M, Iliffe S, for the IMPACT research team. Quality Palliative Care across Europe for Cancer and Dementia: some common challenges. Aging Ment Health. 2014;18(4):400–10.View ArticlePubMedGoogle Scholar
- Fitch K, Bernstein SJ, Aguilar MD, Burnand B, LaCalle JR, Lázaro P, van het Loo M, McDonnell J, Vader JP, Kahan JP. The RAND/UCLA Appropriateness Method User’s Manual. Santa Monica: RAND; 2001.Google Scholar
- Whittemore R, Knafl K. The integrative review: updated methodology. J Adv Nurs. 2005;52:546–53.View ArticlePubMedGoogle Scholar
- van Riet PJ, Vernooij-Dassen M, Sommerbakk R, Moyle W, Hjermstad MJ, Leppert W, Vissers K, Engels Y, IMPACT research team. Implementation of improvement strategies in palliative care: an integrative review. Implement Sci. 2015;10(1):103.View ArticleGoogle Scholar
- Jamtvedt G, Young JM, Kristoffersen DT, O'Brien MA, Oxman AD. Audit and feedback: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2006;2:CD000259.Google Scholar
- Grol R. Improving the quality of medical care: building bridges among professional pride, payer profit, and patient satisfaction. JAMA. 2001;286:2578–85.View ArticlePubMedGoogle Scholar
- Doran GT. There’s a S.M.A.R.T. way to write management’s goals and objectives. Manage Rev (AMA FORUM). 1981;70(11):35–6.Google Scholar
- Engels Y, van den Hombergh P, Mokkink H, van den Hoogen H, van den Bosch W, Grol R. The effects of a team-based continuous quality improvement intervention on the management of primary care: a randomised controlled trial. Br J Gen Pract. 2006;56:781–7.PubMedPubMed CentralGoogle Scholar
- Hulscher ME, Laurant MG, Grol RP. Process evaluation on quality improvement interventions. Qual Saf Health Care. 2003;12:40–6.View ArticlePubMedPubMed CentralGoogle Scholar
- Lewin S, Glenton C, Oxman AD. Use of qualitative methods alongside randomised controlled trials of complex healthcare interventions: methodological study. BMJ. 2009;339:b3496.View ArticlePubMedPubMed CentralGoogle Scholar
- Pawson R, Manzano-Santaella A. A “realist diagnostic workshop”. Evaluation. 2012;18(2):176–91.View ArticleGoogle Scholar
- Woitha K, van Beek K, Ahmed N, Jaspers B, Mollard JM, Ahmedzai S, Hasselaar J, Menten J, Vissers K, Engels Y. Validation of quality indicators for the organisation of palliative care: a modified RAND Delphi study in seven European countries (the Europall project). Palliat Med. 2013;0(0):1–9.Google Scholar
- Lorenz KA, Rosenfeld K, Wenger N. Quality indicators for palliative and end-of-life care in vulnerable elders. JAGS. 2007;55:S18–36.View ArticleGoogle Scholar
- Fordham S, Dowrick C, May C. Palliative medicine: is it really specialist territory? J Royal Soc Med. 1998;91:568–72.Google Scholar
- Mitchell GK. How well do general practitioners deliver palliative care? A Syst Rev Palliat Med. 2002;16:457–64.View ArticleGoogle Scholar
- Shipman C, Gysels M, White P, Worth A, Murray SA, Shepherd J, et al. Improving generalist end of life care: national consultation with practitioners, commissioners, academics and service user groups. BMJ. 2008;537:e1720.View ArticleGoogle Scholar
- Wise J. Dying remains a taboo subject for patients and GPs, finds survey. BMJ. 2012;344:e3356.View ArticlePubMedGoogle Scholar
- Payne S, Leget C, Peruselli C, Radbruch L. Quality indicators for palliative care: debates and dilemmas. Palliat Med. 2012;26(5):679–80.View ArticlePubMedGoogle Scholar
- Flottorp SA, Oxman AD, Krause J, Musila NR, Wensing M, Godycki-Cwirko, Baker R, Eccles MP. A checklist for identifying determinants of practice: a systematic review and synthesis of frameworks and taxonomies of factors that prevent or enable improvements in healthcare professional practice. Implement Sci. 2013;8:35.View ArticlePubMedPubMed CentralGoogle Scholar
- Berwick DM. The science of improvement. JAMA. 2008;299(10):1182–4.View ArticlePubMedGoogle Scholar
- Astbury B, Leeuw FL. Unpacking Black Boxes: mechanisms and theory building in Evaluation. Am J Evaluation. 2010;31(3):363–81.View ArticleGoogle Scholar
- Walshe CE, Caress AL, Todd CJ. Case studies: a research strategy appropriate for palliative care? Palliat Med. 2004;18:677–84.View ArticlePubMedGoogle Scholar
- Cabana MD, Wachter RM. The hazards of drawing conclusions from before-and-after studies of system level interventions. Arch Pediatr Adolesc Med. 2011;165(5):472–3.View ArticlePubMedGoogle Scholar
- Marchal B, van Belle S, van Olmen J, Hoeree T, Kegels G. Is realist evaluation keeping its promise? A review of published empirical studies in the field of health systems research. Evaluation. 2012;18:192.View ArticleGoogle Scholar
- Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, Procter EK, Kirchner JE. A refining compilation of implementation strategies: results from the Expert recommendations for Implementing Change (ERIC) project. Implement Sci. 2015;10:21.View ArticlePubMedPubMed CentralGoogle Scholar