State-wide implementation of patient-reported outcome measures (PROMs) in specialized outpatient palliative care teams (ELSAH): A mixed-methods evaluation and implications for their sustainable use

Abstract

Background

Such patient-reported outcome measures (PROMs) and patient-centered outcome measures as the Integrated Palliative Care Outcome Scale (IPOS), Phase of Illness, and IPOS Views on Care (IPOS VoC), facilitate patient-centered care and help improve quality. To ensure sustainability, implementation and usage should be adapted according to setting. When settings involve several distinct teams that differ in terms of views and working practices, it is more difficult to integrate outcome measures into daily care. The ELSAH study aimed to learn how health professionals working in specialized outpatient palliative care (SOPC) viewed the use of these outcome measures in daily care, and what they express is needed for successful sustainable, state-wide application.

Methods

We used a parallel mixed-methods design involving three focus groups (n = 14) and an online-survey based on normalization process theory (n = 76). Most participants were nurses and physicians from 19 SOPC-teams in Hesse, Germany. We used a triangulation protocol including convergence coding matrices to triangulate findings.

Results

The majority of health professionals were able to integrate the outcome measures into their working lives and said that it had become a normal part of their day-to-day work. To ensure their sustainable integration into daily care, the motivation and concerns of health professionals should be taken into consideration. Health professionals must clearly recognize how the measures help improve daily care and quality evaluation.

Conclusions

To implement the outcome measures in a number of teams, it will be necessary to take individual team characteristics into account, because they influence motivation and concerncs. Further, it will be necessary to offer opportunities for them to engage in peer support and share information with other teams. The sustainable use of outcome measures in SOPC will require continuous support within each team as well as across teams. When several distinct teams are working in the same setting, a cross-team coordination unit can help to coordinate their work efficiently.

Trial registration

German Clinical Trials Register DRKS-ID: DRKS00012421; www.germanctr.de/DRKS00012421

Background

In palliative care, patient-reported outcome measures (PROMs), patient-centered outcome measures and caregiver-reported outcome measures enable patients’ and families’ needs to be recognized and addressed [1]. Evidence that their use improves patient-relevant outcomes is growing [2]. Further, their use allows quality of care on a provider level to be improved through case-by-case evaluation and by monitoring care on a policy level [3].

In palliative care, a variety of outcome measures exist. Some focus on specific domains e.g. on physical, social, spiritual or cultural domains [4]. Some consider special target groups like patient, relatives, health professionals or the healthcare system [5]. The European Association for Palliative Care (EAPC) recommends to use validated outcome measures, that include patients and relatives, and cover all relevant aspects of care [1]. Initiatives like the Australian Palliative Care Outcomes Collaboration (PCOC) and the Outcome Assessment and Complexity Collaborative (OACC) developed comprehensive approaches to evaluate palliative care [6, 7].

The implementation of outcome measures is complex and needs to be adapted to the setting in which it is used [8]. The Consolidated Framework for Implementation Research (CFIR) implies that successful implementation must take characteristics of the intervention, the implementation process, the individuals, as well as the inner and outer setting into account [9]. Implementation in a broad setting involving different healthcare professionals presents an additional challenge because of differing backgrounds, views, management and working practices. Nevertheless, a degree of standardization and guidance helps promote fidelity and improve data quality. This condition must be fulfilled if benchmarking and national and international care comparisons are to be possible [1].

Health professionals’ views must be taken into consideration if usage is to be sustainable. In a similar study, a negative attitude of health professionals to quality measurement reduced the chance of successful implementation [10]. Previous research has also shown that participants may refuse to use PROMs if they feel not to have enough time and training for use [11, 12]. Lind et al. found a general feeling of fatigue followed changes to routines in various palliative-care settings [13]. Implementation research has shown that if staff do not feel that a task is appropriate to their job responsibilities, rejection and reduced fidelity may result [14]. Implementation frameworks can help organize tasks and improve implementation [15]. Normalization Process Theory (NPT) focuses on the work that needs to be done to implement, embed, and sustain practices in different contexts. For this purpose, the theory considers the four core constructs coherence (how health professionals make sense of a new practice), cognitive participation (relational work to redesign the work), collective action (to sustain use), and reflexive monitoring (how the health professionals appraise the impact on work) [16]. This theory is therefore particularly suitable for studying the normalization of change in the work of teams, and considering the perspectives of involved health professionals. Therefore, NPT has been widely used in research to evaluate and understand implementation processes [17, 18].

Specialized outpatient palliative care (SOPC) provides comprehensive care to patients with life-limiting diseases and complex needs in their familiar surroundings [19]. The aim of our ELSAH study (‘Evaluation of Specialized Outpatient Palliative Care by taking the example of Hesse’) was to implement outcome measures in all SOPC-teams in Hesse, Germany. Before our study patient records were based on the German National Hospice and Palliative Care Registry and, although including data on structure and process quality, symptoms, treatment and support needs, they barely considered outcomes from the perspective of patients and relatives [20]. To find out more about day-to-day care and include a quality assessment from their perspective, we added patient-reported and caregiver-reported outcome measures [21].

In a first step, we examined key features of successful care from the perspective of patients, relatives and health professionals. We found that ‘treatment of complex symptoms, comprehensive care and a sense of security, as well as a focus on the quality of relationships, respect for individuality and the facilitation of self-determination’ [22] are relevant topics that need to be covered. On this basis, we assembled a set of validated outcome measures, which is based on the Outcome Assessment and Complexity Collaborative (OACC) suite of measures [6]. The set of outcome measures includes the Integrated Palliative care Outcome Scale (IPOS) [23], Phase of Illness [24], and IPOS Views on Care (IPOS VoC) [25]. In the next step, we examined the feasibility, acceptability and appropriateness of these outcome measures in a sample of five SOPC-teams. We found that they should be used considerately in order to minimize the burden on patients and relatives, and ensure their administration was manageable. It was also important that participants understood that the measures were useful [26].

However, it is not yet clear how the sustainable implementation of these outcome measures can take place in a broad SOPC setting involving different healthcare professionals, backgrounds, views, management and working practices. Because the health professionals are the ones who use the outcome measures, and their attitudes influence sustainable use, their perspective is decisive. Therefore, we evaluated the state-wide use of these outcome measures in daily care from the perspective of health professionals in this study.

Methods

Aim

We aimed to understand how health professionals working in SOPC viewed the use of outcome measures in daily care, and to determine what they express is needed for successful sustainable, state-wide application. We assumed that the implementation and application of outcome measures in daily care is a complex, dynnamic, and intersubjective process [27]. Therefore, we chose to use normalization process theory (NPT) as an explanatory framework in order to investigate embedding into routine care and to gain an insight into participants' experience of implementation [16].

Design

We used a parallel mixed-method design. Althought we first conducted the qualitative focus groups and then a semi-structured online-survey, we analyzed them in parallel due to organisational reasons [28]. We considered mixed-methods appropriate and used an online survey to gather information on the views of professionals, and focus group discussions to understand participants’ views in more in depth [29]. We followed Mixed Methods Article Reporting Standards (MMARS) (supplemental material A) [30].

Setting

We gradually implemented the outcome measures in daily care from November 2018 to June 2019 via the electronic documentation systems (EDS) of 19 SOPC-teams caring for adults in Hesse, Germany. In a further three teams, implementation could only take place after data collection due to delayed software adaptation. Hesse is a federal state in Germany containing rural and urban areas and about 6.3 million inhabitants [31]. In total, 22 SOPC-teams provide state-wide care for adults. Teams mainly include nurses and physicians, but in some cases also involve psychologists and social workers. All SOPC-teams belong to the Professional Association of Specialized Palliative Homecare in Hesse, which collects standardized data in order to improve quality of care [32].

We conducted a training meeting with every team on their premises before implementation. Topics were reasons for implementation, the content and practical use of the outcome measures, and organizational integration into daily care routines. After the training meeting the SOPC-team members applied the outcome measures in every case. Table 1 describes the outcome measures used. For the self-report, the SOPC-team members included the items into the conversations. Because we found in a previous study that it needs tact to address sensitive topics, we did not arrange predefined time points for self-report, but made the default to assess every topic when the information was collectible [26]. If a self-report was not possible, e.g. due to deteriorating health, the health professionals asked relatives or completed proxy-reports themselves.

Table 1 Outcome measures used

Data collection

Focus groups

Two experienced qualitative researchers (HS, KK) conducted focus groups to investigate SOPC-team members’ experiences and suggested changes. We developed the focus group topic guide by including the NPT core constructs (supplemental material B). We brought members from different teams together in order to stimulate a discussion [34]. We sent an email invitation to our contact partner in every team and asked them to forward it to team members and to choose one to three team members, independent of profession, to participate in a focus group. We did not sample for specific characteristics, but included the people who had interest to participate (convenience sampling) [35]. Participants gave their written informed consent to participate, permitted a video recording to be made, and provided demographic data in a questionnaire. We wrote field notes on the researchers’ thoughts directly afterwards.

Online survey

We used lime-survey [36] to conduct an anonymous semi-standardized online survey, based on the NoMAD (Normalization MeAsure Development) instrument [37, 38]. Together with other authors, the authors of the NPT, May and Finch, developed the NoMAD tool, which aims to understand participants’ implementation experiences. This validated generic quantitative measure contains 43 items and has been used in numerous previous studies [39,40,41]. It includes statements related to the core constructs of NPT and asks participants to rate their agreement on a 5-point Likert scale (strongly agree to strongly disagree plus the option ‘not relevant to my role’ and the possibility to omit questions) [37].

In line with the developer’s recommendations, we adapted NoMAD to suit our research interest by adding both an introduction and questions on participants’ characteristics [38]. We further added a question on how relevant they considered each outcome measure and gave them the opportunity to add comments in free text (supplemental material C). We emailed a participation link to all team leaders, and asked them to forward it to all team members involved in using the outcome measures (convenience sampling).

Analysis

Focus groups

We transcribed and pseudonymized the audio recordings using MAXQDA [42]. May et al. presented several possibilities to analyze qualitative data in line with NPT [18]. We (HS, KK) used qualitative content analysis, and combined inductive and deductive coding using MAXQDA [43]. By setting the four core constructs of NPT (coherence, cognitive participation, collective action, reflexive monitoring) and their respective outcome measure as a-priori codes, we were able to link the content to the online-survey and identify issues associated with a specific measure. We discussed and added inductive codes and assigned them to the four core constructs of NPT after initial coding.

Online survey

As suggested by the authors of the NoMAD, we (HS, KK) used Microsoft Excel, version 2016, to analyze the online survey descriptively by looking at the response distribution and percentages [38]. We (HS, KK) used MAXQDA to analyze the free-text answers using qualitative content analysis [42]. We used the same coding tree, that we have elaborated for analysing the focus groups.

Mixed-Methods integration

We triangulated the findings of the focus groups and the survey on and interpretation level by using a triangulation protocol [44]. We created a convergence coding matrix for NPT core constructs and another convergence coding matrix for specific feedback on outcome measures. During the process, we examined the key quantitative and qualitative findings for convergences, dissonances, complementary information and silences, and developed an overarching conclusion for each component [44].

Deduction of topics requiring particular attention

In the analysis we identified topics, that reoccured in all NPT core constructs. We assumed that, from the participants' point of view, these issues need to be addressed if further use is to be successful. From this, we have deduced that these topics require particular attention for sustainable use of the outcome measures. Therefore, we present the deduction and the topics contents separately in this article.

Results

Study sample

We conducted three focus groups with n = 14 SOPC-team members. Five participants cancelled at short notice due to time constraints. Discussions took place in September and October 2019 in the rooms of an independent palliative association in Fulda and at the University of Frankfurt, and lasted about 105 min. The time between implementation and data collection was 3–10 months, depending on team.

The anonymous online-survey was conducted from November 20, to December 31, 2019. At this time the outcome measures had been used for six to twelve months. In the end, n = 76 complete data sets were available, and 104 hits were registered in total. Detailed results of the online-survey are presented in supplemental material C.

Participants’ characteristics are shown in Table 2.

Table 2 Characteristics of participants

In the following, we present the general appraisal and results according to NPT. Subsequently, we show key themes requiring particular attention, including specific findings on outcome measures.

Appraisal of use in daily care

In the overall appraisal, about 69% of participants in the online-survey said the use of the outcome measures currently belonged to their day-to-day work. Participants that disagreed generally believed this would not be the case in the future (supplemental material C).

We present our findings according to the four core NPT constructs in Table 3. For each subject in the online-survey, we present a synopsis of online-survey results and focus group findings, provide a statement on the agreement between findings, and draw a conclusion. In Table 3, we also present a summary for each core construct of NPT.

Table 3 Convergence coding matrix for NPT core constructs

Topics requiring particular attention

We found that four key topics recurred across all NPT constructs (Fig. 1). From the perspective of health professionals in SOPC these topics require particular attention if the revised documentation is to be used sustainably in daily care: 1) Daily care, 2) quality evaluation, 3) motivation and engagement, and 4) fears and concerns. We illustrate our findings using translations of pseudonymized quotations from the focus groups.

Fig. 1

NPT core constructs leading to four key themes

Daily care

Most participants knew the difference between the old and new documentation, but several participants (42%) did not see that the new documentation provided additional value in their practical work. Correspondingly, many participants (46%) did not feel the time for documentation was well spent.

The patient has always been taken into account to a great degree, but just not in the documentation. Now the documentation is written as if the patient really said all this, but everything is actually the same as before. (1738E, female nurse)

Most participants (71%) assumed, that everyone in their team could use the documentation in day-to-day care, but we found they had difficulties integrating it. Participants that felt they did not have sufficient resources at their disposal thought they had too little time for documentation.

Many participants (50%) thought the use of the outcome measures promoted cooperation with their fellow employees. While some mentioned the results in their team meetings, others did not look at them after entering the data. It became clear that not all participants knew how to integrate the results into their work, which may explain a lack of appreciation of their usefulness.

Quality evaluation

Many participants (53%) doubted whether quality could be evaluated through the use of documentation. This reflected a general concern whether outcome measures accurately reflected the actual care situation:

What I want to make comprehensible is that I can’t use a table to illustrate the way a person is, the emotions a person has, the process of dying in a person, or how a person deals with an illness. (1738E, female nurse)

Some participants struggled with the idea of an objective assessment. They criticized proxy assessments as having no validity because the assessment of health professionals can differ. They also criticized patient self-assessments on the grounds that patients’ conditions might influence their assessment.

But isn't it again a bit of a subjective judgment that varies from one colleague to another, when we go through the symptoms and I assess something? [...] So I look-, and of course ask about pain, but of course I assess what he tells me in terms of the different categories. My colleague might see things quite differently. How coherent is the picture then? 1736E, female nurse)

It became obvious that most participants (57%) could identify with the topics covered by the measures. However, many participants (43%) had no knowledge of reports or experiences of the usefulness of such measures, which may explain why they were sceptical about their usefulness.

Motivation and engagement

Only 41% of participants were keen to use the outcome measures in practice. Beyond the overall burden of documentation, the assumed benefit also influenced their engagement. Those that assumed care would benefit showed more interest in using it than those that did not.

Uncertainty in using the documentation in practice may also have had a negative impact on motivation. Most participants had dealt with the topics theoretically (86%) and felt they knew how the outcome measures should ideally be used (83%), but the discussions showed that some details were unclear. For example, they were particularly unsure how to address IPOS questions on psychosocial issues.

Yes, because how do I interpret the question, right? ‘At peace with myself’. Hmm. I have cancer, shit, I'm going to die. How can you be at peace with yourself ... That's just an expression covering one, two questions, or three, four (laughs), that's actually something everybody has a hard time with, all of us. (1742E, female nurse)

We found that sharing information across teams and team-related feedback helped motivate them. Most online-survey participants agreed that sufficient support was provided during implementation (63%), and at the time of the online-survey (60%), and they agreed that the management team provided sufficient support (72%). Nevertheless, participants in the focus groups wished for more continuous training and regular support from experts in the use of outcome measures.

Fears and concerns

Some participants (43%) expressed concern about a negative impact on care. They were concerned they would have less time for patients because they would have to invest time in completing the documentation. They also feared data might be misinterpreted, and that their work would be poorly evaluated, even though they were doing a good job in practice:

The Phase of Illness assessment, that's where the ‘stable’ bothers me. [...] But the thing is always: is he stable BECAUSE he has SOPC? [...] Stable actually means: does he still have an indication for SOPC? [...] And that’s when I ask myself: will this be used against us at some point? If we use that in the evaluation for the [health] insurers, for the funding agencies, will that be used against us? [...] Is that stable at some point the time they say ‘stable, stable, and you’re providing SOPC!’? (1745E, female physician)

Several participants (48%) denied that the use increased the quality of relationships with patients and relatives. Participants reckoned the quality of relationships depended more on personal interactions than outcome measurements. They also feared that the quality of relationships might deteriorate because patients might find some of the items upsetting:

But I find it very difficult to ask such questions because I’m then rather afraid that they might close themselves off completely. They are at home; we are already intruding on their intimacy, and so I think we should show a great deal of tact. (1744E, female nurse)

Specific feedback on the outcome measures

Participants regarded IPOS, and Phase of Illness symptom documentation as most helpful in their day-to-day care. They struggled with the psychosocial questions asked in IPOS (anxiety, family anxiety, depression, feeling at peace, share feelings). Although most participants considered the content to be relevant to care, several participants found it difficult to integrate the questions into a conversation, and to classify the answers on the scale. Participants said it was often impossible to interpret answers to such questions at the beginning of care because trust first had to be built up before such topics could be addressed. These issues could therefore not be raised when care was only provided for a short period.

Some participants criticized the wording of the questions. They also said it was difficult for staff members to make proxy assessments and almost impossible with cognitively impaired patients. Most participants appreciated the usefulness of IPOS VoC in evaluating care, but not in their practical work. Detailed feedback on the outcome measures is presented in Table 4.

Table 4 Convergence coding matrix regarding specific feedback on the outcome measures

Discussion

Main findings

Overall, participants said that they had accepted the use of the outcome measures in daily care and that it had become part of their day-to-day work. However, participants expressed that sustainable integration into daily care will require that special attention is drawn to their usefulness in daily care, in quality evaluations, in motivating and engagement and appreciating the concerns of health professionals.

Comparison of findings with those reported in the literature

Feedback from participants in our study was heterogeneous, suggesting that the effectiveness of the implementation varied. While implementation worked well in some teams, this was not the case in other teams, suggesting that it is more difficult to implement in different teams than in individual teams. Top-down implementation might reduce the sense of accountability and possibly even lead to resistance when compared to bottom-up implementation [15]. Although the implementation by our research team was carried out in consultation with the Professional Association of Specialized Palliative Homecare in Hesse and thus with all SOPC-teams, it can be assumed that individual health professionals did not feel they had been included in the decision-making process.

In addition to the initial implementation, we found that the way the measures are used by the teams in daily care influences sustainability. Other studies found that supportive leadership and peer support facilitated the implementation and encouraged its further use [45, 46]. Bradshaw et al. said that the benefits need to be demonstrated in order to increase motivation and engagement and to promote and sustain a collaborative effort, adding that measures may otherwise be dismissed as pointless [47]. Another study agreed that good training and guidance is required in the use of PROM in care [48]. It became clear that training before and during implementation is not enough, but that continuous support, which should be adapted to the needs of individual teams, should be provided to users.

Sustainable motivation requires that results are interpreted and fed back to health professionals [46]. Our participants distinguished between benefits in their practical work and benefits in terms of quality assurance, but for its sustainable use it is important to understand both aspects. Use in daily care, e.g. in team discussions, may clarify the direct benefits in practical work and for individual patients. Similar to our findings, another study showed that health professionals appreciated the opportunity to recognize unmet needs through the electronic use of PROMs, but were also concerned that it could disrupt care delivery processes, especially when intimate questions were asked of patients with whom the duration of contact had been only short [49]. Other research, however, showed that psychosocial support is also possible on first contact [50]. On specific outcome measures, further feedback from our participants was in line with the results of similar studies [47, 51]. This shows the need for ongoing training, particularly on how to integrate the psychosocial questions of IPOS into daily care. Peers with experience of using the measures could regularly address issues surrounding integration and other problems in the team. By ensuring they were accessible to colleagues as low-threshold contact persons, they could also help train new staff. Cross-team information sharing amongst peers could improve the promotion of long-term, sustainable use. Implementation in several teams thus also offers opportunities for mutual support.

To enhance a good quality of nursing documentation, Groot et al. recommended to use familiar terms [52]. Following this recommendation, we used the term documentation in conversations with SOPC-team members about the outcome measures. Looking back, we realized that the term has negative associations and often worries health professionals because they do not see documentation as a supportive component of care, but rather as something separate. In further implementations, therefore other terms like patient- and care-relevant outcomes should be used, in order to underline the relevance for care and the patients and relatives and to strenghten the motivation for use.

We found concerns about negative consequences to be fundamental but related more to the health care system than to the outcome measures. For example, although the participants of our study considered the content of the outcome measure ‘Phase of Illness’ important, they also saw it as a potential threat. They were afraid the health care system might force them to exclude patients from SOPC, if the measure ‘Phase of Illness’ was recorded as stable. This critical attitude may result from our participants having dealt with quality assessment for some time, as this may have given them the impression that control by a higher authority (the health system), as reflected in more and more assessments, is increasing. Krawczyk et al. stated that health professionals rather focus on the micro- and meso-level in daily care, so the macro perspective needs to be explained to them [53]. Although this might reduce scepticism, individual teams may not have the resources to do this. A community of several teams could concentrate resources on key events and address these issues together. Further, internal benchmarking of teams within their community could enable best practices and areas of improvement to be identified, without producing fears of control [54].

Strengths and limitations

The use of normalization process theory helped us focus on the relevant aspects of integration into daily care. The mixed-methods design offered the opportunity to gain a multi-perspective insight [38]. If we had used a sequential design with the online survey first, we could have considered the results in the focus groups, but this was not possible for organizational reasons. However, the online-survey enabled low-threshold participation, and anonymous participation may have made it easier for participants to make critical statements. The focus groups allowed a deeper understanding of the participants’ views.

For data protection purposes, we sent the invitations to participate to the team leaders and asked them to forward them to team members. Gatekeeping by the team leaders may therefore have influenced the results. Some participants could not participate because of their work obligations, or excessively long journeys to focus groups. In our focus group just two physiscians participated, but in total an adequate number of physicians could be recruited, so that the physician’s perspective could also be integrated. Furthermore, we did not include patients in this evaluation phase. However, it became apparent that team members were strongly patient-oriented and tried to call attention to their needs.

Implications

We identified certain factors that may help promote the sustainable integration of PROMs and caregiver reported outcome measures in daily care. In addition to findings relating to the use of specific outcome measures, we also present measures that collaborating teams should use collectively. These results are probably transferable to similar home-care settings that are responsible for caring for adults. The transfer of results to the care of children and adolescents, or to inpatient palliative care, may be limited due to the differing needs of such patients and the different structure of their daily care [55]. We could only derive implications for a sustainable application from the feedback of SOPC-team members after a short period of application. Further research should focus on applying and evaluating strategies for sustainability and integrate the perspectives of patients and relatives.

The use of PROM as a basis for funding and accreditation has raised fears among our participants. They feared that, for example, higher compensation could be paid if the outcome measures were rated well and deductions could be made if they were rated badly. This should be carried out with caution, as stakeholders have differing interests and there is no clear evidence for its appropriateness in this regard [53]. However, we recommend internal use for comparisons with other teams and for internal quality assurance. Further research is needed on how results should be fed back to health professionals.

It became obvious that not all participants felt that the use of the outcome measures had or should become part of their day-to-day work. Previous research has shown that the relationship between implementers and users influences the success of an implementation. It may be helpful for similar implementation projects to make peers responsible for implementation, as their expertise may be better accepted than that of external staff [56]. Furthermore, it became clear that the measures should only be implemented over the long term, as the process is a continuous one that requires constant support and further development, along with appropriate resources.

Conclusion

The sustainable integration of outcome measures into daily care will require that particular attention is paid to responding to the motivation and concerns of health professionals and making them aware of the value of the measures in daily care and quality evaluation. Combining the implementation of PROMs in a number of SOPC-teams is more complicated than implementation in a single team because teams’ specific characteristics must be taken into account and a comparable basis created. At the same time, combined implementation offers opportunities for mutual support and the pooling of resources. The provision of support by experienced peers within and across teams will enable concerns to be addressed and benefits in daily care to be explained on a low-threshold level. Centrally organized information sharing across teams is a suitable means of communicating the benefits of PROMs in quality evaluation, and, through the efficient use of resources, can help complex topics to be addressed. Both strategies can promote motivation and allay concerns. However, the task is ongoing and requires staff and time.