Interrater agreement of multi-professional case review as reference standard for specialist palliative care need: a mixed-methods study
BMC Palliative Care volume 22, Article number: 181 (2023)
A wide variety of screening tools for the need for specialist palliative care (SPC) have been proposed for the use in oncology. However, as there is no established reference standard for SPC need to compare their results with, their sensitivity and specificity have not yet been determined. The aim of the study was to explore whether SPC need assessment by means of multi-professional case review has sufficient interrater agreement to be employed as a reference standard.
Comprehensive case descriptions were prepared for 20 inpatients with advanced oncologic disease at the University Hospital Freiburg (Germany). All cases were presented to the palliative care teams of three different hospitals in independent, multi-professional case review sessions. The teams assessed whether patients had support needs in nine categories and subsequently concluded SPC need (yes / no). Interrater agreement regarding SPC need was determined by calculating Fleiss’ Kappa.
In 17 out of 20 cases the three teams agreed regarding their appraisal of SPC need (substantial interrater agreement: Fleiss’ Kappa κ = 0.80 (95% CI: 0.55–1.0; p < 0.001)). The number of support needs was significantly lower for patients who all teams agreed had no SPC need than for those with agreed SPC need.
The proposed expert case review process shows sufficient reliability to be used as a reference standard. Key elements of the case review process (e.g. clear definition of SPC need, standardized review of the patients’ support needs) and possible modifications to simplify the process are discussed.
German Clinical Trials Register, DRKS00021686, registered 17.12.2020.
The issue of identifying patients in need of specialist palliative care (SPC) in oncology has occupied experts for more than a decade [1,2,3], and the widespread introduction of screening for SPC need is currently being sought [4, 5]. Accordingly, a wide variety of screening tools have been proposed: Some are generic [6, 7], while others are indication- [1, 3, 8, 9] or entity-specific [10,11,12,13]. The majority of the tools require assessment of criteria by staff [1, 3, 8, 9]; some authors suggest the use of patient-reported outcome measures [14, 15]. The proposed screening tools include a wide variety of criteria: disease-, treatment- and care-related aspects, as well as patient and family needs [2, 3, 8, 9, 16]. Although all these tools have very similar objectives, the suggested criteria for determining SPC need differ considerably among them.
In practice, the question arises as to which of the proposed screening tools should be employed. Which assessment best identifies patients with SPC need (sensitivity) and at the same time sorts out those who do not need SPC as reliably as possible (specificity)? Determining sensitivity and specificity requires a reference standard against which the results of the screening tool can be compared. A reference standard is defined as the best available method for determining the presence or absence of the target condition; it differs from a gold standard which is error-free . To our knowledge, no validated reference standard has yet been suggested for determining the need for SPC in oncology patients [16, 18]. In its absence, individual screening instruments have been shown to correlate with various parameters, such as SPC referral [9, 19], mortality / prognosis [2, 6, 16, 19, 20] and patient need questionnaires . But do these parameters accurately and reliably reflect SPC need?
SPC referral and reception are closely related to SPC need. Unfortunately, in current practice, it cannot be assumed that patients reliably receive SPC when they need it. Oncologists’ views, awareness and knowledge of SPC [21,22,23], resources and patient wishes, for example, can all influence SPC referral , making it a very imprecise measure of actual SPC need.
Mortality and prognosis (e.g. surprise question) can be easily measured . They are employed in validation studies of screening tools for SPC need based on the concept of early integration of palliative care (PC) [2, 15]. Early integration is typically defined as referral to palliative care within 2–3 months of diagnosis of advanced disease in cancer patients, as symptom burden increases on average as death approaches [26,27,28,29]. Due to limited resources in SPC and data suggesting that SPC mainly benefits patients with pre-identified support needs but not those without support needs , the concept of timely integration of SPC was introduced. Timely integration shifts the focus away from disease progression and aims “to identify patients with high support care needs and to refer these individuals to specialist palliative care in a timely manner” . If the aim is to identify patients for timely integration, prognosis and mortality surely correlate with SPC need but are not a suitable sole reference standard.
Patient reported outcome measures like the Integrated Palliative Outcome Scale (IPOS)  or Edmonton Symptom Assessment System (ESAS)  cover a broad and relevant range of patient needs and thus reflect the idea of needs-based, timely integration well. IPOS has been employed for validation of the screening of criteria proposed by the National Comprehensive Cancer Network (NCCN) . Conversely, both IPOS and ESAS have been proposed as screening tools for SPC need, including initial suggestions for cut-offs [14, 15]. However, to our knowledge, it has never been confirmed that the questionnaires themselves reflect SPC need, and the proposed cut-offs have also not been sufficiently validated.
When there is no gold standard, clinical expertise is frequently used as a reference standard . We are aware of two recently published studies that employed clinical expertise as the reference standard for SPC need assessment: Teike Lüthi et al. (2021) asked a PC team of physician and nurse to determine SPC need , Effendy et al. (2022) used independent assessments made by the treating oncology physician and nurse . Neither of the studies report interrater agreement of the reference standard.
The reported study is a pre-study to validate the reference standard in the context of the ScreeningPall study, which aims to develop easy-to-use screening criteria for SPC need (study protocol see ).
The aim of this pre-study was to explore whether SPC need assessment by means of multi-professional case review has sufficient interrater agreement to be employed as a reference standard. Additionally, we assessed the challenges in its use as a reference standard for SPC need.
The pre-study was designed as a mixed-methods study, data were collected between 08/2021 and 11/2022. It combines a reliability study, in which three PC teams independently assessed the same 20 patients with metastatic or locally advanced incurable cancer in structured multi-professional case reviews regarding SPC need, with a qualitative analysis of case review transcripts to identify challenges.
Definition of SPC need
In our study, we employ a definition of SPC need that is aligned with the requirements of the German health system. The question in case review is (translated from German): ‘Due to the needs of the patient and the relatives, is there currently a challenging situation with complex symptoms (physical, psychological, social or spiritual) that requires specialist palliative care? Specialist palliative care is characterized by practitioners with specific palliative care qualifications and experience, a multi-professional team approach and 24 h availability’ (based on the German S3-Guideline ). The teams were asked to focus on the current presence of SPC need and to assess it independently of the service that would carry out the SPC (e.g. consultation service, outpatient service). There also was a lower-level option of an ‘advisory session to inform the patient about possibilities and accessibility of specialised palliative care’, which could be chosen e.g. in cases of possible future need. It did not count as ‘SPC need’.
The pre-study was conducted at the University Medical Center Freiburg in Germany, a tertiary care centre with a Comprehensive Cancer Center. Patients were recruited in inpatient radiotherapy. The two cooperating PC teams from Erlangen and Cologne are also located at large university hospitals with Comprehensive Cancer Centers in Germany.
Inclusion criteria for patients were metastatic or locally advanced, incurable cancer with low probability of long-term control of the disease (estimated survival < 2 years) as assessed by the treating physician, ≥ 18 years of age and informed consent by patient or proxy. Patients with malignant haematological diseases as their main oncological diagnosis were excluded. We aimed to include 20 case descriptions for case reviews with approximately equal numbers of patients with and without SPC need for determination of interrater agreement. The sample size was based on a pragmatic decision, 20 cases was the maximum number for which the effort was still considered feasible for the external teams. Patient recruitment was to be terminated when 10 patients had been assessed as having SPC need by the Freiburg team. Due to recruitment of a higher number of patients without SPC need than with SPC need, we selected patients without SPC need with the aim of obtaining a heterogeneous sample regarding tumour entity, age and gender.
Case descriptions were based on information from routine documentation and standardized medical history taking by a physician and / or nurse from the PC team during a bedside visit (for details see Additional file 1).
Multi-professional case review
Each case was discussed by the three PC teams from Freiburg (internal team), Erlangen and Cologne university hospitals (external teams), without them knowing the results from the other PC teams. PC teams consisted of at least one physician, one nurse, one social worker or case manager, and one psychologist or pastoral worker (four different professional groups). An overview of the structured process of the multi-professional case review is presented in Fig. 1. The case reviews were digitally recorded and transcribed verbatim; the outcome and observations on the process were documented in writing on site (for further information see Additional file 1).
Statistical and qualitative analysis
Fleiss' Kappa was calculated to test the interrater agreement on SPC need among the three teams , employing IBM SPSS 28 Software . Descriptive statistics and the median test (alpha = 5%, two-sided) were used for an exploratory analysis of differences in support needs profiles among the three case groups ‘agreement SPC need’, ‘agreement no SPC need’ and ‘no agreement regarding SPC need’. There were no missing values.
For the cases in which the three teams did not agree regarding SPC need, a qualitative analysis  was conducted to identify possible factors that caused the differing conclusions. A team of four (social worker, social scientist, psychologist, nurse/medical documentarist; two members of the study team, two not involved) first read the transcripts line by line individually and annotated them, taking into account pragmatic, syntactic and semantic aspects. The team then developed a code system for possible influencing factors. This consisted of the nine predefined support needs categories and the SPC needs category (see Additional file 1), as well as the possible influencing factors identified in the team discussion: missing/additional information in the presentation of the patient cases, interruptions and interactions in the case presentation, and the roles of case presenter and facilitators (local senior physicians). Subsequently, the transcripts were coded, i.e. the texts were examined sequentially and the text passages were assigned to categories and discussed in the team. The results of the team discussions were recorded in case summaries and these were finally compared in a cross-evaluation with regard to the similarities and differences of the influencing factors.
History taking and case review took place for 32 patients before we reached 10 patients for whom the Freiburg team had concluded SPC need (criterion for recruitment termination). Two cases were excluded due to problems in medical history taking (e.g. focus on emotional support instead of medical history taking). Of the 30 remaining cases 20 were selected (for characteristics see Table 1): Of the 30 cases nine were assessed as having SPC need, 21 as having no SPC need by the internal team. All patients with SPC need were included with the aim of a well-balanced ratio of cases with and without SPC need. Of the 21 patients assessed as having no SPC need by the interal team, two were included due to special interest after controversial discussion in the internal team and nine based on the pre-defined criterion of heterogeneity regarding tumour entity, gender and age. There were no missing data on the relevant variables for the selected cases (for characteristics of excluded/non-selected cases, see Additional file 2).
Between four and nine professionals took part in case review sessions, with variations among teams. The composition of the Freiburg team changed weekly according to the duty rosters; in the external teams, there were two block appointments for 10 case reviews each and team compositions were consistent. In all case reviews, at least four different professional groups were represented.
Interrater agreement and assessment of SPC need in the three PC teams
In 17 of the 20 cases, there was agreement among the three PC teams on whether the patients currently had a need for SPC. This corresponds to an interrater agreement of Fleiss’ Kappa κ = 0.80 (95% CI: 0.55–1.0; p < 0.001), which indicates substantial agreement . The internal team identified SPC need in nine cases, the two external teams in ten and twelve. The agreement for individual cases is shown in Table 2.
Support needs profiles and SPC need
Table 2 shows the support needs profiles. Explorative comparison of the needs profiles showed that patients for whom SPC need was identified by all three PC teams had a significantly higher number of support needs in their needs profiles than patients for whom SPC need was consistently not identified (see Table 3). In the three cases without agreement, the median number of support needs was higher than that of patients without SPC need and lower than that of those with SPC need (due to the small sample size, the median test was not informative).
Qualitative analysis of cases with no agreement on SPC need
Relevant different conclusions of the teams occurred in situations involving (Case A) the ambiguous patient statement “I mainly fight for my sons” by a widow with a currently effective therapy regime (see Fig. 2), (Case B) divergent assessment of whether the family and professional support system in place is capable of handling the situation of a man with a learning disability who cannot return home to live by himself and (Case C) anticipation of possible SPC need in the near future in a man currently with no relevant physical needs but a high need for psycho-spiritual and social work support (Cases B and C in Additional file 3). In Case A, a procedural aspect—a slip of the tongue during the case presentation—might have led to different levels of attention being paid to specific information. Additional file 4 summarises further observations on the case review procedure.
We aimed to explore the use of the clinical expertise of multi-professional teams as a reference standard for SPC need. The substantial agreement among the three teams indicates reliability ; however, the confidence interval is still large due to the rather small sample. Further studies might contribute to a more precise estimation of interrater agreement.
The number of needs in the needs profile reflects the concept of timely integration of palliative care , where an increasing number of support needs in the course of the illness is associated with the timing of SPC integration. While cases with a high or low number of support needs are easy to agree upon regarding SPC need, the PC teams disagreed about three cases. Examples of case characteristics that led to patients being put into this group are unclear coping skills of the patient and/or support system, ambiguous statements of the patient and possible needs in the near future. For the reference standard, these are sources of error which are difficult to avoid.
Regarding the processes of SPC assessment, the following features should be considered for further development and use as a reference standard:
How should “SPC need” be defined in studies? We assumed that a definition of SPC need that is clear and as close to everyday care as possible will be easier to assess and most useful for the subsequent validation of screening tools in our context. Therefore, our definition reflects what we would like to identify: patients who need the expertise and resources of SPC due to high support needs .
What information should the reference standard for SPC need be based on? Patients’ SPC need is determined by their current support needs, and whether they and their support system can cope . In their study, Teike Lüthi et al. based their assessment on computer-based patient records alone or on patient records combined with palliative care liaison rounds . The appropriateness of that approach depends on local completeness and the timelines of information in the records. In our study, medical history taking was necessary, as patient records did not contain sufficient information e.g. on the coping of patients and the support system.
Who should make the assessment of the reference standard for SPC need? Considering studies that show the impact of professionals' knowledge, experience and sensitisation in palliative care on the assessment of SPC need [21,22,23], we believe that experienced palliative care professionals should carry out the assessment as opposed to treating professionals. The approach of Teike Lüthi et al. involving assessment by dyads  instead of multi-professional teams is promising, especially combined with a holistic needs profile. As physicians and nurses in palliative care are usually attentive and sensitive to psycho-spiritual needs [41, 42], the participation of psychologists and social workers might not be necessary.
Strengths of the study
To our knowledge our study is the first to show sufficient interrater agreement of expert opinion on SPC need and explore the prerequisites of the assessment processes. The suggested approach is not a gold standard but a reference standard, as the assessment cannot be free of error . However, just like the surprise question for determining prognosis, it might be the best available option for the complex assessment of SPC need. The suggested procedure is transferable to other settings and healthcare systems, applicable and likely reliable.
Limitations of the study
Our results are based on data from only 20 patients. A larger patient sample would allow a more precise measurement of interrater agreement and further exploration of patient case characteristics and process features that might cause difficulties in the assessment of SPC need.
Case reviews were not strictly blinded as the senior physician at Freiburg presented the cases to the external expert teams. The background to that decision was a pre-test with Freiburg staff not involved in the study, in which the written case descriptions were prepared in advance (see Additional file 1). It became apparent that the high effort of working through the 20 case descriptions resulted in selective attention to certain information based on personal / professional priorities and in a game of predicting the team's answers in terms of SPC needs (and thus pre-judgment). Consequently, we preferred oral case presentation by a Freiburg team member, allowing for the exact same procedure as in Freiburg and the presence of a person with further in-depth information if needed. Guidelines for the senior physician stipulated that he would only speak during team review and assessment phases if asked for additional information on the case. In reality, however, non-verbal reactions cannot be fully ruled out and transcripts reveal minor deviations from the guideline to not speak, e.g. repeated discussion in one external team about the differences in radiotherapy use between the two hospitals, triggered by questions to the case presenter from the external senior physician.
Case selection was not random and the predefined rule of heterogeneity was not followed as strictly as it could have been. For example, the case C (no current physical but high psychological needs) was deliberately included after controversial discussion in the internal team—and the three teams did not agree on it. The use of real patient cases instead of constructed, controlled cases makes it possible to analyse the challenges of case reviews as reference standard. However, we cannot determine the influence of that selection on the interrater agreement.
The qualitative analysis is based only on the three cases where there was disagreement, as these contribute most to the understanding of pitfalls and the need for adaptation of our processes in future research.
The approach of medical history taking and structured, multi-professional case review shows sufficient interrater agreement to be employed as a reference standard for SPC needs in studies validating screening tools. However, further research is needed to confirm these results. For use in studies in other countries or care services, we recommend a review and possibly adaption to national and local requirements.
Availability of data and materials
The datasets generated and/or analysed during the current study are not publicly available due to ensuring data protection and anonymity for the patients, but are available from the corresponding author on reasonable request.
Begum A. Using a screening tool to improve timely referral of patients from acute oncology-haematology to palliative care services. BMJ Open Quality 2013; 2(1).
Glare PA, Chow K. Validation of a simple screening tool for identifying unmet palliative care needs in patients with cancer. J Oncol Pract. 2015;11(1):e81–6.
Hui D, Mori M, Watanabe SM, Caraceni A, Strasser F, Saarto T, et al. Referral criteria for outpatient specialty palliative cancer care: an international consensus. Lancet Oncol. 2016;17(12):e552–9.
Hui D, Heung Y, Bruera E. Timely Palliative Care: Personalizing the Process of Referral. Cancers (Basel) 2022; 14(4).
van Oorschot B, Pigorsch S, Werner L, Roch C, Müller E, Alt-Epping B, et al. Von der „frühen“ zur „zeitgerechten“ Integration der Palliativversorgung in die Behandlung onkologischer patient:innen. Die Onkologie. 2022;28(11):964–73.
Chan AS, Rout A, ’Adamo CRD, Lev I, Yu A, Miller K. Palliative referrals in advanced cancer patients: utilizing the supportive and palliative care indicators tool and Rothman index. Am J Hosp Palliat Care. 2022;39(2):164–8.
Gómez-Batiste X, Martínez-Muñoz M, Blay C, Amblàs J, Vila L, Costa X, et al. Utility of the NECPAL CCOMS-ICO(©) tool and the Surprise Question as screening tools for early palliative care and to predict mortality in patients with advanced chronic conditions: a cohort study. Palliat Med. 2017;31(8):754–63.
Adelson K, Paris J, Horton JR, Hernandez-Tellez L, Ricks D, Morrison RS, et al. Standardized criteria for palliative care consultation on a solid tumor oncology service reduces downstream health care use. J Oncol Pract. 2017;13(5):e431–40.
Molin Y, Gallay C, Gautier J, Lardy-Cleaud A, Mayet R, Grach M-C, et al. PALLIA-10, a screening tool to identify patients needing palliative care referral in comprehensive cancer centers: A prospective multicentric study (PREPA-10). Cancer Med. 2019;8(6):2950–61.
Paiva CE, Paiva BSR, Menezes D, Zanini LE, Ciorlia JB, Miwa MU, et al. Development of a screening tool to improve the referral of patients with breast and gynecological cancer to outpatient palliative care. Gynecol Oncol. 2020;158(1):153–7.
Kim JY, Peters KB, Herndon JE2, Affronti ML. Utilizing a palliative care screening tool in patients with glioblastoma. J Adv Pract Oncol. 2020;11(7):684–92.
Matsumoto Y, Umemura S, Okizaki A, Fujisawa D, Kobayashi N, Tanaka Y, et al. Early specialized palliative care for patients with metastatic lung cancer receiving chemotherapy: a feasibility study of a nurse-led screening-triggered programme. Jpn J Clin Oncol. 2022;52(4):375–82.
Leitlinienprogramm Onkologie der Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften e. V. (AWMF), Deutschen Krebsgesellschaft e. V. (DKG) und Deutschen Krebshilfe. S3-Leitlinie Palliativmedizin für Patienten mit einer nicht heilbaren Krebserkrankung. Version 2; 2019. https://www.awmf.org/leitlinien/detail/ll/128-001OL.html. Accessed 6 Nov 2019.
Hui D, Titus A, Curtis T, Ho-Nguyen VT, Frederickson D, Wray C, et al. Implementation of the Edmonton symptom assessment system for symptom distress screening at a community cancer center: a pilot program. Oncologist. 2017;22(8):995–1001.
Ramsenthaler C, Haberland B, Schneider S, Hartmann L, Heußner P, Dreyling M, et al. Identifying Patients with Cancer Appropriate for Early Referral to Palliative Care Using the Integrated Palliative Care Outcome Scale (IPOS) - a cross-sectional study of acceptability and deriving valid cut-points for screening. Palliat Med. 2018;32:98.
Glare PA, Plakovic K, Schloms A, Egan B, Epstein AS, Kelsen D, et al. Study using the NCCN guidelines for palliative care to screen patients for palliative care needs and referral to palliative care specialists. J Natl Compr Canc Netw. 2013;11(9):1087–96.
Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L. et al. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. bmj 2015;351:5527.
Fisher G, Shadmi E, Porat-Packer T, Zisberg A. Identifying patients in need of palliative care: Adaptation of the Necesidades Paliativas CCOMS-ICO© (NECPAL) screening tool for use in Israel. Palliat Support Care; 022:1–7.
Ostgathe C, Wendt KN, Heckel M, Kurkowski S, Klein C, Krause SW, et al. Identifying the need for specialized palliative care in adult cancer patients–development and validation of a screening procedure based on proxy assessment by physicians and filter questions. BMC Cancer. 2019;19(1):646.
Hui D, Anderson L, Tang M, Park M, Liu D, Bruera E. Examination of referral criteria for outpatient palliative care among patients with advanced cancer. Support Care Cancer. 2020;28(1):295–301.
Wu X, Zhang X, Zhang J, Cui X. Analysis of knowledge, attitude and behavior of oncology medical staff in palliative care. Ann Palliat Med. 2020;9(3):985–92.
Salins N, Ghoshal A, Hughes S, Preston N. How views of oncologists and haematologists impacts palliative care referral: a systematic review. BMC Palliat Care. 2020;19(1):175.
Formagini T, Poague C, O’Neal A, Brooks JV. “When I Heard the Word Palliative”: Obscuring and clarifying factors affecting the stigma around palliative care referral in oncology. JCO Oncology Practice. 2022;18(1):e72–9.
Cripe JC, Mills KA, Kuroki LK, Wan L, Hagemann AR, Fuh KC, et al. Gynecologic Oncologists’ perceptions of palliative care and associated barriers: a survey of the society of gynecologic oncology. Gynecol Obstet Invest. 2019;84(1):50–5.
Veldhoven CMM, Nutma N, de Graaf W, Schers H, Verhagen CAHHVM, Vissers KCP, et al. Screening with the double surprise question to predict deterioration and death: an explorative study. BMC Palliat Care. 2019;18(1):118.
Temel JS, Greer JA, Muzikansky A, Gallagher ER, Admane S, Jackson VA, et al. Early palliative care for patients with metastatic non–small-cell lung cancer. N Engl J Med. 2010;363(8):733–42.
Vanbutsele G, Pardon K, van Belle S, Surmont V, de Laat M, Colman R, et al. Effect of early and systematic integration of palliative care in patients with advanced cancer: a randomised controlled trial. Lancet Oncol. 2018;19(3):394–404.
Ferrell BR, Temel JS, Temin S, Alesi ER, Balboni TA, Basch EM, et al. Integration of palliative care into standard oncology care: American society of clinical oncology clinical practice guideline update. J Clin Oncol. 2017;35(1):96–112.
Haun MW, Estel S, Rücker G, Friederich H-C, Villalobos M, Thomas M et al. Early palliative care for adults with advanced cancer. Cochrane Database of Systematic Reviews 2017; (6).
Rodin R, Swami N, Pope A, Hui D, Hannon B, Le LW, et al. Impact of early palliative care according to baseline symptom severity: secondary analysis of a cluster-randomized controlled trial in patients with advanced cancer. Cancer Med. 2022;11(8):1869–78.
Murtagh FE, Ramsenthaler C, Firth A, Groeneveld EI, Lovell N, Simon ST, et al. A brief, patient- and proxy-reported outcome measure in advanced illness: Validity, reliability and responsiveness of the Integrated Palliative care Outcome Scale (IPOS). Palliat Med. 2019;33(8):1045–57.
Bruera E, Kuehn N, Miller MJ, Selmser P, Macmillan K. The Edmonton Symptom Assessment System (ESAS): a simple method for the assessment of palliative care patients. J Palliat Care. 1991;7(2):6–9.
Rutjes AWS, Reitsma JB, Coomarasamy A, Khan KS, Bossuyt PMM. Evaluation of diagnostic tests when there is no gold standard. A review of methods. Health Technol Assess 2007; 11(50):iii, ix-51.
Teike Lüthi F, Bernard M, Vanderlinden K, Ballabeni P, Gamondi C, Ramelet A-S, et al. Measurement properties of ID-PALL, a new instrument for the identification of patients with general and specialized palliative care needs. J Pain Symptom Manage. 2021;62(3):e75–84.
Effendy C, Silva, Jony Francisco Dos Santos, Padmawati RS. Identifying palliative care needs of patients with non-communicable diseases in Indonesia using the SPICT tool: a descriptive cross-sectional study. BMC Palliative Care 2022; 21(1):13.
Müller E, Müller MJ, Boehlke C, Ramsenthaler C, Jäger H, Schäfer H, et al. Development of a screening tool for the need of specialist palliative care in oncologic inpatients: study protocol for the ScreeningPALL Study. BMJ Open. 2022;12(9):e059598.
Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977:159–74.
IBM SPSS Statistics for Windows. Version 27.0. Armonk, NY: IBM Corp.; 2020.
Kruse J. Qualitative Interviewforschung: Ein integrativer Ansatz. 2nd ed. Weinheim, Basel, München: Beltz Juventa; Ciando; 2015.
Acheson RM. The definition and identification of need for health care. J Epidemiol Community Health. 1978;32(1):10–5.
Simon ST, Ramsenthaler C, Bausewein C, Krischke N, Geiss G. Core attitudes of professionals in palliative care: a qualitative study. Int J Palliat Nurs. 2009;15(8):405–11.
Mistry B, Bainbridge D, Bryant D, Tan Toyofuku S, Seow H. What matters most for end-of-life care? Perspectives from community-based palliative care providers and administrators. BMJ Open. 2015;5(6):e007492.
We would like to thank all members of the PC teams of Freiburg, Erlangen und Cologne University Hospitals. Without their expertise, time and commitment, the study would not have been possible. We would also like to thank the directors, senior physicians and staff of the recruiting Department of Radiation Oncology. Last but not least we thank the patients, who contributed not only their time but were also open to disclosing their medical history and allowing us to report on it.
Open Access funding enabled and organized by Projekt DEAL. This work is supported by German Cancer Aid grant number 70114472.
Ethics approval and consent to participate
The study was approved by the Ethics Committee of Albert-Ludwigs-University Freiburg, Germany (Approval No. 20–1103; 17/12/2020). All study participants (patients and PC team members) provided written informed consent before participation. All procedures are in accordance with the ethical standards of the institutional and national research committee and with the 1964 Helsinki Declaration and its later amendments.
Consent for publication
Informed consent has been obtained from patient or proxy.
The authors declare no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Development and Documentation of the Reference Standard. Detailed description of a process and documentation of medical history taking and case review as reference standard, including final versions of the used documents.
Characteristics of not selected patients. Characteristics of patients not selected for case reviews (n=10) and excluded patients (n=2).
Additional results of qualitative analysis. Additional results of qualitative analysis of cases with no agreement regarding SPC need assessment.
Observations on the case review procedure. Observations on the case review process, including effort of case reviewers, framework conditions, supportive needs profile, and oral case presentation.
About this article
Cite this article
Müller, E., Müller, M.J., Seibel, K. et al. Interrater agreement of multi-professional case review as reference standard for specialist palliative care need: a mixed-methods study. BMC Palliat Care 22, 181 (2023). https://doi.org/10.1186/s12904-023-01281-7