- Research article
- Open Access
- Open Peer Review
How to help researchers in palliative care improve responsiveness to migrants and other underrepresented populations: developing and testing a self-assessment instrument
BMC Palliative Care volume 18, Article number: 83 (2019)
European migrant populations are aging and will increasingly be in need of palliative and end of life care. However, migrant patients are often underrepresented in palliative care research populations. This poses a number of drawbacks, such as the inability to generalize findings or check the appropriateness of care innovations amongst migrant patients. The aim of this study was to develop a self-assessment instrument to help palliative care researchers assess and find ways to improve their projects’ diversity responsiveness in light of the aging migrant population, and determine whether in addition to older migrants other groups should be included in the instrument’s focus.
After developing a concept instrument based on the standards for equity in healthcare for migrants and other vulnerable groups, literature review and interviews with palliative care researchers, we conducted a Delphi study to establish the content of the self-assessment instrument and used think aloud methods in a study involving seven projects for usability testing of the self-assessment instrument.
A Delphi panel of 22 experts responded to a questionnaire consisting of 3 items concerning the target group and 30 items on diversity responsiveness measures. Using an a priori set consensus rate of 75% to include items in the self-assessment instrument, experts reached consensus on 25 out of 30 items on diversity responsiveness measures. Findings furthermore indicate that underserved groups in palliative care other than migrant patients should be included in the instrument’s focus. This was stressed by both the experts involved in the Delphi study and the researchers engaged in usability testing. Usability testing additionally provided insights into learnability, error-rate, satisfaction and applicability of the instrument, which were used to revise the self-assessment instrument.
The final self-assessment instrument includes a list of 23 diversity responsiveness measures to be taken at varying stages of a project, and targets all groups at risk of being underrepresented. This instrument can be used in palliative care research to assess diversity responsiveness of projects and instigate action for improvement.
The changing age structure of migrant populations across Europe calls for consideration of the particular needs of older migrants in health service provision . Older migrants experience frailty and poorer health outcomes compared to non-migrants [1, 2]. As a consequence they are expected to gradually need more healthcare, including palliative and end of life care. However, studies have shown low use of palliative care services and higher levels of dissatisfaction with these services amongst this group [3, 4].
In the Netherlands, older migrants with a non-western background currently make up 4 % of the population of adults aged 65 and older. By 2060, their number is expected to have increased to 17% . These older migrants are predominantly of the first generation and have a Surinamese, Turkish or Moroccan background. Perspectives on what constitutes good palliative care have been found to differ between terminally ill patients with a Turkish or Moroccan background and their Dutch care providers [6, 7]. Patients with a Turkish or Moroccan background generally preferred not to have an infaust diagnosis or prognosis disclosed in order to keep hope alive, had a preference for maximum and curative treatment until the end of life, as well as a strong preference for family care [6, 7].
Culture, diversity, and their operationalization have previously been identified as a priority for research in palliative care in Europe in order to address questions on differing needs and appropriateness of services . However, we perpetually do not see migrant patients represented in palliative care research populations. Researchers across the health related fields have reported difficulties engaging migrant patients as well as other socially disadvantaged groups, and have attributed this to barriers with regards to sampling, recruitment, gaining consent, data collection, intervention delivery and retention . In palliative care research, engaging migrant patients may additionally be difficult because patients’ preference for non-disclosure and curative treatment complicate discussion of palliative care. The underrepresentation of migrant patients poses a number of drawbacks, such as the inability to generalise findings or check the appropriateness of care innovations amongst migrant patients .
In the Netherlands, a considerable amount of palliative care research is conducted as part of the national program for palliative care innovation (Palliantie. Meer dan Zorg), funded by The Netherlands Organization for Health Research and Development. Over 60 research, intervention and education projects that aim to improve palliative care have been initiated between the start of this program in 2014 and the moment of study. In order to ensure representation of migrant patients in these projects researchers ideally should ensure their projects are responsive to patient diversity.
Responsiveness refers to a concept of patient experience that includes not just the interpersonal process between practitioner and patient, but the interaction between the health system and the population it serves . Several strategies to improve responsiveness to patient diversity exist for the context of health care organizations . The standards for equity in healthcare for migrants and other vulnerable groups, for example, are an elaborately tested self-assessment instrument that helps organisations assess their capacity to improve accessibility and quality of care for migrants and other vulnerable groups .
The aim of this study was to develop a self-assessment instrument to help researchers assess and find ways to improve their palliative care projects’ responsiveness to diversity, in light of the aging migrant population. It therefore predominantly concerns migrants of the first generation, and we use the term migrant to refer to persons who changed their country of residence, irrespective of the reason for migration or legal status, but not their offspring born in the country of settlement. We used the standards for equity in healthcare for migrants and other vulnerable groups – from here on equity standards – as a basis to develop a pilot instrument. The equity standards, however, have been developed for the context of healthcare organisations. We therefore adjusted the standards to be applicable for the context of palliative care research projects. Furthermore, the equity standards focus on migrants and other vulnerable groups while our aim was to develop an instrument that focuses on migrants only. We therefore also aimed to assess whether the pilot instrument’s focus was appropriate. This lead to the following research questions:
What should be included in a self-assessment instrument evaluating the diversity responsiveness of palliative care projects?
On which groups should a self-assessment instrument evaluating the diversity responsiveness of palliative care projects focus; should other groups, in addition to migrants, be included?
How do users evaluate the usability of this self-assessment instrument for improving the diversity responsiveness of palliative care projects?
We developed a pilot instrument which we tested in two ways simultaneously. We used a Delphi study to test the content of the instrument and think-aloud methods to test the usability of the instrument.
Development of the pilot instrument
We developed the pilot instrument in two steps. First, we read academic literature on strategies to increase responsiveness to patient diversity as well as proposals of projects in the national program for palliative care innovation to establish a framework for diversity responsiveness in palliative care projects. We focused this initial framework on the domains of the equity standards: equity in policy; equitable access and utilisation; equitable quality of care; equity in participation; and, promoting equity (see Fig. 1) . Secondly, we tested the applicability of this framework in semi-structured interviews with researchers involved in five palliative care projects. A description of the projects can be found at the description of the sample under section 2.2 (usability testing) of this article. During the interviews we discussed the perceived relevance and feasibility of each domain of the equity standards in relation to the projects. The insights we gained from analysis of the interview transcripts helped us develop the pilot instrument.
The pilot instrument consisted of a general introduction and instructions for use, followed by 20 diversity responsiveness measures structured in sections of the project stages: set-up; execution; and, follow-up (Table 1). Each measure could be assigned a score between one and five to describe its level of implementation in the project. Each section ended with a textbox where areas for improvement could be described. Lastly, the instrument contained a number of tips and recommendations for improvement of diversity responsiveness.
To test the content of the pilot instrument we conducted a Delphi study. A Delphi study is a structured process that combines expert opinion into group consensus in a series of questionnaires, or ‘rounds’ . Our Delphi study consisted of two rounds.
We translated the content of the pilot instrument into a questionnaire and asked a panel of experts to score the questionnaire items’ relevance and feasibility on a 5-point scale (1:not relevant/feasible; 5:entirely relevant/feasible). Feedback on the results from round one was provided to the experts in the subsequent round with the aim to increase consensus. We used an a priori set consensus rate of 75%, where at least 75% of the scores given by the experts should be 4 ≤ (relevance), or 3 ≤ (feasibility) in order to uphold the item in the instrument. We used a lower cut-off score for feasibility to avoid having to exclude items that scored high on relevance but low on feasibility. Relevance was the main factor because the measures in the self-assessment instrument are to represent a certain standard of diversity responsiveness of palliative care projects.
We selected a purposive sample of experts who could provide varying perspectives, i.e. experts from various research fields, representatives of migrant organizations, healthcare professionals and subsidy providers. There is no clear guidance on the number of experts in a Delphi panel, but most studies use panels of between 15 to 35 people . We invited 50 experts to circumvent issues of non-response and ensure a number between 15 and 35. The study was carried out in Dutch, permitting participation by experts from the Netherlands and Belgium.
The Delphi panel ran from March to July 2018. Questionnaires were sent via email. The questionnaire consisted of 33 items, subdivided into four sections. One section with three items on the groups included in the instrument’s focus and three sections with items on the measures from the pilot instrument according to the project stages: set-up; execution, and follow-up. Some of the 20 measures contained multiple steps. We separated these into several items to diminish ambiguity in the questionnaire which resulted in a total of 30 items. The complete list of items is included as an attachment (See Additional file 1). We instructed experts to score items’ relevance and feasibility and give remarks at the end of each section.
We analysed the data from round one and removed items on which consensus had been reached from the questionnaire. We asked the experts to re-evaluate the remaining items in round two. For each of these items, experts received their own score and the overall mean score. Experts were also asked to share any suggestions they had for alternative formulation of the items.
The Delphi method produced both quantitative and qualitative data. We exported the quantitative data to SPSS to calculate mean scores and consensus rates. We analysed the qualitative data using thematic analysis . Authors KS, MT and JS reviewed the data to identify key themes and subthemes. These were used in order to adjust the questionnaire for round two as well as inform the overall analysis.
To be able to test and refine the instrument we adopted methods of usability testing . Seven projects from the national program for palliative care innovation used our instrument to conduct a self-assessment while we were present for observation. Once, the lead researcher completed the self-assessment individually and a concurrent think aloud protocol was used. In this method participants verbalize their thoughts while performing tasks to evaluate an artifact . In the other cases the research team completed the assessment together and constructive interaction was used as a method to test usability. In this method participants work together in performing a task, thereby verbalizing their thoughts through interaction . Following the self-assessment we held semi-structured interviews with the lead researchers to gather input on their experience and discuss the usability of the instrument in improving diversity responsiveness of palliative care projects.
We selected a convenience sample of projects from the national program for palliative care innovation through one of the palliative care consortia in the Netherlands. Nonetheless, the sampled projects were diverse in focus and status of implementation. The projects aimed to 1) develop, implement and evaluate an advance care planning intervention for primary care practice; 2) develop, implement and evaluate a seamless palliative care trajectory between different care settings; 3) evaluate and strengthen a procedure to improve palliative homecare where GPs, community nurses and palliative care consultants work together to identify and meet the needs of palliative patients; 4) assess the palliative care needs and barriers to care for homeless people; 5) study goals set by patients with metastatic lung cancer and their physicians before treatment with chemotherapy, targeted therapy or immunotherapy and whether these goals are met; and 6 & 7) two projects aimed to strengthen shared decision-making at the end of life by developing and implementing shared decision-making tools, as well as methods to engage all stakeholders in shared-decision making in palliative care. Five of these projects had been involved in the development of the pilot instrument. We purposively included two additional projects (No. 6 & 7) because they were yet to start and could offer distinct insights from that stage.
MT observed the self-assessment and subsequently interviewed the projects’ lead researchers about the usability of the instrument. All self-assessments, the concurrent think-aloud and constructive interaction procedures, and interviews were recorded and transcribed verbatim.
We analysed the transcripts of self-assessment and interviews using thematic analysis . We searched for components of usability, i.e. learnability (how well can users complete self-assessment when they use the instrument for the first time?), error rate (how many errors do users make, how severe are these errors?), satisfaction (how pleasant is it to use the instrument), and applicability (how do users prefer to apply the self-assessment) . We used the results from this analysis to further develop the self-assessment instrument. We additionally identified themes concerning the effect of the self-assessment on diversity responsiveness of palliative care projects.
Twenty-two experts completed the questionnaire in round one (44% response rate). Expertise varied from the field of palliative care (6), diversity in healthcare (5), migrant health (3), culture sensitive care (2), patient participation (1), or other (3). Two experts reported overlapping areas of expertise (i.e. palliative care for migrant patients).
The complete list of questionnaire items with mean scores and percentages is included as an attachment (see Additional file 1.). In round one the experts reached the consensus rate of 75% on 26 out of 33 items, with regards to relevance. With regards to feasibility, the consensus rate of 75% was reached for all but one item (item 23: on diversity in the workforce). However, qualitative analysis indicated that experts did debate feasibility as it was dependent on factors such as availability of means or data.
“In my opinion many of the abovementioned aspects [items] are relevant, but the feasibility is often concerned with time and money. I am not sure how costly each individual measure is, but I think making a case for the relevance (by thorough research and lobby) can increase feasibility.” (2018.12.1 Expert)
Furthermore, experts agreed on the need for increased responsiveness to diversity and recommended to, in addition to migrants, include other groups at risk of being underrepresented. Some experts advised an intersectional approach. Intersectionality  can be used as an analytic framework to identify how social categorizations such as ethnicity, socioeconomic status or gender create overlapping and interdependent systems of discrimination or disadvantage that lead to exclusion. However, experts did mention intersectionality is a complex concept and a truly intersectional approach might be difficult to achieve within palliative care projects. Based on these results we constructed a new description of diversity responsiveness, extending our focus to migrants and other vulnerable groups, and asked the expert panel for consensus on this description in round two.
Sixteen experts completed round two. Expertise varied from the field of palliative care (4), diversity in healthcare (4), migrant health (3), culture sensitive care (2) or other (3). Experts were asked to re-evaluate seven items with regards to relevance, which was the main factor because the measures in the self-assessment instrument are to represent a certain standard of diversity responsiveness of palliative care projects. The panel reached the consensus rate of 75% for one additional item (item 22: on training in cultural competence).
One item was the new description of diversity responsiveness. Experts generally agreed with the new description and offered slight additions. E.g. instead of the term vulnerable groups it was suggested to use the term underserved groups. And while endorsing the extended focus, experts stressed that in deliberating the intersecting factors that put patients at risk of being underserved, the variety of factors at play for migrant patients necessitates continued special consideration of this group. This led to a final description of diversity responsiveness as: “Diversity responsiveness of palliative care projects entails deliberation of all intersecting factors that may cause patients to be underrepresented in palliative care research and underserved by palliative care services. ... These include factors such as educational background, socioeconomic status, physical or mental disability, age, sex, gender, sexual orientation, language, religion, culture, ethnicity and migration history. We ask special consideration of factors at play for migrant patients.” (see Additional file 2).
The remaining five items did not reach the cut-off score (items 4 & 5: on gaining insight into the diversity of the patient population under study and describing this in the project proposal; item 8: on registering patient ethnicity to monitor the use of the intervention under study; item 23: on encouraging diversity in the workforce of the healthcare organisations involved in the study; and, item 31: on adding tools to enhance culture sensitivity product from the project). These five items were removed from the final list of items for the self-assessment instrument. Comments by the Delphi experts indicated that these items were deemed not reasonably achievable or not desirable. In fact, mean scores for these items with regards to relevance were slightly lower than in round one. The final list of items was shared with the Delphi panel.
For the final self-assessment instrument we eventually merged two items and included one as a recommendation rather than a measurable element. The final instrument thus consists of 23 measures of diversity responsiveness.
Results from usability testing gave insight into learnability, error-rate, satisfaction and applicability and the effect of the self-assessment on diversity responsiveness of palliative care projects. We used these to further develop the self-assessment instrument. The observed usability issues and subsequent adaptations to the instrument can be found in Table 2. We will discuss findings with regard to satisfaction, applicability and effect.
Overall, users were satisfied with the self-assessment. They did, however, report some concern about the amount of text and use of difficult or unfamiliar terms.
“I had indeed already read it … and I have to say, it didn’t come easily. … That has to do with words such as intersectionality and there were some other terms. But otherwise it was quite accessible.” (2018.13 Researcher)
Some users mentioned they found the assessment confronting, because it uncovered the paucity in measures they had taken to be responsive. However, the subsequent discussion that lead to actions being formulated was regarded as very positive.
Yes, indeed, I found that to be very nice! … The actions are supported by the group. Especially because it is so confronting, in a certain way, that you see ‘we don’t do any of this’. And there was no one, not anyone that said ‘we don’t have to do it’ and is completely … I don’t think you will convince those. But in this group I really liked that it resulted in supported actions.” (2018.13 Researcher)
Whether diversity responsiveness of palliative care projects can be reasonably achieved with this instrument was said to depend on timing of the self-assessment and whether researchers remembered to use the instrument. Users indicated they prefer to conduct the self-assessment during the early stages of the project, to be able to incorporate actions into project plans. Users saw the potential of using the instrument in differing project stages, but were uncertain about the likelihood of actually repeating the self-assessment. This depended on whether they were reminded, or even obliged, to do so. For instance because the self-assessment is embedded in a process of acquiring funding.
“I think that the need for this instrument especially exists when writing proposals. That people’s attention is drawn to the instrument when they are writing proposals. … I think it would be useful for people. Because, usually, you merely do something based on your own vision and what you know as a team. But usually there is a lot more, and this [the instrument] can help you with that.” (2018.10 Researcher)
As the Delphi study and usability testing occurred simultaneously no adjustments had been made to the pilot instrument based on the results from the Delphi study. Interestingly, the instrument’s focus on migrant patients similarly elicited discussion. Users indicated that underserved groups in palliative care other than migrant patients should be included. For example, palliative care has for many years focused mainly on oncology patients as a result of which patient groups with other conditions have been underserved. Moreover, educational level, (health) literacy level and religion were mentioned as factors that needed consideration. It was suggested that broadening the focus would increase the applicability of the instrument.
“You’re not going to do all of them. It’s as simple as that. … If you target this towards low literacy, I’m just picking something, and not specifically towards cultural diversity. Which is indeed very important, and can be an aspect of it, but if it is not only targeted towards that … I am not going to use one tool for patients with a non-western background, another for … The broader the applicability of a tool … ” (2018.12 Researcher)
We found the self-assessment mainly functioned as a way to raise awareness about diversity amongst palliative care patients. The instrument offered a structured way to think about the diversity responsiveness of a project.
“The good thing about this instrument, in your project proposal you do mention diversity, but this forces you to think more carefully about how you are going to incorporate it in your project. And it points out your shortfalls.” (2018.14 Researcher)
During all self-assessments users identified actions they could take to improve diversity responsiveness in current or future projects. Users expressed an ambition to make these improvements, but did so carefully.
Yes, well I would be interested to conduct an additional small-scale qualitative research … I really like the idea. But we’re not pinned down on what we promise here, are we? (2018.15 Researcher)
With this study we aimed to develop and test a self-assessment instrument to help researchers assess and find ways to improve their palliative care projects’ responsiveness to diversity in light of the aging migrant population. By implementing all changes and omitting some items the self-assessment instrument includes a list of 23 measures to be taken at varying stages of a project and targets all groups at risk of being underrepresented in palliative care research and underserved by palliative care services. The measures cover topics important during project set-up: describing the diversity of the patient population in the project proposal; monitoring engagement of a diverse patient population; establishing a representative project team; project execution: patient and caregiver participation; access to and quality of care within the project and, follow-up: responsive implementation of project results; and, sharing experiences on responsiveness in palliative care projects. The complete final instrument is included as an attachment (see Additional file 2).
Our findings indicate that underserved groups in palliative care other than migrant patients should be included in the instrument’s focus. This was stressed by both the experts involved in the Delphi study and the researchers engaged in usability testing. Indeed, a shift away from single target groups has been the trend in thinking about responsiveness [20, 21]. Our final instrument is now more in accordance with the approach of the equity standards, which targets all groups at risk of inequities in health and healthcare in response to the increasing differentiation of diversity  and the need to recognize multiple-diversity needs, as individual needs are generally expressed by the intersection of many factors such as origin, class, gender, age and ability .
While our Delphi results indicate a need to increase awareness of the way in which intersections of differences put certain groups at risk of being underrepresented in palliative care research and practice, usability testing showed researchers were unfamiliar with the concept of intersectionality and were more practically motivated to extend the focus beyond migrant patients. Extending the focus would increase the instrument’s applicability and projects that worked to improve palliative care for underserved groups other than migrants liked to see their efforts reflected in the self-assessment. Indeed, intersectionality is often seen as difficult to implement in, especially quantitative, health research methodology . Our final instrument, therefore, asks for consideration of the varying dimensions of diversity but leaves the choice for an intersectional approach to the researchers. Furthermore, in the deliberation of all factors that may cause patients to be underrepresented in palliative care research we do continue to ask for special consideration of factors at play for migrant patients.
Results from usability testing show researchers were able to conduct the self-assessment and were satisfied with the process. The self-assessment helps raise awareness and offers a structured way to think about the diversity responsiveness of a project. Researchers should be attentive to the risk of using the self-assessment to pick and choose actions, rather than using it to make structural improvements in diversity responsiveness. Embedding the self-assessment in the process of acquiring funding can encourage future use. Research has shown that researchers whose funding included a mandate on minority participant inclusion adopted more, and more active recruitment strategies in recruiting minorities . At the same time, increased engagement of underrepresented populations has been found to be beneficial to secure funding, design study protocols, increase enrolment rates and select relevant outcomes, resulting in reporting that is more meaningful and understandable for patients and the community .
To our knowledge, we are the first to apply the content of the equity standards, an instrument for the context of healthcare organisations, to the context of health related research. In an elaborate development phase we adjusted the content of the equity standards to fit the context of research, specifically research in palliative care. Our findings suggest the equity standards are applicable in this context and form a good basis for a self-assessment instrument to be used to improve diversity responsiveness of palliative care research. As such our self-assessment instrument may complement the MORECare statement, a checklist issued to set a standard for research in end of life care . The MORECare statement does include statements on engaging patients but puts more emphasis on the vulnerable position of patients approaching the end of life, with statements as “Collaborate with patients and caregivers in the design of the study, vocabulary used in explaining the study, consent procedures and any ethical aspects.” or “Ensure proportionality in patient and caregiver information sheets, appropriate to the study design and level of risk, as excessive information in itself can be quite tiring/distressing for very ill individuals.”. Our self-assessment instrument may additionally provide a set of standards to help engage diverse patients.
Our findings are limited by the fact that our self-assessment instrument was developed in the context of the Netherlands and the national program for palliative care innovation (Palliantie. Meer dan Zorg). The particular needs of migrants and other underserved patients will similarly need to be considered in palliative care research in other European countries, and further research is needed to be able to generalize our findings and make the self-assessment instrument applicable to other contexts. Our findings are additionally limited by the fact that four of the projects involved in usability testing performed at least part of the self-assessment in retrospect because their projects were in a well-advanced project stage. This may have influenced users’ readiness to identify areas for improvement. Furthermore, the use of think aloud methods during usability testing required MT to attend the self-assessment. Her presence may have influenced the research teams’ efforts and prompted socially desirable answers. A next step may be to conduct an implementation study to monitor the actions taken by researchers involved in the study and evaluate their effect on the engagement of underrepresented populations.
We have developed and tested a self-assessment instrument which can be used by researchers involved in palliative care projects to assess and find ways to improve their projects’ diversity responsiveness. It helps raise awareness and offers a structured way to think about diversity responsiveness and identify areas for improvement. We recommend performing the self-assessment during the early stages of a project, with the option to repeat it in each progressing stage. Improving responsiveness of projects will help researchers tackle the problem of underrepresentation of migrants and other underserved populations and enable them to address questions on differing needs and appropriateness of services in palliative care as part of their research.
Availability of data and materials
The datasets generated and analysed during this study are available from the corresponding author on reasonable request.
Kristiansen M, Razum O, Tezcan-Güntekin H, Krasnik A. Aging and health among migrants in a European perspective. Public Health Rev. 2016;37(1):20.
Walkden G, Anderson E, Vink M, Tilling K, Howe L, Ben-Shlomo Y. Frailty in older-age European migrants: cross-sectional and longitudinal analyses of the survey of health, aging and retirement in Europe (SHARE). Soc Sci Med. 2018;213:1–11.
Evans N, Menaca A, Andrew EV, Koffman J, Harding R, Higginson IJ, et al. Systematic review of the primary research on minority ethnic groups and end-of-life care from the United Kingdom. J Pain Symptom Manag. 2012;43(2):261–86.
Fang ML, Sixsmith J, Sinclair S, Horst G. A knowledge synthesis of culturally- and spiritually-sensitive end-of-life care: findings from a scoping review. BMC Geriatr. 2016;16:107.
Statline C. Prognose bevolking; geslacht, leeftijd en migratieachtergrond, 2018-2060. https://statline.cbs.nl/Statweb/publication/?DM=SLNL&PA=83784NED&D1=0&D2=0&D3=0,l&D4=0&D5=0-4&D6=0-2,7,12,17,22,27,32,37,l&HDR=T,G3,G2,G1,G4&STB=G5&VW=T; 2018. Accessed 4 Dec 2018.
de Graaff FM, Francke AL, van den Muijsenbergh ME, van der Geest S. 'Palliative care': a contradiction in terms? A qualitative study of cancer patients with a Turkish or Moroccan background, their relatives and care providers. BMC palliative care. 2010;9:19.
de Graaff FM, Mistiaen P, Devillé WL, Francke AL. Perspectives on care and communication involving incurably ill Turkish and Moroccan patients, relatives and professionals: a systematic literature review. BMC palliative care. 2012;11(1):17.
Gysels M, Evans N, Menaca A, Andrew EV, Bausewein C, Gastmans C, et al. Culture is a priority for research in end-of-life care in Europe: a research agenda. J Pain Symptom Manag. 2012;44(2):285–94.
Bonevski B, Randell M, Paul C, Chapman K, Twyman L, Bryant J, et al. Reaching the hard-to-reach: a systematic review of strategies for improving health and medical research with socially disadvantaged groups. BMC Med Res Methodol. 2014;14(1):42.
Albain KS, Unger JM, Crowley JJ, Coltman CA, Hershman DL. Racial disparities in cancer survival among randomized clinical trials patients of the southwest oncology group. J Natl Cancer Inst. 2009;101(14):984–92.
Valentine NB, de Silva A, Kawabata K, Darby C, Murray CJL, Evans DB. Health system responsiveness: Concepts, domains and operationalization. In: Murray, C.J.L., Evans, D., editors. Health systems performance assessment: Debates, methods, and empiricism. World Health Organization, Geneva. 2003. p. 573–593.
Seeleman C, Essink-Bot ML, Stronks K, Ingleby D. How should health service organizations respond to diversity? A content analysis of six approaches. BMC Health Serv Res. 2015;15:510.
Chiarenza A. Standards for equity in health care for migrants and other vulnerable groups self-assessment tool for pilot implementation. HPH Task Force on Migrant-Friendly and Culturally Competent Health Care; 2014.
Keeney S, Hasson F, McKenna HP. A critical review of the Delphi technique as a research methodology for nursing. Int J Nurs Stud. 2001;38(2):195–200.
Day J, Bobeva MJ. A generic toolkit for the successful management of Delphi studies. EJBRM. 2005;3(2):103–16.
Spencer L, Ritchie J. Qualitative data analysis for applied policy research. In: Huberman AM, Miles MB, editors. The qualitative Researcher's companion. Thousand Oaks: SAGE; 2002.
van den Haak M. A penny for your thoughts: investigating the validity and reliability of think-aloud protocols for usability testing: University of Twente, Enschede; 2008.
Nielsen J. Usability engineering. Boston: Academic Press; 1993.
Crenshaw K. Mapping the margins: Intersectionality, identity politics, and violence against women of colour. Stanford Law Rev. 1991;43(6):1241–300.
Chiarenza A. Developments in the concept of 'cultural competence'. In: Ingleby D, Antonio C, Walter D, Kotsioni I, editors. Inequalities in healthcare for migrants and ethnic minorities. Antwerp - Apeldoorn: Garant Publishers; 2012. p. 66–81.
Cattacin S, Chiarenza A, Domenig D. Equity standards for health care organisations: a theoretical framework. Divers Equal Health Care. 2013;10(4):249–58.
Vertovec S. Super-diversity and its implications. Ethn Racial Stud. 2007;30(6):1024–54.
Yuval-Davis N. Intersectionality and feminist politics. Eur J Wom Stud. 2006;13(3):193–209.
Bauer GR. Incorporating intersectionality theory into population health research methodology: challenges and the potential to advance health equity. Soc Sci Med. 2014;110:10–7.
Quinn SC, Butler J III, Fryer CS, Garza MA, Kim KH, Ryan C, et al. Attributes of researchers and their strategies to recruit minority populations: results of a national survey. Contemp Clin Trials. 2012;33(6):1231–7.
Domecq JP, Prutsky G, Elraiyah T, Wang Z, Nabhan M, Shippee N, et al. Patient engagement in research: a systematic review. BMC Health Serv Res. 2014;14(1):89.
Higginson IJ, Evans CJ, Grande G, Preston N, Morgan M, McCrone P, et al. Evaluating complex interventions in end of life care: the MORECare statement on good practice generated by a synthesis of transparent expert consultations and systematic reviews. BMC Med. 2013;11(1):111.
This study was part of a project of the program Palliantie. Meer dan Zorg funded by The Netherlands Organization for Health Research and Development. The funding body had no further role in the design of the study and collection, analysis, and interpretation of data nor in writing the manuscript.
Ethics approval and consent to participate
According to the Dutch Medical Research Involving Human Subjects Act, this study did not require approval by a medical research ethics committee in the Netherlands. Respondents were researchers and professionals, not patients and data collection methods consisted merely of interviews and questionnaires. We followed the ethical principles for medical research involving human subjects as laid down in the Declaration of Helsinki and adopted by the World Medical Association (WMA Declaration of Helsinki 2000). Each respondent was adequately informed of the aims and methods of the study and written consent for participation in this study was obtained from the respondents. No personal information has been used and the individual’s identity has been protected by removing any personal identifiers from the data. Codes were used to designate the respondents to guarantee their anonymity.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional file 1. Final list of Delphi questionnaire items with mean scores and percentages. The Delphi questionnaire was established by translating the content of a pilot self-assessment instrument for diversity responsiveness in palliative care projects into a questionnaire. The questionnaire consisted of 33 items, subdivided into four sections. One section with three items on the groups included in the instrument’s focus and three sections with items on the measures from the pilot instrument according to the project stages: set-up; execution, and follow-up. The questionnaire was sent to a panel of experts who were asked to score the questionnaire items’ relevance and feasibility on a 5-point scale (1:not relevant/feasible; 5:entirely relevant/feasible). A priori set consensus rate of 75% were used, where at least 75% of the scores given by the experts should be 4 ≤ (relevance), or 3 ≤ (feasibility) in order to uphold the item in the instrument. Listed here are the final items, after revisions suggested by experts, the mean scores and consensus rate percentages.
Additional file 2. Self-assessment instrument Diversity responsiveness in palliative care projects. The aim of this self-assessment instrument is to help researchers establish a project responsive to the diversity of the palliative care patient population. It helps to assess your project’s responsiveness at present and identify areas for improvement. The instrument is structured according to three project stages: 1) the project set-up; 2) the project execution; 3) project follow-up. By means of 23 diversity responsiveness measures you assess your project’s responsiveness at present. Every measure has three response options indicating the level of implementation in the project (no, partially, completely). You can specify actions for improvement on each measure. This self-assessment instrument has been developed as part of the project “Palliatieve zorgprojecten langs de diversiteitsmeetlat”, an implementation project of the program Palliantie. Meer dan Zorg funded by The Netherlands Organization for Health Research and Development. The development of self-assessment instrument is based on the standards for equity in healthcare for migrants and other vulnerable groups  and additional of literature research, expert consultation and a usability study.
About this article
Cite this article
Torensma, M., Onwuteaka-Philipsen, B.D., Strackee, K.L. et al. How to help researchers in palliative care improve responsiveness to migrants and other underrepresented populations: developing and testing a self-assessment instrument. BMC Palliat Care 18, 83 (2019). https://doi.org/10.1186/s12904-019-0470-1
- Palliative care research
- Self-assessment instrument
- Migrant patients
- Underrepresented populations