- Research article
- Open Access
- Open Peer Review
Development and validation of the quality care questionnaire –palliative care (QCQ-PC): patient-reported assessment of quality of palliative care
BMC Palliative Carevolume 17, Article number: 40 (2018)
In this study, we aimed to develop and validate an instrument that could be used by patients with cancer to evaluate their quality of palliative care.
Development of the questionnaire followed the four-phase process: item generation and reduction, construction, pilot testing, and field testing. Based on the literature, we constructed a list of items for the quality of palliative care from 104 quality care issues divided into 14 subscales. We constructed scales of 43 items that only the cancer patients were asked to answer. Using relevance and feasibility criteria and pilot testing, we developed a 44-item questionnaire. To assess the sensitivity and validity of the questionnaire, we recruited 220 patients over 18 years of age from three Korean hospitals.
Factor analysis of the data and fit statistics process resulted in the 4-factor, 32-item Quality Care Questionnaire-Palliative Care (QCQ-PC), which covers appropriate communication with health care professionals (ten items), discussing value of life and goals of care (nine items), support and counseling for needs of holistic care (seven items), and accessibility and sustainability of care (six items). All subscales and total scores showed a high internal consistency (Cronbach alpha range, 0.89 to 0.97). Multi-trait scaling analysis showed good convergent (0.568–0.995) and discriminant (0.472–0.869) validity. The correlation between the total and subscale scores of QCQ-PC and those of EORTC QLQ-C15-PAL, MQOL, SAT-SF, and DCS was obtained.
This study demonstrates that the QCQ-PC can be adopted to assess the quality of care in patients with cancer.
Despite advances in cancer therapy for cancer, most patients with cancer still suffer due to the disease itself, as well as its treatment [1,2,3]. Patients with cancer have fatigue, pain, loss of appetite, depression, and social and spiritual distress . Considering these various aspects, the definition of quality of life (QOL) in cancer patients is included personal insights of cancer patients’ symptoms, including physical, mental, social, and cognitive functions . As attention to QOL in oncologic care throughout the cancer trajectory has increased, palliative care is being better integrated into oncologic care at earlier stages, than in the past [1,2,3].
The World Health Organization (WHO) defines palliative care as “an approach that improves the QOL of patients and their families facing the problems associated with life-threatening illness, through the prevention and relief of suffering by means of early identification and impeccable assessment and treatment of pain and other problems, physical, psychosocial, and spiritual”. Recently, it was demonstrated that early palliative care resulted in not only a significant improvement in QOL [5,6,7], but also longer survival among patients with cancer [5, 8].
Because palliative care is one of the factors that influences the patient’s QOL, it is also important that the patient is provided with palliative care of good quality defined by two crucial, humane dimensions: high quality of care, and full acceptance of the patient’s intentions so that care is provided in a humane and culturally appropriate manner .
To design palliative care intervention strategies tailored to the patient, it is crucial to identify patients at a high risk for poor quality care by using a valid assessment tool. However, most instruments measure quality of care in patients at end of life (EOL) [10,11,12,13,14,15]. There are few assessment tools focusing on quality of care in patients with cancer receiving palliative care [16, 17], and there are limitations in that most of the current palliative care assessment tools did not include spiritual and cultural domains, and there are few tools to assess the EOL care experience reported by the patient . In addition, most of the palliative care assessment tools developed so far measure the QOL of the patient, not the quality of care [19,20,21,22,23].
Therefore, in the present study, we aimed to develop a new instrument to assess quality of palliative care and report on the validation of the developed tool, which is called Quality of Care Questionnaire-Palliative Care scale (QCQ-PC).
QCQ-PC includes four stages of development proposed by the European Organization for Research and Treatment of Cancer Quality of-Life Group : 1) item generation and reduction; 2) scale construction; 3) pilot testing; and 4) field testing. The study was approved by the Institutional Review Board of the Seoul National University Hospital, Korea. Suitable patients signed informed consent forms. The requirements for the recruitment of patients were as follows: (1) 18 years old or older; (2) diagnosed with cancer (cancer was confirmed by oncologists); (3) able to read and comprehend Korean, be able to fill in the questionnaire; and (4) aware of cancer diagnosis.
Phase I: Item generation and reduction
Phase I was intended to gather a list of relevant quality of palliative care questionnaires and previous palliative care clinical guidelines. We performed an extensive literature review using PUBMED and other databases searching the keywords ‘palliative care’, ‘end of life care’, and ‘quality of care’. We conclusively reviewed and adopted End of Life Quality Standards (2013) published by The National Institute for Health and Care Excellence (NICE) in the United Kingdom, Clinical Practice Guidelines for Quality Palliative Care (3rd Ed. 2013) released by National Consensus Project (NCP) in the United States and A National Framework and Preferred Practices for Palliative and Hospice Care Quality issued by National Quality Forum (NQF) in the United States.
We also considered the quality of care index of the prior study. We investigated and used the Assessment of Chronic Illness Care (ACIC) questionnaire and the Patient Assessment of Chronic Illness Care (PACIC) questionnaire translated by National Evidence-based Healthcare Collaborating Agency (NECA) in a previous study, which was a study of the performance evaluation methods of sub-categories in the management of chronic disease patients in a community by the primary health care center (2015). In addition, the FAMCARE Patient Scale developed by Kristjanson (1993) and verified with respect to validity and reliability was used. Based on the literature, we constructed a list of items for the QCQ-PC from 104 quality care issues divided into 14 scales.
We discussed these 104 items in semi-structured interviews with 31 health care professionals in November 2016 and December 2016. The health care professionals participating in the survey included physicians, nurses, and social workers who worked in university hospitals and were engaged in palliative care. The semi-structured interviews did not include patients because the items built by healthcare professionals were intended to be pilot-tested separately. The survey was conducted twice in total, and the respondents evaluated the validity and feasibility of the developed items by scoring them on a scale from 1 to 5. In the second investigation, the average score of all respondents for each item was compared with the score given by them, and the score changed when the respondent’s evaluation changed. The items that did not meet the following criteria in both the primary and secondary surveys were deleted (in the case of validity, if the respondent average score was three or more, or the respondent score was less than three points in less than 25% of the respondents; in the case of feasibility, the respondent average score is 2.5 or more, or less than three points in less than 30% of the total respondents). Thirteen items (12.5% of the developed items) that did not meet the criteria were deleted; therefore, a total of 91 items were included in the final index.
Phase II: Scale construction
Among the 91 palliative care quality evaluation indices developed through the process of item development and reduction, we constructed scales of 43 items that only the patients were asked to answer. The 48 items that were excluded were the items that directly asked the medical staff to evaluate the quality of care. For the scoring format, we selected a four-point Likert scale for all of 43 items to evaluate in phase III (Strongly Agree, Agree, Disagree, Strongly Disagree – see Additional file 1: Appendix B).
Phase III: Pilot testing
We conducted pilot tests to find possible administration problems, such as miss-phrasing, and to determine which items should be modified or removed. Before field testing, a total of 15 cancer patients responded to the QCQ-PC questionnaire and debriefed the questionnaire. The mean age of the 15 patients was 60.6 years, with 7 males and 8 females. Regarding educational level, most subjects had more than college or university education. All 15 patients were collected from Seoul National University Hospital. All 15 patients were collected from Seoul National University Hospital. The questionnaire designed for debriefing included which questions were misleading, whether the respondents were offended, or whether they were unable to respond. After the questionnaire, we received comments on the questionnaire. Taking into consideration the responses of these debriefing questionnaires, we clarified and simplified the items that were difficult to understand or answer. We also added one item to reflect the needs of the respondents.
Phase IV: Field testing
To confirm reliability and validity, we performed a field test with the QCQ-PC questionnaire.
Because the most patients group receiving palliative care are patients diagnosed with cancer, we performed this field testing for the QCQ-PC questionnaire to cancer patients. To obtain a heterogeneous sample by cancer type, patients were recruited to include various primary sites of cancer; these patients were registered at four university hospitals in Korea. We included patients under curative treatment as well as patients under palliative treatment. This study used factor analysis with varimax rotation to analyze construct validity, and performed multi-trait scaling analysis to examine the extent to which QCQ-PC items could be combined into a more controlled multi-item set. For validation, we applied ten rules, for which at least five respondents were required per item [25, 26]. Since we had a total of 44 questions, 220 respondents were necessary for the validation process.
We investigated the correlation between each question and the scale that included that question, and estimated the convergent validity of the QCQ-PC items. In verifying discriminant validity, we assessed the magnitude of the correlation of an item with its own scale as compared to other scales. We considered the convergent validity of items by analyzing correlations that were ≥0.4, and corrected for overlap, as confirmation of validity . We identified scaling errors as cases in which an item correlated significantly less with its own scale as compared to its correlation with other scales. To test the reliability of QCQ-PC, we estimated Cronbach alpha, a degree of internal consistency of responses. An alpha ≥0.70 was generally regarded as acceptably high for the collection of responses into a single score .
Furthermore, fit statistics (INFIT/OUTFIT) analysis was performed to evaluate the fitness of the item response, and it was interpreted as a good response when the fit was between 0.7 and 1.3 . The slope was checked to confirm the discrimination of the items. In the case of a fit value less than 1.5, the discrimination of the items was considered unsuitable (see Additional file 1: Appendix A).
We asked the respondents to fill in additional questionnaires on previous validated scales. The following questionnaires were included: The European Organization for Research and Treatment of Cancer Quality of Life Questionnaire Core-15 item palliative care (EORTC QLQ–C15-PAL) measuring patient-reported QoL , the McGill Quality of Life Questionnaire (MQOL) constructed by physical symptoms, psychological symptoms, existential well-being, and support , the Smart Management Strategy for Health Assessment Tool – Short Form (SAT-SF) for self-management strategies assessments , and the Decisional Conflict Scale (DCS) - assesses the level of ‘decisional conflict’ that patients experience while making health care decisions .
Data analyses were conducted using WINSTEP version 4.0 for item-fit analysis, and SPSS version 24.0 for other statistical analyses.
A total of 220 patients under curative or palliative therapy were enrolled in this study during the 2-month survey period. Most patients (96.4%) completed the questionnaire, but eight patients were excluded from the analysis because they did not answer more than three items. The remaining 212 patients included in the analysis responded to all questions.
The demographic characteristics of patients who participated in the study are shown in Table 1. There was no significant difference in gender ratio; more than half the patients were aged between 50 and 69 years old, and most patients were being treated.
We conducted factor analysis on the total sample of respondents (n = 220). We initially had four significant factors with 44 items; twelve items were discarded through the fit statistics process. Upon reanalysis of the remaining 32 items, the items were classified according to four significant factors. Table 2 lists the item-to-factor correlations for the 32 items and four factors – factor 1: appropriate communication with health care professionals (ten items); factor 2: discussing value of life and goals of care (nine items); factor 3: support and counseling for needs of holistic care (seven items); and factor 4: accessibility and sustainability of care (six items). We attained similar of factor analysis results with both multiple and simple imputations.
Table 3 shows the mean and reliability of the QCQ-PC subscale. In all four scales, the degree of reliability was high, with good internal consistency (Cronbach alpha range: 0.889–0.973).
Validity: Multi-trait scaling analysis
When item-to-self scale correlations were tested, all item-convergent validity exceeded 0.4. Other scales were also compared through item-discriminant validity, and no scaling error was observed.
We obtained the correlation between total and subscale scores of QCQ-PC and EORTC QLQ-C15-PAL and MQOL (see Table 4). Both total and subscale scores significantly correlated with the emotional functioning and global health scores of EORTC QLQ-C15-PAL (Pearson correlation [r] range 0.165–0.278). Existential well-being and support scores of MQOL significantly correlated with both the total and subscale scores of QCQ-PC (Pearson correlation [r] range 0.218–0.341, p < 0.001). However, total and subscale scores of QCQ-PC did not correlate with physical functioning.
Comparisons with SAT-SF (smart management strategy for health assessment tool – Short form) scores
SAT is a tool to identify self-management strategies of cancer patients. We used the short form of SAT to examine the association with the QCQ-PC score. As expected, there was a significant correlation between the SAT total score and the QCQ-PC score (Pearson correlation [r] range 0.285–0.372, p < 0.001). Compared to each subscale of SAT, significant correlation was observed (Pearson correlation [r] range 0.239–0.375, p < 0.001).
Comparisons with DCS
QCQ-PC scores showed correlation with the DCS total score and subscale score (Pearson correlation [r] range 0.325–0.397, 0.200–0.384). DCS is scored in the opposite direction; the higher the score, the higher the decision conflict.
The 32-item QCQ-PC is patient-reported and has excellent psychometric properties. The QCQ-PC consists of four factors that focus on communication with health care professionals, discussing value of life and goals of care, support and counseling for holistic care needs, and accessibility and continuity of care, all of which are central issues in palliative care. Our results are consistent with the findings of many previous studies and recommendations [1, 3, 6], but diverge in terms of quality of care at EOL [10,11,12,13,14,15]. The factors are unique and different from standard oncology care .
This study suggests that QCQ-PC has excellent psychometric properties, such as high construct validity and internal consistency. That QCQ-PC exhibited less correlation with EORTC QLQ-C15-PAL than with MQOL, and does not have significant correlation with physical functioning, suggesting that QCQ-PC more measures the emotional, social, and spiritual aspects of care in palliative settings rather than symptom control. Additionally, most correlations of QCQ-PC subscales with QOL subscales were below 0.4, suggesting that QCQ-PC may differentiate aspects of palliative care from those addressed by the QOL assessment tools. Therefore, QCQ-PC is recommended for use together with a QOL assessment tool, such as EORTC QLQ-C15-PAL, in order to more accurately to measure the quality of care.
In this study, QCQ-PC had significant negative correlation between the total and subscale scores and the DCS scores. Lack of communication with healthcare professionals and discussing value of life and goals of care may cause patient dissatisfaction with respect to care during cancer treatment, and lead patients to choose interventions that physicians believe are inappropriate. In addition, it is particularly crucial for healthcare providers to support and counsel on holistic care needs, to deliver continuous palliative care for patients with cancer, to maintain good interpersonal relationships with them, and to relieve their decisional conflict [1, 2, 6, 10]. We can improve their decision conflicts by recognizing high risk groups in quality care and providing palliative care.
Perhaps the most interesting finding of this study is that QCQ-PC appears to be sensitive to health management in palliative care settings. Using a conceptual model, self-health management with cancer can be influenced by quality care including communication with health care professionals, discussing value of life and goals of care, support and counseling for holistic care needs, and accessibility and continuity of care. Our findings showing that the QCQ-PC scales were associated with strategy scales for health management support this conceptual model.
This study has several limitations. First, since our research was conducted only in Korea, cross-cultural validation studies are necessary for generalizations to other countries. Second, we did not assess test-retest. As QOL and quality of care among cancer patients can frequently change, test-retest may not be feasible and may not limit the psychometric properties of this QCQ-PC. Finally, as this study relied exclusively on patients with cancer, our findings might not be generalizable to non-cancer patients who need palliative care. Furthermore, our study included not only incurable cancer patients, but also patients who had received treatment more than 5 years previously. However, to assess the quality of early palliative care as well, it may be preferable to include all cancer patients rather than only those with incurable cancer for constructing the questionnaire. We suggested that it is necessary to investigate further research on the patients of early palliative care and palliative care with this QCQ-PC questionnaire. Though, some modification of items among the QCQ-PC might be applied to patients with other advanced diseases.
In conclusion, we believe that this QCQ-PC, a self-reported assessment tool with proper psychometric properties, can be effectively used to identify patients with cancer at high risk, and to evaluate the efficacy of trials, such as palliative care with the QOL assessment tool.
Decisional conflict scale
End of life
- EORTC QLQ-C15-PAL:
European Organization for Research and Treatment of Cancer Quality of Life Questionnaire Core-15 item palliative care
The McGill quality of life questionnaire
Quality care questionnaire-palliative care
Quality of life
Smart management strategy for health assessment tool – short form
Greer JA, Jackson VA, Meier DE, Temel JS. Early integration of palliative care services with standard oncology care for patients with advanced cancer. CA Cancer J Clin. 2013;63(5):349–63.
Parikh RB, Kirch RA, Smith TJ, Temel JS. Early specialty palliative care—translating data in oncology into practice. N Engl J Med. 2013;369(24):2347–51.
Ferrell BR, Temel JS, Temin S, Alesi ER, Balboni TA, Basch EM, Firn JI, Paice JA, Peppercorn JM, Phillips T. Integration of palliative care into standard oncology care: American Society of Clinical Oncology clinical practice guideline update. J Clin Oncol. 2017;35(1):96–112.
Bottomley A. The cancer patient and quality of life. Oncologist. 2002;7(2):120–5.
Temel JS, Greer JA, Muzikansky A, Gallagher ER, Admane S, Jackson VA, Dahlin CM, Blinderman CD, Jacobsen J, Pirl WF. Early palliative care for patients with metastatic non–small-cell lung cancer. N Engl J Med. 2010;363(8):733–42.
Zimmermann C, Swami N, Krzyzanowska M, Hannon B, Leighl N, Oza A, Moore M, Rydall A, Rodin G, Tannock I. Early palliative care for patients with advanced cancer: a cluster-randomised controlled trial. Lancet. 2014;383(9930):1721–30.
Temel JS, Greer JA, El-Jawahri A, Pirl WF, Park ER, Jackson VA, Back AL, Kamdar M, Jacobsen J, Chittenden EH. Effects of early integrated palliative care in patients with lung and GI cancer: a randomized clinical trial. J Clin Oncol. 2016;35(8):834–41.
Bakitas MA, Tosteson TD, Li Z, Lyons KD, Hull JG, Li Z, Dionne-Odom JN, Frost J, Dragnev KH, Hegel MT. Early versus delayed initiation of concurrent palliative oncology care: patient outcomes in the ENABLE III randomized controlled trial. J Clin Oncol. 2015;33(13):1438–45.
Brook RH, McGlynn EA, Shekelle PG. Defining and measuring quality of care: a perspective from US researchers. Int J Qual Health Care. 2000;12(4):281–95.
Yun YH, Kim S-H, Lee K-M, Park SM, Lee CG, Choi YS, Lee WS, Kim S-Y, Heo DS. Patient-reported assessment of quality care at end of life: development and validation of quality care questionnaire–end of life (QCQ–EOL). Eur J Cancer. 2006;42(14):2310–7.
Mayland CR, Williams EM, Ellershaw JE. Assessing quality of care for the dying: the development and initial validation of a postal self-completion questionnaire for bereaved relatives. Palliat Med. 2012;26(7):897–907.
Heckel M, Bussmann S, Stiel S, Weber M, Ostgathe C. Validation of the German version of the quality of dying and death questionnaire for informal caregivers (QODD-D-Ang). J Pain Symptom Manag. 2015;50(3):402–13.
Mayland CR, Lees C, Germain A, Jack BA, Cox TF, Mason SR, West A, Ellershaw JE. Caring for those who die at home: the use and validation of ‘care of the dying evaluation’(CODE) with bereaved relatives. BMJ Support Palliat Care. 2014;4(2):167–74.
Mystakidou K, Tsilika E, Kouloulias V, Parpa E, Katsouda E, Kouvaris J, Vlahos L. The" palliative care quality of life instrument (PQLI)" in terminal cancer patients. Health Qual Life Outcomes. 2004;2(1):8.
Lo C, Burman D, Swami N, Gagliese L, Rodin G, Zimmermann C. Validation of the QUAL-EC for assessing quality of life in patients with advanced cancer. Eur J Cancer. 2011;47(4):554–60.
Shimizu M, Fujisawa D, Kurihara M, Sato K, Morita T, Kato M, Miyashita M. Validation study for the brief measure of quality of life and quality of care: a questionnaire for the National Random Sampling Hospital Survey. Am J Hosp Palliat Med. 2017;34(7):622–31. https://doi.org/10.1177/1049909117693576.
Gonçalves F. Initial assessment of patients without cognitive failure admitted in palliative care. Am J Hosp Palliat Med. 2014;31(1):33–7.
Aslakson RA, Dy SM, Wilson RF, Waldfogel J, Zhang A, Isenberg SR, Blair A, Sixon J, Lorenz KA, Robinson KA. Patient- and Caregiver-Reported Assessment Tools for Palliative Care: Summary of the 2017 Agency for Healthcare Research and Quality Technical Brief. J Pain Symptom Manage. 2017;54(6):961–72. e916
Zhang L, Wang N, Zhang J, Liu J, Luo Z, Sun W, Woo SM-L, Chen C, Zhang K, Miller AR. Cross-cultural verification of the EORTC QLQ-C15-PAL questionnaire in mainland China. Palliat Med. 2016;30(4):401–8.
Hu L, Li J, Wang X, Payne S, Chen Y, Mei Q. Prior study of cross-cultural validation of McGill quality-of-life questionnaire in mainland mandarin Chinese patients with cancer. Am J Hosp Palliat Med. 2015;32(7):709–14.
Zeng L, Bedard G, Cella D, Thavarajah N, Chen E, Zhang L, Bennett M, Peckham K, De Costa S, Beaumont JL. Preliminary results of the generation of a shortened quality-of-life assessment for patients with advanced cancer: the FACIT-pal-14. J Palliat Med. 2013;16(5):509–15.
Henoch I, Axelsson B, Bergman B. The assessment of quality of life at the end of life (AQEL) questionnaire: a brief but comprehensive instrument for use in patients with cancer in palliative care. Qual Life Res. 2010;19(5):739–50.
Hearn J, Higginson I. Development and validation of a core outcome measure for palliative care: the palliative care outcome scale. Palliative care core audit project advisory group. Qual Saf Health Care. 1999;8(4):219–27.
Group EQoL: Guidelines for developing questionnaire modules. In.: European Organisation for Research and Treatment of Cancer (EORTC), Brussels; 2002.
Hong S, You S. Developmental research: theory, method, design and statistical analysis. In: Handbook of complementary methods in education research; 2006. p. 207.
Cochran WG. Sampling techniques. Chichester: Wiley; 2007.
Fayers PM, Machin D. Quality of life: the assessment, analysis and interpretation of patient-reported outcomes. Chichester: Wiley; 2013.
Tavakol M, Dennick R. Making sense of Cronbach's alpha. Int J Med Educ. 2011;2:53–5.
Smith AB, Rush R, Fallowfield LJ, Velikova G, Sharpe M. Rasch fit statistics and sample size considerations for polytomous data. BMC Med Res Methodol. 2008;8(1):33.
Shin DW, Choi JE, Miyashita M, Choi JY, Kang J, Baik YJ, Mo HN, Park J, Kim H-J, Park EC. Cross-cultural application of the Korean version of the european organization for research and treatment of cancer quality of life questionnaire-Core 15-palliative care. J Pain Symptom Manag. 2011;41(2):478–84.
Cohen SR, Mount BM, Tomas JJ, Mount LF. Existential well-being is an important determinant of quality of life: evidence from the McGill quality of life questionnaire. Cancer. 1996;77(3):576–86.
Yun YH, Jung JY, Sim JA, Choi H, Lee JM, Noh DY, Han W, Park KJ, Jeong SY, Park JW. Patient-reported assessment of self-management strategies of health in cancer patients: development and validation of the smart management strategy for health assessment tool (SAT). Psycho-Oncol. 2015;24(12):1723–30.
O'Connor AM. Validation of a decisional conflict scale. Med Decis Mak. 1995;15(1):25–30.
This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: HC15C1391).
Availability of data and materials
The dataset can be obtained on request from the first author, Young Ho Yun (email@example.com).
Ethics approval and consent to participate
The patient data reported in this study were derived from a survey approved by the Institutional Review Board of Seoul National University Hospital, and patients signed appropriate informed consent forms (IRB number 1703–157-840).
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix A. Factor analysis (Item discrimination, slope, and factor loading) and fit statistics (INFIT/OUTFIT) analysis; Table describing the result of factor analysis and fit statistics of 44 items. Appendix B. Quality Care Questionnaire –Palliative Care (QCQ-PC); Palliative care quality questionnaire validated in this paper. (DOCX 50 kb)