Table 3 Process evaluation methods based on UK MRC guidance on process evaluations of complex interventions (Moore et al. 2012)
Dimension (definition*) | Subdimension (definition*) | Measurements | Data collection method (qualitative or quantitative; timing) |
|---|---|---|---|
Implementation** (the process through which interventions are delivered, and what is delivered in practice) | HOW delivery is achieved (implementation process: structures, resources and mechanisms through which delivery is achieved) | - Resources: time spent by trainer on preparation and delivery of intervention - Resources: total trial cost associated with delivery of intervention (printing cost training materials, salary trainers, rent training locations and catering) - Implementation process of all ACP+ activities | - Structured diary filled in by trainers (quantitative; weekly) - Expenses from researchers and trainers (quantitative; continuous) - Semi-structured interviews with trainers (qualitative; every 4 months) - Semi-structured group interviews with ACP reference persons per IF (qualitative; after T1) |
WHAT is delivered (the quantity and quality of what is delivered) 1) Dose (how much intervention is delivered) | - Number and type of intervention activities‡ delivered in each IF | - Structured diary filled in by trainers (quantitative; weekly) | |
2) Reach†(the extent to which a target audience comes into contact with the intervention) | - Number of ACP Reference Persons of each IF attending two-day training /total number of staff in each IF - Attendance rate of staff during in-house training sessions (for ACP Conversation Facilitators and ACP Antennas) in each IF/ total number of staff in each IF - Number of residents informed about ACP in each IF/total number of residents at T0 in each IF - Number of residents for whom a family member is informed about ACP in each IF/total number of residents at T0 in each IF - Number of GPs informed about ACP in each IF/total number of GPs at T0 in each IF - Number of volunteers informed in each IF/total number of volunteers at T0 in each IF - Number of residents or family members of residents offered minimum one ACP conversation/total number of residents at T0 in each IF - Number of residents with an advance directive/total number of residents at T0 in each IF | - Attendance lists (quantitative; before start of each training or information session) - Survey about number of residents, family and volunteers informed, to be filled in by key contact person in IF (quantitative; after month 6 and at the end) - Information provided by key contact person in IF, based on ACP+ registry document (quantitative; continuous) - Facility level data (quantitative; T1) | |
3) Fidelity (the consistency of what is implemented with the planned intervention) | - Number of activities delivered as intended (dose delivered as intended) in each IF/total number of activities - Type of activities delivered, according to participating staff - Content and quality of training workshops for ACP Reference Persons delivered as intended, as observed by researchers - Number of ACP Reference Persons per IF that attended training session scored high on fidelity/total number of care staff at T0 in each IF - Median score of trainer competencies for each training (across and in each IF) - Median score of quality of each training (across and in each IF) | - Structured diary filled in by trainers (weekly) - Semi-structured interview with trainers (qualitative; every 4 months) - Two post-intervention focus group with trained staff across IF (qualitative; after T1) - Semi-structured group interview with ACP Reference Persons in each IF (qualitative; after T1) - Observation of two-day training for ACP Reference Persons by researchers, using checklist of minimum requirements and overall rating of fidelity and quality (quantitative) - Attendance list (quantitative; at each ACP+ training) - Post training survey for participants (quantitative; after each ACP+ training) | |
4) Adaptations (alterations made to an intervention in order to achieve better contextual fit) | - Adaptations made to activities of the ACP+ activities (e.g. number, duration, content), according to trainers and Trial Monitor - Experiences with of participants regarding adaptations made and the contextual fit of activities of the ACP+ programme | - Semi-structured interview trainers (qualitative; every 4 months) - Semi-structured group interview with ACP Reference Persons in each IF (qualitative; after T1) - Notes made by Trial Monitor based on communication with trainers and IFs | |
Mechanisms of impact (the intermediate mechanisms through which intervention activities produce intended (or unintended) effects) | Responses and interactions (how participants interact with the intervention) | - Staff experiences with and views with regard to the ACP+ intervention and activities | - Semi-structured interview with one manager per IF (qualitative; after T1) - Two post-intervention focus group with trained staff across IF (qualitative; after T1) - Semi-structured group interview with ACP Reference Persons in each IF (qualitative; after T1) |
Mediators (intermediate processes which explain subsequent changes in outcomes) | - Evaluation of perceived mediators (or preconditions 1, 2, 6, 7 and interventions 1, 2, 3A, 3B, 4A, 4B, 4C, 6A, 6B, 8 in Theory of Change map [10]), as intermediate processes that might explain changes in outcomes. | - Semi-structured interview with one manager per IF (qualitative; after T1) - Two post-intervention focus group with trained staff across IF (qualitative; after T1) - Semi-structured group interview with ACP Reference Persons in each IF (qualitative; after T1) | |
Unanticipated pathways or consequences†| - Potential unanticipated consequences of the ACP+ programme in residents and/or family, in staff, in GP according to participants | - Semi-structured interview with one manager per IF (qualitative; after T1) - Two post-intervention focus groups with trained staff across IF (qualitative; after T1) - One post-intervention focus group with ACP Reference Persons across Ifs (qualitative; after T1) - Three semi-structured interviews with residents and family in each IFs (qualitative; after T1) | |
Context (factors external to the intervention which may influence its implementation, or whether its mechanisms of impact act as intended) | Contextual moderators†potentially inhibiting or facilitating the implementation, organisation, sustainability and outcomes of ACP | - Contextual barriers and facilitators for 1) implementation (‘the process through which interventions are delivered, and what is delivered in practice’), according to participants - Contextual barriers and facilitators for 2) sustainability (‘the potential for an intervention to become part of routine practice’), according to participants - Contextual barriers and facilitators for 3) outcomes (knowledge, attitudes, self-efficacy and practice), according to participants | - Semi-structured interview with one manager per IF (qualitative; after T1) - Two post-intervention focus groups with trained staff across IF (qualitative; after T1) - Semi-structured group interview with ACP Reference Persons in each IF (qualitative; after T1) - Semi-structured interviews trainers (qualitative; every 4 months) |
Intention for Maintenance†(extent to which the programme is intended to be part of routine organisational practice and policy) | - Staff’s intention for performing ACP+ activities in the future - Organisational intention for long-term implementation - Participants’ recommendations for improving sustainability | - Semi-structured interview with one manager per IF (qualitative; after T1) - Semi-structured group interview with ACP Reference Persons in each IF (qualitative; after T1) |