Skip to main content

Table 3 Process evaluation methods based on UK MRC guidance on process evaluations of complex interventions (Moore et al. 2012)

From: Implementing the theory-based advance care planning ACP+ programme for nursing homes: study protocol for a cluster randomised controlled trial and process evaluation

Dimension (definition*)Subdimension (definition*)MeasurementsData collection method (qualitative or quantitative; timing)
Implementation** (the process through which interventions are delivered, and what is delivered in practice)HOW delivery is achieved (implementation process: structures, resources and mechanisms through which delivery is achieved)- Resources: time spent by trainer on preparation and delivery of intervention
- Resources: total trial cost associated with delivery of intervention (printing cost training materials, salary trainers, rent training locations and catering)
- Implementation process of all ACP+ activities
- Structured diary filled in by trainers (quantitative; weekly)
- Expenses from researchers and trainers (quantitative; continuous)
- Semi-structured interviews with trainers (qualitative; every 4 months)
- Semi-structured group interviews with ACP reference persons per IF (qualitative; after T1)
WHAT is delivered (the quantity and quality of what is delivered)
1) Dose (how much intervention is delivered)
- Number and type of intervention activities‡ delivered in each IF- Structured diary filled in by trainers (quantitative; weekly)
2) Reach† (the extent to which a target audience comes into contact with the intervention)- Number of ACP Reference Persons of each IF attending two-day training /total number of staff in each IF
- Attendance rate of staff during in-house training sessions (for ACP Conversation Facilitators and ACP Antennas) in each IF/ total number of staff in each IF
- Number of residents informed about ACP in each IF/total number of residents at T0 in each IF
- Number of residents for whom a family member is informed about ACP in each IF/total number of residents at T0 in each IF
- Number of GPs informed about ACP in each IF/total number of GPs at T0 in each IF
- Number of volunteers informed in each IF/total number of volunteers at T0 in each IF
- Number of residents or family members of residents offered minimum one ACP conversation/total number of residents at T0 in each IF
- Number of residents with an advance directive/total number of residents at T0 in each IF
- Attendance lists (quantitative; before start of each training or information session)
- Survey about number of residents, family and volunteers informed, to be filled in by key contact person in IF (quantitative; after month 6 and at the end)
- Information provided by key contact person in IF, based on ACP+ registry document (quantitative; continuous)
- Facility level data (quantitative; T1)
3) Fidelity (the consistency of what is implemented with the planned intervention)- Number of activities delivered as intended (dose delivered as intended) in each IF/total number of activities
- Type of activities delivered, according to participating staff
- Content and quality of training workshops for ACP Reference Persons delivered as intended, as observed by researchers
- Number of ACP Reference Persons per IF that attended training session scored high on fidelity/total number of care staff at T0 in each IF
- Median score of trainer competencies for each training (across and in each IF)
- Median score of quality of each training (across and in each IF)
- Structured diary filled in by trainers (weekly)
- Semi-structured interview with trainers (qualitative; every 4 months)
- Two post-intervention focus group with trained staff across IF (qualitative; after T1)
- Semi-structured group interview with ACP Reference Persons in each IF (qualitative; after T1)
- Observation of two-day training for ACP Reference Persons by researchers, using checklist of minimum requirements and overall rating of fidelity and quality (quantitative)
- Attendance list (quantitative; at each ACP+ training)
- Post training survey for participants (quantitative; after each ACP+ training)
4) Adaptations (alterations made to an intervention in order to achieve better contextual fit)- Adaptations made to activities of the ACP+ activities (e.g. number, duration, content), according to trainers and Trial Monitor
- Experiences with of participants regarding adaptations made and the contextual fit of activities of the ACP+ programme
- Semi-structured interview trainers (qualitative; every 4 months)
- Semi-structured group interview with ACP Reference Persons in each IF (qualitative; after T1)
- Notes made by Trial Monitor based on communication with trainers and IFs
Mechanisms of impact (the intermediate mechanisms through which intervention activities produce intended (or unintended) effects)Responses and interactions (how participants interact with the intervention)- Staff experiences with and views with regard to the ACP+ intervention and activities- Semi-structured interview with one manager per IF (qualitative; after T1)
- Two post-intervention focus group with trained staff across IF (qualitative; after T1)
- Semi-structured group interview with ACP Reference Persons in each IF (qualitative; after T1)
Mediators (intermediate processes which explain subsequent changes in outcomes)- Evaluation of perceived mediators (or preconditions 1, 2, 6, 7 and interventions 1, 2, 3A, 3B, 4A, 4B, 4C, 6A, 6B, 8 in Theory of Change map [10]), as intermediate processes that might explain changes in outcomes.- Semi-structured interview with one manager per IF (qualitative; after T1)
- Two post-intervention focus group with trained staff across IF (qualitative; after T1)
- Semi-structured group interview with ACP Reference Persons in each IF (qualitative; after T1)
Unanticipated pathways or consequences†- Potential unanticipated consequences of the ACP+ programme in residents and/or family, in staff, in GP according to participants- Semi-structured interview with one manager per IF (qualitative; after T1)
- Two post-intervention focus groups with trained staff across IF (qualitative; after T1)
- One post-intervention focus group with ACP Reference Persons across Ifs (qualitative; after T1)
- Three semi-structured interviews with residents and family in each IFs (qualitative; after T1)
Context (factors external to the intervention which may influence its implementation, or whether its mechanisms of impact act as intended)Contextual moderators† potentially inhibiting or facilitating the implementation, organisation, sustainability and outcomes of ACP- Contextual barriers and facilitators for 1) implementation (‘the process through which interventions are delivered, and what is delivered in practice’), according to participants
- Contextual barriers and facilitators for 2) sustainability (‘the potential for an intervention to become part of routine practice’), according to participants
- Contextual barriers and facilitators for 3) outcomes (knowledge, attitudes, self-efficacy and practice), according to participants
- Semi-structured interview with one manager per IF (qualitative; after T1)
- Two post-intervention focus groups with trained staff across IF (qualitative; after T1)
- Semi-structured group interview with ACP Reference Persons in each IF (qualitative; after T1)
- Semi-structured interviews trainers (qualitative; every 4 months)
Intention for Maintenance† (extent to which the programme is intended to be part of routine organisational practice and policy)- Staff’s intention for performing ACP+ activities in the future
- Organisational intention for long-term implementation
- Participants’ recommendations for improving sustainability
- Semi-structured interview with one manager per IF (qualitative; after T1)
- Semi-structured group interview with ACP Reference Persons in each IF (qualitative; after T1)
  1. ACP advance care planning; IF intervention facility; GPs general practitioners
  2. Types of training activities: 1) Two-day training for ACP Reference Persons (delivered by ACP Trainer), across all intervention nursing homes; 2) Two training sessions of each two hours for ACP Conversation Facilitators (delivered by ACP Reference Persons, supported by ACP Trainer), in-house; 2) One training session of 1,5 h for ACP Antennas (delivered by ACP Reference Persons, supported by ACP Trainer), in-house
  3. *Definition by the MRC Framework by Moore et al. (2012)
  4. **The term implementation is used within complex intervention literature to describe both post-evaluation scale-up (i.e. the ‘development-evaluation-implementation’ process) and intervention delivery during the evaluation period. Within this document, discussion of implementation relates primarily to the second of these definitions (i.e. the quality and quantity of what is actually delivered during the evaluation)
  5. †Added by the research team