Publication - Research and analysis

Tackling child poverty pathfinders: evaluability assessment

Published: 19 May 2023
Directorate: Tackling Child Poverty and Social Justice Directorate
Part of: Children and families, Communities and third sector, Equality and rights
ISBN: 9781805258254

An evaluability assessment of the Child Poverty Pathfinders in Glasgow and Dundee to inform the development of an evaluation plan for the Pathfinder approach. Includes an evaluability assessment report and accompanying theories of change and initial monitoring framework to support evaluation.

Supporting documents

Evaluation Plan

Conclusions and recommendations

1. What is a Child Poverty Pathfinder?

No single definition of a Pathfinder has been established yet, and the two models in operation have little in common.

There are a number of key features which apply to Pathfinders, such as being: (i) designed to drive systems change; (ii) flexible and adaptable in how they are implemented; (iii) a person-centred approach; (iv) place-based and co-designed in partnership with local organisations; and (v) a holistic support service.

As part of this commission, a ToC has been developed for each Pathfinder to understand the causal pathways and the processes of change, as well as to make explicit assumptions about how change happens. A M&E framework has also been developed for each Pathfinder to support the process of learning and improvement within each Pathfinder as well as an external evaluation.

Recommendation 1.1: There would be benefits from improving the common understanding of 'what' the Pathfinder programme is, and how the overall programme, Glasgow Pathfinder and Dundee Pathfinder are connected. A suggestion for this is provided in section 2.

Recommendation 1.2: The differences between the design of the Dundee and Glasgow models should be capitalised on for learning on what works.

2. What factors need to be considered before evaluating the Pathfinders?

There is a lack of shared understanding of the definition of the Pathfinder programme. it will be difficult to translate evaluation findings into informing the development of the Pathfinder programme unless there is a clearer sense of what is meant by "a Pathfinder". We have provided a suggested definition in this report.

This issue is further compounded by the current lack of understanding around what Scottish Government's role in the Pathfinders is or should be.

The above points means that at present it may not be feasible to evaluate through a programme-level lens. Therefore, for the purposes of this evaluability assessment, the evaluand should be the two Local Child Poverty Pathfinders (rather than 'the programme').

There are concurrent child poverty interventions and an existing evaluation of the Glasgow Pathfinder. These could be used to leverage the findings from an evaluation of the Pathfinders but may also create additional complexities or give rise to duplicative efforts.

Recommendation 2.1: Use the suggested Pathfinder definition provided in this report as a basis for an agreed definition of a Pathfinder.

Recommendation 2.2: Agree on the role that Scottish Government should play in the continuation of the Pathfinders.

Recommendation 2.3: The evaluand should be the individual Pathfinder models and the evaluation should focus on gathering learning from these.

Recommendation 2.4: The effect of concurrent child poverty interventions – in particular the WFWF – will need consideration. The evaluation should consider whether families may simultaneously benefit from these interventions as well as the Pathfinder, and account for this when estimating the Pathfinders' impact.

Recommendation 2.5: The existing evaluation of the Glasgow Pathfinder should be taken into account when conducting the Phase 2 evaluation, both to avoid duplication, and to build further on those findings.

3. What ethical framework should guide an evaluation?

Given the sensitivity of the topic area and vulnerability of families in poverty, the ethical considerations around research and evaluation are important considerations in how programmes are evaluated.

Recommendation 3.1: The ethical framework developed in appendix 3 should guide the evaluative process.

4. What key aims should an evaluation focus on?

We have suggested four aims for the evaluation. These are to evaluate (i) the impact on families, child poverty, and the system that supports them; (ii) the value for money; (iii) the processes of the Pathfinder; and (iv) the learning that can be captured.

These are presented separately but have inherent overlap between them.

We have identified a set of research questions associated with the above aims, which are provided as a longlist of detailed evaluation questions in the evaluation aims and methods section below.

Recommendation 4.1: When conducting the evaluation, the interlinkages between the aims should be taken into account, both in terms of how they are evaluated, and the conclusions drawn.

Recommendation 4.2: The evaluation questions should be further refined to align with the methodology chosen to evaluate the Pathfinders.

Recommendation 4.3: The questions should not only focus on the 'difference' the Pathfinders have made in terms of impact, but also on the processes of change. For this, the Theories of Change of the Pathfinders are of central importance in this evaluation.

5. What are the methodological options for meeting these aims?

We have identified the key methodological options for the four evaluation aims.

Impact on families, child poverty and the system that supports them could be evaluated through quasi-experimental studies, a theory-based approach using contribution analysis, or a learning partner approach with a developmental evaluation. The latter two represent clearly viable options, but with the trade-off that they cannot provide a statistically robust measure of the impact on child poverty itself. Quasi-experimental approaches, on the other hand, can achieve this, but would benefit from an initial feasibility study to test their efficacy.

For evaluating value for money, the main options would be CEA or social CBA, with the latter being preferrable. Robustly conducting an economic evaluation will require specific data to be available, comprehensive identification of the outputs/outcomes and costs of the programme, and financial proxies to quantify non-monetary outcomes. We provide suggestions for this in appendices 5 and 6. Identifying the exact benefits included in a CBA can be tied with the impact evaluation approach.

A process evaluation will draw on the same ToC and data as the impact aspects of the evaluation, but will build on these to answer different questions around 'how' the outcomes were achieved.

The impact and process evaluations should together draw evidence in relation to the extent to which the Pathfinders have created systems change.

Recommendation 5.1: If a quasi-experimental approach to evaluating is desired, an initial feasibility study would improve the efficacy of this. However, running a feasibility study followed by full roll-out of the approach would require longer than the current Phase 2 evaluation timelines. In Dundee, we recommend a difference-in-difference approach, and an interrupted time series analysis (ITSA) for Glasgow.

Recommendation 5.2: Whether or not a quasi-experimental evaluation is adopted, a learning partner should be appointed to assess impact (using a developmental evaluation approach). If a quasi-experimental approach is adopted, then either a contribution analysis theory-based evaluation should also be used to evaluate impact, or qualitative-based case studies should be included as evidence alongside the quasi-experimental approach.

Recommendation 5.3: Child poverty is complex and depends on a wide range of different factors which may be beyond the scope of the Pathfinders' intervention, particularly given the relatively small scale of the programme. Therefore, the impact evaluation should not rely wholly on measuring changes in absolute or relative child poverty figures. Instead, it should assess impact based on other factors which directly relate to poverty, such as changes in income, as well as wider positive outcomes.

Recommendation 5.4: Social cost benefit analysis should be used to evaluate the Pathfinders' value for money.

Recommendation 5.5: The process evaluation should be linked with the evidence on impacts on child poverty and systems change to understand 'how' these were achieved.

6. How can learning be assessed and integrated

Learning is of central importance in the evaluation. The evaluation will seek to extract learning to support Scottish Government's decisions about the continuation and expansion of the Pathfinders.

Learning must be captured from the Pathfinders to ensure wider impacts on child poverty are achieved. The ToCs and M&E frameworks for the Pathfinders designed as part of this commission which will provide a strong foundation for doing this.

The data and evidence collated by the Pathfinders will also enable a more robust and useful summative evaluation to be undertaken.

Recommendation 6.1: The Pathfinders should be sufficiently supported and resourced to maintain robust MEL processes, to ensure learning is effectively captured to improve delivery and to inform Scottish Government decision-making processes.

7. What is the best way of procuring an evaluation?

The different aspects of the evaluation will be able to begin at various times, and so a staged approach will likely be necessary.

A solid procurement process will ensure that the right evaluation partners are commissioned and that the Pathfinder staff and stakeholders are engaged.

Recommendation 7.1: A quasi-experimental evaluation would require an extension of the Phase 2 timelines in order to allow time for a feasibility study, evaluation set up, observation period, and reporting period.

Recommendation 7.2: To allow time for impacts to be realised, a theory-based evaluation may also require slightly longer than the current Phase 2 timelines.

Recommendation 7.3: The Dundee and Glasgow Pathfinders should be evaluated by the same organisation but following different approaches where relevant and as set out in this report.

Evaluation aims and methods

Pulling together the evaluation aims and methods, the table below summarises the techniques to be used in relation to each aim. In addition, it sets out the evaluation questions we are proposing within each aim. These questions are a more specific guide to the individual parts of the evaluation, and when answered should mean that the aim has been achieved.

Evaluation plan

Aim 1 Impact on families, child poverty and the system that supports them

Method

Theory-based approaches using contribution analysis

Management and performance data analysis, focussing on families' financial situation, and other outcomes which relate to poverty
Engagement with families to collect qualitative information
Engagement with stakeholders, partners and Pathfinder staff through interviews and focus groups
Longitudinal survey with Pathfinder staff
Review feedback from supported families

Embedded monitoring using a learning partner and developmental evaluation approach

Set up monitoring systems to enable data to be collected against the ToC.
Set up a learning partnership for the Pathfinders to facilitate experimentation, adaptation and improvement to support a process of systems change.

Quasi-experimental methods applied separately for Dundee (difference-in-difference) and Glasgow (ITSA), potentially paired with case studies

1A: Who received support from the Pathfinder – scale and demographics?

1B: To what extent were families in need successfully identified, engaged and supported – are there areas of unmet need?

1C: What was the impact on the finances, employment, resilience, health, and wellbeing of families?

1D: What was the impact on the families' confidence to manage future challenges?

1E: What was the impact on child poverty?

1F: To what extent does the support provided prevent crisis within families?

1G: To what extent are families obtaining and sustaining high quality and fair employment?

1H: To what extent are agencies working in partnership beyond organisational boundaries?

1I: To what extent is the support provided to families more flexible, holistic, targeted to need, and accessible?

1J: To what extent is the right support available for families – are there any gaps in resource, partners or services leading to unmet need?

1K: To what extent are families accessing support before crisis point – prevention? (Glasgow)

1L: To what extent are employers providing more fair, flexible work locally that is more accessible for families in poverty? (Dundee)

Aim 2 Value for money

Method

Social cost benefit analysis

2A: Do the impacts measured lead to any reduction in demand for public services?

2B: Do the impacts measured contribute to any impacts on the wider economy?

2C: Does the value of benefits outweigh the costs?

Aim 3 Process evaluation

Method

Longitudinal surveys with Pathfinder staff, stakeholders, and partners
Interviews and focus groups with stakeholders and partners

3A: How well do the Pathfinders fit within, and work with, the existing service landscape?

3B: How are local needs being identified, understood and used to inform service design and delivery?

3C: What mechanisms are in place to learn from what is being delivered? How is the learning shared within and across places?

3D: To what extent are decisions being informed by evidence on what works and the ongoing learning from the Pathfinder?

3E: To what extent is support co-produced with families?

3F: To what extent is the required data, learning and insight shared between partners?

3G: To what extent are public and third sector organisations working successfully together in partnership?

3H: To what extent are resources between partners pooled successfully where needed?

3I: To what extent are the most successful approaches scaled up?

3J: To what extent are partners working well together including shared vision, trust, governance, communication, and effective delivery?

3K: What is the cost effectiveness of the service – how does it compare with other similar forms of support?

3L: How accessible are the Pathfinders to those who benefit from them?

3M: Do the Pathfinders' designs ensure equality in how they are accessed and the support provided?

Aim 4 Learning

Method

Combine findings from previous stages
Hold group discussions or workshops with stakeholders and Scottish Government staff

4A: What have we learned about what works in addressing Child Poverty?

4B: What have we learned about what needs to be in place to support implementation?

4C: What have we learned about the challenges and how to overcome them to support implementation?

Evaluation plan data

In one of our evaluability assessment workshops, we combined the data sources identified in the M&E framework and the above evaluation methods to map out what data is available for each element of the evaluation, as well as what might be missing. This is summarised in the tables overleaf.

Dundee data for evaluation plan

Evaluation aim: Impact on families, child poverty and the system that supports them

Method: Management and performance data analysis, focussing on families' financial situation, and other outcomes which relate to poverty

Available data

Better-off calculations
Social Security awards from Pathfinders
Client spreadsheet
Fuel well data
Case management data
Housing benefit and council tax reduction data
Universal credit accounts