Tackling child poverty pathfinders: evaluability assessment

Appendix 4 – Feasibility of theory-based evaluation methods for Child Poverty Pathfinders

A theory-based approach to evaluating the Pathfinders is feasible and well suited to this type of intervention. The Pathfinders are operating in complex environments, where it is unlikely impacts may be attributed to one programme or intervention. Contribution analysis (CA), specifically, is an ideal theory-based evaluation methodology to assess the impact of complex programmes – where the focus is on assessing the degree to which a programme's interventions can be said to have contributed to (rather than have definitively caused) outcomes and impact.

"[W]ithin contribution analysis, a plausible narrative is considered to have been developed when four different conditions are met (Mayne 2008).

1. The ...intervention is based on a sound Theory of Change, accompanied by agreed and plausible assumptions, that explains how the intervention sought to bring about any desired changes.

2. The activities of the … intervention were implemented properly.

3. There is adequate evidence showing that change occurred at each level of the Theory of Change.

4. The relative contribution of external factors or other development interventions can be dismissed or demonstrated." (INTRAC, 2017)

As part of the commission, we have co-designed Theories of Change with assumptions and risks with the Pathfinders, and developed detailed M&E frameworks based on data audit exercises undertaken with the Pathfinders (see ToC and MEL report), which will enable evaluators to examine the data collated by the Pathfinders (and collect further data) and to assess the evidence at each link in the Theories of Change, and develop a contribution story. There is also an existing body of evidence underpinning the Theories of Change from previous evaluations, which have tested elements of the Theories of Change including the 'support models' used by the Glasgow and Dundee Pathfinder (see below), which suggests that key elements of the Theories of Change are sound. The process study planned will also provide useful information on the implementation aspects of the Pathfinders.

We have discussed the potential of using CA here as part of a summative evaluation. However, as discussed above it may also be used to guide developmental evaluation. Developmental evaluation is well-suited to interventions in complex settings. In Developmental Evaluation, evaluation occurs at regular intervals, such that learning is continuously fed back into programme delivery, allowing it to improve, adapt and respond, a critical aspect of the Pathfinders. We have developed detailed M&E frameworks for the Pathfinders which will enable them to track change over time. They have already started establishing M&E and data systems (see ToC and MEL report) and the M&E framework will support them to continue establishing their systems. Evaluation expertise is often required to set up M&E frameworks and provide early guidance. However once evaluative thinking becomes normative within the delivery team, developmental evaluation can be embedded without the need for evaluative expertise.^[191]

Given a key aim of the Pathfinders is to gather learning and evidence of what is working to support national efforts to reduce child poverty at scale, an evaluation approach that places emphasis on learning seems well-suited. Developmental evaluation though time intensive may be a cost-effective option in the short-term. In any case, any investment on developing robust M&E processes by the Pathfinders in the implementation phase will not only support learning, but also the ability of the Pathfinders to adapt and improve. Furthermore, the data and evidence collated by the Pathfinders will also, of course, enable a more robust and useful summative evaluation to be undertaken.

Existing evidence on the approaches taken by the Pathfinders

There is a reasonable body of evidence on the effectiveness of key features of both the Glasgow and Dundee pathfinder. The key features which the literature emphasises as crucial for addressing poverty is offering personalised, holistic, and whole-family support. The Supporting Families framework, outlines that strong local partnerships that can identify and support families in need seamlessly across a range of services will be critical for addressing child poverty.^[192] Similarly, a review of best practice in this area conducted by the government of Northern Ireland concluded that successful interventions were characterised by: parental engagement, targeted approaches, harnessing existing resources, and holistic services.^[193]

There is also previous research that suggests that having one access-point to several services can help expand individuals access to a variety of services.^[194] In their research, the Joseph Rowntree Foundation, highlights that children and family service should be combined and delivered together wherever possible, to ensure that the whole family is supported.^[195] Moreover, echoing Glasgow Pathfinder's approach, they also recommend co-ordinating local services so that families can access the support they need without being passed around between services.^[196] Similarly, this is reflected in the Supporting Families which advocates for a single-access point for families as a gateway to holistic and co-ordinated services.^[197]

Previous research has also shown that the key worker model is effective in delivering direct and holistic support, especially for families and children with several and complex needs.^[198] Although not solely focused on child poverty, the Institute of Health Visiting states that an integrated whole-systems approach is necessary for ensuring that a child has the best start to life. This is because often children and their families need support from a range of services. It also suggests that this also allows for 'trusting relationships' to be formed between the key worker and the family. "Across the children's sector, there is increasing acknowledgement that meeting the complex needs of children, … , means focusing on whole-system approach to understanding and mitigating risk in a young person's wider environment."^[199] Similarly, the identification and service delivery through home visits, can also be important in reducing poverty and its impact.^[200] This is especially true if this occurs within the first few years of a child's life.^[201] This is also reflected in the Supporting Families Framework, which advocates that vulnerable families need to be identified by local service providers and supported, so that no family is left behind and it also reinforces a preventative approach.^[202] Moreover, the outreach approach taken to approach families has been shown to be important in reaching families who may not know that support is available or are not comfortable accessing services by themselves.^[203]