Evaluating evolving and complex programmes: learning and reflections from the child poverty pathfinders' evaluation

Key challenges and trade-offs

All evaluations involve trade-offs between time, resources, and robustness of outputs. However, evaluating programmes that aim to tackle urgent and complex issues, like child poverty, can create particular tensions between what might be deemed the ‘ideal’ evaluation approach and policy development and/or programme delivery priorities. The Table below highlights five key trade-offs encountered on the child poverty pathfinder phase 2 evaluation. These are by no means unique to this evaluation – indeed, the Table draws on one produced by Ipsos colleagues reflecting on other child poverty-related evaluations.

The remainder of this section elaborates on these trade-offs and tensions in more detail, drawing on examples from the child poverty pathfinders’ evaluation where relevant. It does so intentionally briefly: further discussion on most of these issues is included in the main evaluation report, for readers interested in more detail.

The second half of the paper offers suggestions on how to navigate these trade-offs on future evaluations of projects sharing similar features with the child poverty pathfinders (e.g. place-based, evolving, complex, system change elements).

Table 1: Key trade-offs in evaluating complex, iterative programmes

Delivery

Policy development and programme delivery often needs to happen fast, reflecting timescales for funding cycles and policy decisions

Evaluation

Sufficient time for planning an evaluation typically supports more sensitive / robust evaluation. This includes time to consider what data needs to be collected by delivery partners and, where relevant, planning for accessing administrative data from external organisations.

Delivery

Programme delivery needs to be responsive and evolve over time, and to be adapted based on learning about what works

Evaluation

Programme consistency, both over time and across areas, facilitates monitoring and outcome measurement, by enabling tracking of key outcomes from pre- to post-intervention and robust comparisons of these outcomes between areas.

Delivery

Requests to take part in evaluation (and to collect and provide data) place burden on service providers and users

Evaluation

Input from service providers and service users is needed for meaningful evaluation, particularly where secondary data is not (yet) available

Delivery

Vulnerable families are often more likely to engage in evaluation activities when selected/asked by service providers

Evaluation

Evaluation is often less prone to bias when external evaluators can control sampling and recruitment

Delivery

There is a strong need to understand the value for money of public investments.

Evaluation

The most robust forms of value for money (VfM) analysis require robust, quantitative impact data, which is not always available, particularly early on. Further, there is no established process for assessing the VfM of ‘system change’.

Timing of delivery, impact and evaluation

When projects are aiming to tackle urgent policy priorities, like child poverty, there is a strong impetus for delivery to happen at pace. Funding cycles, policy decision-making timelines, and senior stakeholder expectations can all mean that delivery may need to start before key elements of the evaluation activities are fully designed. In particular, a desire to take immediate action can mean that delivery begins before the key outcomes are decided and before data collection processes are implemented in order to track whether the programme leads to any measurable change.

At the same time, there is a recognition that meaningful change – particularly where the target is system change – may take considerable time. However, due to short-term budget cycles in government, public sector evaluations tend to be commissioned for fixed time periods that may cover only part of the programme implementation and outcome timeline. Aligning design, delivery and evaluation timelines can thus be a significant challenge. This was certainly the case with the pathfinders’ evaluation: delivery had already begun in both Dundee and Glasgow by the time the Scottish Government commissioned external evaluations, and there were a number of challenges around data collection and its availability to support the evaluation, as discussed in the main evaluation report.

A lack of ‘fit’ between policy and delivery timelines on the one hand, and commissioned evaluation timelines on the other can also create challenges around involving delivery partners in shaping an evaluation to meet their needs (as well as the Scottish Government’s). Engaging with delivery partners around outcomes to date can be especially difficult when a project is evolving and delivery partners are focused on the next stages of a project. In this context, a retrospective evaluation of what has been achieved to date may seem less relevant to some partners.

An additional, but crucial, timing issue relates to timelines for accessing data held by external organisations to support evaluation. The potential value of routinely collected administrative data for supporting evaluation - by enabling tracking of outcomes, and comparisons with control groups, without additional direct data collection from participants - is increasingly recognised. However, it can take a very long time to negotiate access to such data.

This was certainly the case for the child poverty pathfinders evaluation. At the outset of the phase 2 evaluation, it was hoped that data collected and held by HMRC, Social Security Scotland, and DWP on income, benefits and employment would facilitate analysis of impacts of the Dundee pathfinder on these outcomes without the pathfinder team needing to collect detailed and comprehensive data from families directly. However, while an additional Quasi-Experimental^[1] element to the evaluation is still being planned, at the time of writing, there remained uncertainties over whether all the data requested would be made available, even 18 months after requests were first made.

Evolution vs. consistency

As ‘pathfinders’, a key feature of the programmes in both Dundee and Glasgow was that they were intended to evolve over time as they learned from delivery. A commitment to learning and continual improvement is a common requirement of many government funded programmes. However, an iterative approach to design and delivery can lead to a variety of evaluation challenges. These include:

• The availability of quantitative baseline data on key outcomes, since understandings of which outcomes are key to track may evolve over time. This is problematic from an evaluation perspective, as being able to draw robust conclusions around whether an intervention has led to any change requires collecting the same data from participants before and after their involvement, to measure whether there has been any change.
• Defining what is ‘in scope’ for evaluation, which can be challenging when the boundaries of an initiative are themselves somewhat fluid. If the focus of a programme in terms of target groups, activities, or outcomes is not completely fixed, earlier decisions about the focus of evaluations may also need to be revisited. This has implications for both evaluation design and costing, as discussed below.

These issues are arguably particularly acute when there is a desire to accurately quantify the impact of programmes. The most robust approaches to quantifying impact – including Quasi-Experimental Designs (QEDs) and Randomised Controlled Trials (RCTs)^[2] – are premised on the availability of quantitative baseline data and on a high degree of consistency in programme design and delivery over time. This may mean they are less appropriate for programmes like the pathfinders which assume interventions will be flexible, at least in their earlier stages.

Burden on providers and families

Those involved in delivering programmes to participants will have many demands on their time and there is a need to avoid over-burdening them with requests for data and input to evaluations. However, in practice, it is very difficult to avoid tensions arising between minimising the burden on both delivery teams and participants, and a desire for robust data. In particular, there may be challenges around:

• Ensuring that data collection does not become a barrier to participants engaging with a programme, vs. collecting robust and detailed quantitative baseline data on both the characteristics of those engaging and on key outcomes. This was described by pathfinder stakeholders as a recurrent tension on these and similar projects working with underserved populations – potential participants may be deterred from taking part if this is seen as contingent on providing delivery staff with extensive baseline information, some of which may be personal or sensitive.
• The burden on project teams from data collection requirements. In Dundee, the relatively small delivery team meant that additional requests for monitoring data were very challenging to accommodate without negatively impacting on delivery to clients. In Glasgow, the external evaluation was only one of a variety of evaluation and research activities taking place on or around the pathfinder. Both of these factors can create unavoidable challenges for external evaluations – in terms of the balance between burden on teams and the scope and robustness of data collection, and the potential for duplication of research activities commissioned by different stakeholders – which will require careful navigation.

Challenges in engaging participants with diverse views

When evaluating the impact of an intervention aimed at a particular group, the ideal scenario is to be able to hear from both those who were fully engaged with the programme and those who were less engaged, or did not engage at all. However, on projects involving families or individuals in challenging circumstances, it is often the case that those who do not engage with projects are also unlikely to engage with evaluators. External evaluators may lack direct access to participants, unless the project has collected consent to allow sharing of contact details with external researchers. This can both increase the burden on delivery teams, since evaluators are reliant on their help in recruiting participants for interview, and increase the potential that interviews are biased towards those who had a more positive experience of the project.

Conceptualising and assessing value for money

Scottish Government evaluations are placing increasing focus on the assessment of value for money, particularly where (as with the pathfinders) they represent significant investments and there are plans to scale up the approaches developed across a wider area. However, conceptualising and measuring this is not necessarily straightforward in practice.

The most commonly recognised approaches to value for money assessment are based on measuring whether a programme has achieved its target aims for the people it supports, and comparing this with the overall costs in order to establish a cost-benefit ratio. There are established approaches to support these calculations.^[3] However, these approaches assume both that there is sufficiently robust quantitative data available on outcomes for individual participants (and, for robust analysis, a comparison group), and that the value of a project lies only or mainly in its outcomes for individual participants.

Challenges around collecting or accessing robust data on individual outcomes have been described above – these same challenges will limit the scope for Social Cost Benefit Analysis (SCBA). Where robust quantitative data on individual outcomes is not available, alternative approaches for value for money will be required. In Dundee, for example, the agreed final approach was to develop a ‘break-even model’, which would help inform understanding of what level of outcomes a project like the pathfinder would need to achieve in order to ‘break even’ in terms of benefits to costs, but which stopped short of saying whether or not the pathfinder had yet achieved this, given data limitations.

An additional challenge for the pathfinders evaluation was how to assess the value for money of ‘whole system change’. This was a core component of both pathfinders, but particularly the Glasgow pathfinder, which was conceived as a whole system change programme rather than as an individual ‘intervention’ or set of interventions. It is also likely to be a key feature of other Scottish Government programmes where public service reform is a key aim. However, there is currently no recognised approach to assessing the value for money of whole system change. Given this, the evaluation of the Glasgow pathfinder did not attempt to assess its value for money, but developed suggestions for principles, approaches, and indicators for assessing the value for money of a multi-agency system change programme in the future.

Particularly where standard VfM approaches are not feasible, there may be a need to consider alternative ways of conceptualising and considering value for money. For example, the pathfinder evaluation also explored stakeholder perceptions of value for money across the key dimensions of economy, efficiency and equity, which have been identified as three of the ‘four Es’ of value for money (alongside effectiveness, which was the focus of the evaluation as a whole).^[4]