Evaluating evolving and complex programmes: learning and reflections from the child poverty pathfinders' evaluation

Appendix B: Potential indicators of the value for money of ‘whole system change’

Potential indicators and types of data relating to the value for money of ‘whole system change’ are outlined below, drawing on suggestions developed for the Glasgow pathfinder and on the Performance Framework developed by the Glasgow Community Planning Partnership. This is intended to stimulate reflection and discussion rather than to be a prescriptive formula or checklist. For each evaluation, it would be essential to consider what type of data is already available locally, and how complex or time consuming it might be to collect and collate it if not, in planning what is feasible in this regard.

Economy indicator 1: How much does it cost to deliver system change?

• Type of data: cost data (£) + qualitative data on hidden / less easily quantifiable costs.
• Additional considerations: Asking partners to estimate time spent on ‘system change’ is quite burdensome and can be difficult to judge if this time is part of their wider job. Given this, it may be more realistic to focus on estimating direct costs and capturing wider costs (including perceived opportunity costs) more qualitatively.

Economy indicator 2: Is the overall programme delivered at an appropriate price for the outcomes?

• Type of data: Data on cost above + qual & quant outcome data (see discussion re. effectiveness below).
• Potential data to evidence this over medium-long term: Combination of review of cost data vs. data on system change outcomes achieved. This would be over and above any analysis of the costs and benefits of individual interventions developed and tested as part of a programme.
• Additional considerations: Assessments of economy need to take account of the potential that costs might increase initially, but should reduce as system change starts to bed in. This means the question may not be fully answerable even in the medium-term (though it is still important to consider progress).

This also needs to reflect the principle that VfM should be considered not only in the context of what has happened in terms of outcomes, but what would have happened if there had been no system change. This may, therefore, need to involve some counterfactual scenario analysis – e.g. comparing child poverty levels in 2030 with the levels estimated/expected by 2030 in 2022, or comparing overall changes in child poverty rates with a ‘control’ area/s (e.g. potentially comparing with changes for Scotland as a whole, or rates in a similar area of England).

Efficiency indicator 1: Could any of the changes have been delivered differently for the same or better impact?

• Type of data: Qualitative data on how budgets were spent and views on whether this was too much or not enough given observed impacts.
• Potential data to evidence this over medium-long term: Qualitative discussions with key stakeholders (internal and external) to identify whether any of the costs were unnecessary or too high relative to the system change outcomes achieved, or whether more could have been achieved with additional resource.
• Additional considerations: This will likely need to consider individual elements of the programme and the evidence for their impact on system change outcomes (acknowledging challenges re. attribution), as well as views on whether each element could have been delivered more efficiently.

Efficiency indicator 2: Has the pathfinder led to efficiencies elsewhere in the system)?

• Examples of type of data:
  • Quantitative data on crisis referrals / spend
  • Quantitative / qualitative data on demand on other services
  • Quantitative / qualitative data evidencing levels / types of failure demand
• Potential data to evidence this over medium-long term: This could draw on a range of data collected locally (mapping relevant data available in the area the programme takes place could form part of evaluation planning), such as:
  • Reduction in crisis grants/referrals (using Scottish Welfare Fund data)
  • Number of Police / fire call outs where there is no crime (e.g. concern about a vulnerable child)
  • Lower demand for debt-related services: Rent and Council Tax arrears, demand on Advice services, bailiff referrals
  • Analysis of complaints to identify the number of complaints where the cause is being offered the wrong service
  • Surveys of referral partners to explore views on whether referral processes are smoother/more effective
  • Greater Manchester CBA research (GMCA, 2022) provide a bank of values (savings to the exchequer) and methodological considerations that could potentially be used to monetise the impacts of lower demand for services, if there is evidence of this.
• Additional considerations: It has to be acknowledged that attribution will be difficult here – e.g. how much of any change in crisis referrals is a result of the system change programme vs. the wider external context?

Answering this question should also consider whether there are areas of the system where costs / demand have increased - e.g. demand for childcare places; uptake of Health Visitor appointments; uptake of different benefits. Increases in demand could be positive (e.g. demand for childcare places because more parents are working) but are nonetheless important to consider in the context of the overall long-term efficiency impacts of system change.

The counterfactual is again relevant: what would have happened to demand on other services if the system had not changed? Demand may not go down (particularly initially), but it may rise less rapidly, for example. Ideally, changes in demand over time would be compared with changes in demand in an area(s) with similar levels of child poverty in 2022 but without an intervention, to form a control group.

Equity indicator 1: Has system change led to resource being better targeted (i.e. at families most at risk of poverty)?

• Type of data:
  • Data on where services are located vs. where families in poverty live
  • Data on uptake of services by families at risk of poverty
• Additional considerations: Mapping whether relevant data is available in the area the programme takes place, and how feasible it would be to access and collate it, would need to form part of evaluation planning. Trying to establish uptake across all relevant services may be a considerable undertaking, so collation of this data may need to be more focused or targeted.

What is considered a positive outcome in service uptake may also change over time (e.g. an increase in uptake of crisis support may be positive in the short term but not in the longer term).

Effectiveness indicator 1: Has there been the necessary culture change across partners to support system change?

• Type of data:
  • Quantitative/qualitative data on trust between partners
  • Quantitative/qualitative data on morale and optimism among relevant staff across the public and third sector
• Additional considerations: Qualitative interviews or staff surveys could also explore whether staff feel more empowered that they can make a positive difference to child poverty as evidence that system change is progressing in the right direction.

Effectiveness indicator 2: To what extent has there been a shift in support delivery or spend from crisis to prevention?

• Type of data: Spend on crisis support versus prevention support
• Potential data to evidence this over medium-long term:
  • ‘Spend mapping’ exercises. These could be undertaken at intervals to track the direction of travel.
  • Analysis of departmental budget planning discussions and documents.
  • Qualitative interviews with departmental heads, third sector partners and the wider public.
• Additional considerations: Stakeholders from the Glasgow pathfinder emphasised a desire to move from a ‘vicious circle’ of crisis intervention and spiralling costs to a ‘virtuous circle’, where money saved on costly crisis response is reinvested in early intervention and prevention.