Child poverty pathfinders in Dundee and Glasgow: phase two evaluation

10. Value for money: Glasgow

This chapter discusses value for money in the context of the Glasgow pathfinder. It draws particularly on a stakeholder workshop focused on value for money and on interviews with professional stakeholders.

Stakeholder views on economy, efficiency and equity in the Glasgow pathfinder

As in chapter 6, this section uses the ‘4Es’ as a conceptual framework to consider qualitative evidence on the value for money of the Glasgow pathfinder’s system change activity. It focuses on the first three of these, economy, efficiency and equity, since early indicators of the effectiveness of the Glasgow pathfinder’s approach to system change were discussed in the previous chapter.

Economy: the costs of delivering system change

In terms of assessing what it costs to deliver system change, professional stakeholders highlighted challenges around defining the boundaries of the Glasgow pathfinder. As discussed, the MACT team is the key element funded by the Scottish Government, who contribute around half of the annual £1million cost to bring in and ‘buy-out’ staff to focus on system change. Other ‘direct’ costs have included paying for NWD network events (though these were considered to have been run at minimum cost) and IT (e.g. the purchase of AdvicePro for Glasgow Helps).

However, there are various wider costs that are not covered by the Scottish Government grant and which are less easy to measure, such as time contributions by the Scottish Government and others and time spent on the pathfinder and related activities by NWD network members. As in Dundee, it was argued that in part these are activities that some people would be doing as part of their job anyway, so are not necessarily ‘additional’ costs. In other words, they might already be working in and on ‘the system’, and the pathfinder is just encouraging them to rethink how they do this. However, it was also acknowledged that spending time thinking about how to work differently can take people away from other immediate priorities, so there is an opportunity cost, even if partners are committed to the longer-term aims:

“We could probably put an opportunity cost on the time and stuff like that. [It takes] time to drive a wee bit of cultural change as well. So, I’d say we're not having a massive financial outlay, it's only people committing time to it along with multiple competing priorities. [...] We are still keen to make sure we are working with everyone on this because in essence, it's probably going to save us money in the long run.”

(Glasgow professional stakeholder, NWD network)

As such, it was felt important to capture these costs, even if they were arguably not wholly additional.

Efficiency: optimising resources

The efficiency dimension of value for money is usually considered in terms of whether an initiative itself is delivered, as far as possible, in a way that optimises resources. However, in the context of a system change programme, the aim of the programme itself is to deliver a more efficient system.

Across the NWD network members interviewed for this evaluation, there was a strong belief that their investment in the pathfinder would deliver efficiency savings in the long-term. They anticipated it would ultimately reduce duplication, through more efficient referral processes. In turn, this would reduce staff time by enabling better targeting of support across partners.

"So by being linked up to those other agencies that can provide that support, ultimately it will free up a lot of (staff) time. I think it would be more than breaking even from our side, what we were getting back comparatively to what we're putting in.”

(Glasgow professional stakeholder, NWD network)

While these benefits were mostly envisaged as a future return on their investment of time in the pathfinder, there were some examples where network members felt there had already been improvements in referral processes, as described in chapter 8.

In terms of whether the wider work of the pathfinder was delivering efficiency savings, again, it was largely seen as too early to say. However, the progress on funding flexibilities, discussed in the previous chapter, was seen by some stakeholders as an area where the pathfinder had arguably already delivered efficiencies, by starting to reduce duplication in performance management and overheads in relation to managing multiple funding pots.

On the question of whether the pathfinder itself is being delivered in an efficient manner, there was some frustration with the volume of meetings and length of the process for progressing actions, particularly among NWD network members. However, even those who expressed frustration in this regard recognised that taking the time to get it right and to properly involve multiple partners might ultimately lead to better outcomes:

“It's been quite a slow process … I suppose if they've taken longer then it might mean that it's actually done right because they're taking time to consider it."

(Glasgow professional stakeholder, NWD network)

Equity: the distribution of benefits

On equity, although again it was considered too early to quantify impacts, there was a clear sense that the better use of data fostered by the pathfinder was helping to target resources where they were most needed, particularly in relation to the DoCs. At the same time, there was some concern about whether being exclusively ‘data driven’ could risk missing families who are not included in existing administrative data. In addition to working families, discussed in chapter 8, asylum seekers were also mentioned as a potentially under-represented group in this context.

Broader equity considerations relating to the focus on child poverty were also raised; in particular, is there a danger of excluding those without children who are experiencing poverty? However, in the context of whole system change, it was felt that there was more likely to be a “rising tide lifting all boats” effect, whereby improving the system of support benefits all citizens, not only families with children.

How should the value for money of a multi-agency system change programme be conceptualised?

As the above discussion suggests, the value for money workshop identified a number of key challenges in conceptualising and measuring value for money of a multi-agency system change approach like the Glasgow pathfinder. These are set out below.

• The lack of an established approach to measuring or estimating the social or exchequer values of changes to a system. Typically, cost-benefit analyses focus on estimating the value of benefits to costs of individual interventions with clearly defined short to medium-term outcomes for individual beneficiaries. There are well established approaches to estimating these values, as outlined in the UK Treasury’s Green Book. A social cost benefit analysis or similar approach could be applied to individual test interventions developed as part of the pathfinder. This includes Glasgow Helps, for which the pathfinder team had already produced an SCBA report (Sharif, 2023), and the DoC projects, with which the pathfinder was working to develop value for money assessments. But an equivalent method for monetising and assessing the value of a programme of whole system change, where the intended outcomes are at the system level (although with the ultimate aim of also improving individual outcomes), does not currently exist.
• Difficulty in confidently attributing specific system change outcomes to the child poverty programme, or to specific elements of this (such as the MACT). Isolating the impact of a particular programme or set of activities on a complex system over time is arguably even more challenging than estimating the impact of a particular intervention on individuals. There will be many other factors impacting on how a system develops over time which may also influence outcomes, either positively or negatively. In the Glasgow context, comparisons with the programme of transformation in children’s services were considered relevant. While there was a strong view that this programme had been instrumental in reducing the numbers of children being looked after away from home by 50% from 2016 to 2023, disaggregating the specific impact of particular activities within this broader change programme on this reduction was much more difficult (see Glasgow HSCP, 2023).
• What is in scope in terms of costs and drivers? Linked to the previous point, there was discussion of which elements of the Glasgow pathfinder should be considered ‘in scope’ in terms of both estimating costs and considering what has driven change. The MACT was the most obvious direct cost (particularly in terms of Scottish Government grants), and any changes to the system that are considered to flow directly from their activity could be reasonably confidently attributed to the pathfinder. However, as discussed, the MACT are working with many other partners, through the NWD network, GCPP and beyond. Each of these partners contribute their own resources which also contribute to system change outcomes. But estimating and attributing both costs and outputs from these wider partners to the pathfinder is more challenging, since they are not solely focused on the pathfinder.
• Appropriate time frames for system change. As has been discussed in previous chapters, there was a strong sense that Glasgow was early in its system change journey. It was argued that upfront investment was required to drive change, but that the economic and social value of these changes were likely to take longer to achieve. Indeed, one participant argued that, in their experience of previous attempts at PSR, a key reason for failure was the “expectation that savings or community / society level changes will happen more quickly than they ever do.”

These challenges notwithstanding, a number of clear principles for conceptualising and assessing the value for money of system change emerged from the workshop and from interviews. These are set out below.

Principle 1: Consider the ‘value for money’ of system change using different outcome indicators across different time frames

While there was agreement that whole system change may take a very long time to fully achieve, it was also recognised that there was a need to evidence short-term achievements that could give confidence the longer-term aims would be achieved. There were different views on the best way of dividing up the timeline. However, for Glasgow, this could comprise:

• Early indicators (1-3 years): These may be more qualitative, such as perceived signs of culture change or agreement to funding flexibilities (as discussed in the previous chapter) but should provide some confidence around the direction of travel.
• Interim system outcomes (4-9 years): in the medium term, more substantial changes to the system should become apparent (e.g. budgets moving towards preventative spending).
• Long-term sustainable outcomes (10+ years): Longer-term, if successful, system change should reduce failure demand (where people require more costly and intensive support because they did not get the right support at the right time), improving both outcomes for families and reducing (or stabilising) the overall cost of the system of support

Principle 2: Consider broader cost savings, at short, medium, and longer-term

In the early stages of a programme in particular, there was a significant perceived risk of dismissing the value of investing in system change because efficiency gains were not yet apparent. It was emphasised that the value for money of system change should not be measured using short-term cost savings (which were viewed as unlikely) but should include broader system benefits in both the short and medium-termfor example, greater joining up of services, which can lead in the longer-term to more substantial administrative restructuring and optimization of delivery. Some potential broader indicators are discussed at the end of this chapter.

Principle 3: Consider the value for money of system change vs. the alternative of leaving the system ‘as is’

Stakeholders repeatedly emphasised that continuing to work within an unreformed system of public services was not financially sustainable. Medium to long-term, demand is forecast to rise very substantially across public services, including potentially cost for services for people of working age^[34]. As such, while system change programmes need to be assessed in terms of whether they are on track to avoid this long-term scenario (by joining up services, reducing duplication, etc.), reverting to the status quo if these outcomes are not immediately achieved may not be a viable option. This highlights the importance of developing a robust ‘quasi-experimental’ design for measuring the impact of system change against a counterfactual of what would have happened in the future without the systems change, rather than comparing to the current situation. This would aid understanding of how to interpret falling costs, should they arise.

Principle 4: Use a mixed method approach to measuring benefits

Given all of the above considerations, assessing value for money in the context of system change is likely to require both qualitative and quantitative evidence. Some of the impacts considered essential in driving long-term reform (like culture change) are harder to ‘count’, and there is no obvious way of applying a monetary value to them. However, in the medium to longer-term, quantitative evidence of changes in demand, shifts / changes in budgets and spending, etc. will also be needed as ‘harder’ tests of what system change has achieved.

The end of this chapter returns to how these principles might be operationalised to assess value for money, particularly in the medium- to long-term.

Measuring and assessing value for money of a multi-agency whole system change programme

This final section reflects on potential medium- and long-term indicators of the value for money of system change, building on the principles identified above. It is intended to be provisional and will require further discussion and refinement by stakeholders. However, it is hoped it will provide a potential structure for considering value for money in both Glasgow and other areas embarking on programmes of PSR. The suggestions below are structured around the ‘three Es’, above, while discussion of potential indicators of effectiveness builds on the evaluation theory of change. The draft Performance Management Framework (PMF) that has been developed by the GCPP, with input from the pathfinder MACT, includes a number of proposed measures of relevance to future value for money assessments, some of which are referenced below.

Economy indicator 1: How much does it cost to deliver system change?

• Type of data: cost data (£) + qualitative data on hidden / less easily quantifiable costs.
• Potential data to evidence this over medium-long term:
  • Direct costs of MACT (SG + GCC).
  • Direct costs of events, test interventions (DoC), other elements funded as part of the programme (GCC)
  • Could ask partners to try and estimate time spent on different elements of the pathfinder (but see notes).
• Additional considerations: Asking partners to estimate time is quite burdensome and can be difficult if this time is part of their wider job, and it could be argued they would be doing ‘system-wide’ management / engagement anyway. Given this, it may be more realistic to focus on estimating direct costs and capturing wider costs (including perceived opportunity costs) more qualitatively. This report has started to do that, but it may be appropriate to use, for example, NWD network surveys to gather more detail on this.

Economy indicator 2: Is the overall programme delivered at an appropriate price for the outcomes?

• Type of data: Data on cost above + qual & quant outcome data (see discussion re. effectiveness below).
• Potential data to evidence this over medium-long term: Combination of review of cost data vs. data on system change outcomes achieved. This would be over and above any analysis of the costs and benefits of individual interventions developed and tested as part of the programme (which would be intended to inform learning and adaptations to future interventions).
• Additional considerations: Assessments of economy need to take account of the potential that costs might increase initially, but should reduce as system change starts to bed in. This means the question may not be fully answerable even in the medium-term (though it is still important to consider progress).

This also needs to reflect the principle that VfM should be considered not only in the context of what has happened in terms of outcomes, but what would have happened if there had been no system change. This may, therefore, need to involve some counterfactual scenario analysis – e.g. comparing child poverty levels in Glasgow in 2030 with the levels estimated/expected by 2030 in 2022, or comparing overall changes in child poverty rates in Glasgow with a ‘control’ area/s (e.g. potentially comparing changes in Glasgow with changes for Scotland as a whole, or rates in a similar English city).

Efficiency indicator 1: Could any of the changes have been delivered differently for the same or better impact?

• Type of data: Qualitative data on how budgets were spent and views on whether this was too much or not enough given observed impacts.
• Potential data to evidence this over medium-long term: Qualitative discussions with key stakeholders (internal and external to the MACT) to identify whether any of the costs were unnecessary or too high relative to the system change outcomes achieved, or whether more could have been achieved with additional resource.
• Additional considerations: This will likely need to consider individual elements of the programme (e.g. MACT, DoCs, investment of partner time / resource in NWD etc) and the evidence for their impact on system change outcomes (acknowledging challenges re. attribution), as well as views on whether each element could have been delivered more efficiently.

Efficiency indicator 2: Has the pathfinder led to efficiencies elsewhere in the system)?

• Type of data:
  • Quantitative data on crisis referrals / spend
  • Quantitative / qualitative data on demand on other services
  • Quantitative / qualitative data evidencing levels / types of failure demand
• Potential data to evidence this over medium-long term: (a number of the proposed measures in the PMF are relevant to answering this question) e.g.:
  • Reduction in crisis grants/referrals
  • Number of Police / fire call outs where there is no crime (e.g. concern about a vulnerable child)
  • Lower demand for debt-related services: Rent and Council Tax arrears, demand on Advice network, bailiff referrals
  • Analysis of complaints to identify the number of complaints where the cause is being offered the wrong service
  • GCPP partnership survey could also measure perceptions of whether referrals processes are more joined up/ efficient over time.

  Greater Manchester CBA research (GMCA, 2022) provide a bank of values (savings to the exchequer) and methodological considerations that could potentially be used to monetise the impacts of lower demand for services, if there is evidence of this.

• Additional considerations: It has to be acknowledged that attribution will be difficult here – e.g. how much of any change in crisis referrals is a result of the pathfinder vs. the wider external context?

Answering this question should also consider whether there are areas of the system where costs / demand have increased - e.g. demand for childcare places; uptake of Health Visitor appointments; uptake of different benefits. Increases in demand could be positive (e.g. demand for childcare places because more parents are working) but are nonetheless important to consider in the context of the overall long-term efficiency impacts of system change.

The counterfactual is again relevant: what would have happened to demand on other services if the system had not changed? Demand may not go down (particularly initially), but it may rise less rapidly, for example. Ideally, changes in demand in Glasgow over time would be compared with changes in demand in an area(s) with similar levels of child poverty in 2022. Finding perfect matches in Scotland may, however, be difficult given Glasgow’s profile. An imperfect option would be to look at changes in overall level of demand across Scotland to see whether the relative change was different in Glasgow (for measures where data is collated centrally).

Equity indicator 1: Has system change led to resource being better targeted (i.e. at families most at risk of poverty)?

• Type of data:
  • Data on where services are located vs. where families in poverty live
  • Data on uptake of services by families at risk of poverty
• Potential data to evidence this over medium-long term: Again, the PMF suggests mapping of key service provision per ward. This could be undertaken for ‘booster’ wards and repeated at intervals to assess change. Data on uptake of services is also relevant here – monitoring patterns of uptake of services across Glasgow among priority families known to be at higher risk of poverty, and whether this changes over time.
• Additional considerations: Monitoring uptake of pathfinder specific services (e.g. GH and DoCs) may be more straightforward than measuring uptake of, e.g., NWD partner services. However, as the pathfinder is exploring a shared case management system and approaches for the NWD network, monitoring the proportion of cases that are accounted for by families known to be at higher risk of poverty may become more feasible in future.

What is considered a positive outcome in service uptake may change over time (e.g. an increase in uptake of crisis support may be positive in the short term but not in the longer term).

Effectiveness indicator 1: Has there been the necessary culture change across partners to support system change?

• Type of data:
  • Quantitative/qualitative data on trust between partners
  • Quantitative/qualitative data on morale and optimism among relevant public and third sector staff
• Additional considerations: Future qualitative interviews or staff surveys could also explore whether staff feel more empowered that they can make a positive difference to child poverty as evidence that system change is progressing in the right direction.

Effectiveness indicator 2: To what extent has there been a shift in support from crisis to prevention?

• Type of data: Spend on crisis support versus prevention support
• Potential data to evidence this over medium-long term:
  • The PMF proposes ‘spend mapping’ exercises. These could be undertaken at intervals to track the direction of travel.
  • Analysis of departmental budget planning discussions and documents.
  • Qualitative interviews with departmental heads, third sector partners and the wider public.
• Additional considerations: The current evaluation theory of change is missing this key element which is a central aim of the pathfinder (and wider PSR). Stakeholders emphasised a desire to move from a ‘vicious circle’ of crisis intervention and spiralling costs to a ‘virtuous circle’, where money saved on costly crisis response is reinvested in early intervention and prevention.