Child poverty pathfinders in Dundee and Glasgow: phase two evaluation

11. Conclusions and learning

This final chapter starts with a summary of the key findings. It then focuses on cross-cutting learning from the pathfinders. It draws on the data presented in the previous chapters and on the evaluators’ reflections on these. While we hope it will prove useful for Dundee and Glasgow, it has a particular focus on learning for other areas of Scotland who are, or may be, developing person-centred interventions, plans for PSR, or both. Boxed ‘questions for consideration’ are interspersed through the chapter. These are aimed particularly at future, similar partnerships such as the Fairer Futures Partnerships, but some may also provide food for thought for the current pathfinders.

Key findings

Overall, there is positive qualitative evidence from both families and professionals that the support offered through the pathfinders has been meaningfully different. It has helped families address immediate crises and supported them to shift towards tackling longer-term or underlying issues. There are also encouraging signs of progress toward system change – though this is clearly a long-term endeavour.

Key findings from the Dundee pathfinder

Impact for families

Qualitative data suggests that the targeted outreach to families who would not normally engage with support has been effective – from both a practical point of view (using Council Tax Reduction data to identify families) and in terms of building relationships (the proactive contact by key workers made the support seem accessible and approachable).

The qualitative evidence from parents about their experiences of support was overwhelmingly positive and supports the scope for this approach to help families address their immediate needs and take positive steps towards employability and/or a more financially and emotionally stable situation. The non-judgemental, trusting relationships they formed with key workers were critical to these impacts.

However, qualitative interviews also highlight the significant ongoing barriers some families face to moving out of poverty on a sustainable basis, including structural barriers (particularly wider benefits levels and rules, and the local job market), and complex health issues (and the capacity of wider services to respond to these).

The data also raises some questions around how to strike the right balance between an approach based around forging strong, trusting relationships with particular professionals, and avoiding ‘dependency’.

Systems change

Qualitative interviews with professionals provide some evidence that learning from the pathfinder has started to feed into wider services and there were felt to have been some improvements to joint working locally and improved understanding between national partners.

However, data sharing remained a barrier to more effective joint working and there was also some scepticism across partners over the likelihood of sustainable system change being achieved unless barriers relating to national systems and rules (such as those relating to Universal Credit work requirements or legal gateways for data sharing) could be overcome.

There was mixed evidence on whether the pathfinder had helped families become better equipped to navigate the system and, at this stage, evidence that the system itself had become easier to navigate was limited.

Key findings from the Glasgow pathfinder

Impact for families

Qualitive evidence from parents supported by Glasgow Helps confirms the potential for this approach to engage families who have struggled to access appropriate support previously – and that it has been effective in addressing a very wide range of needs. There are also some signs from No Wrong Door partners that the pathfinder is starting to help join up services in a way that enables them to link families to the right support, more quickly.

Quantifying the impact of the pathfinder on families is currently very difficult, due to the lack of data, the wide scope of pathfinder activities, and the early stage that some of the initiatives were at. However, parent case studies highlight the ways in which they have been able to move from addressing short-term financial needs to addressing longer-term barriers to sustainable outcomes, including health and wellbeing and support around employment and training.

Systems change

Professionals expect that whole system change will take at least 10 years to achieve, so evidence on this longer-term outcome is limited at this point. That said, there are positive early signs of progress, including: improved collaboration and partnership working between GCC, the third sector, the Scottish Government and parts of the wider public sector; promotion of shared culture and values; and progress on embedding a No Wrong Door approach across a wide network of services.

There was a strong consensus across interviewees that the approach in Glasgow was the right one in terms of the likelihood of achieving long-term reform, and that investing in the multi-agency change capacity team to develop and drive this had been critical.

Cross-cutting learning from the Dundee and Glasgow pathfinders

Are the principles of the pathfinders transferable?

In spite of the differences between them, at their centre, both pathfinders are rooted in an approach to supporting families that is person-centred, relationship-based, holistic, and relies on partnership working. Both are also now focusing particular attention on place-based working. These approaches and principles could clearly be implemented in other parts of Scotland; indeed, it must be acknowledged that most areas and services working with families would probably say they are already aiming to do many of these things. However, many would also probably point to the barriers that have prevented them achieving this to the degree they want in practice – including limitations that have been reflected in the pathfinders, such as tensions between resources and demand, and data sharing issues.

Perhaps the key difference with the pathfinders is that they have invested in focused multi-agency work to try and overcome barriers to implementing these approaches in the specific context of child poverty. In Glasgow, this investment has focused specifically on ‘whole system change’. Again, the principle of taking a ‘whole system change approach’ is transferrable, as are the broad themes and workstreams the Glasgow pathfinder has focused on. At the same time, it must be recognised that Glasgow is Scotland’s biggest council (in terms of population), so there would need to be careful consideration of how to resource this kind of programme in other areas. Even if financial resources were made available to allow for a similar ‘multi-agency change capacity’ team to drive change, it might be more challenging for other, smaller local authorities to free staff resources for this purpose.

Another point is that both pathfinders are city-based. There are different challenges around both bringing partners together (including co-location of teams) and delivering place-based interventions in rural areas of Scotland. Learning from the Fairer Futures Partnerships, which include several local authorities with large rural areas, and from the Clackmannanshire Family Wellbeing Partnership (which is being evaluated separately) will be important to understanding whether and how pathfinder principles and approaches can be implemented in rural settings.

Overall, while the principles of the pathfinders are transferable, there is more to be done to understand whether and how these can be implemented and scaled in practice, particularly given differences in resources and geography across Scotland.

What does it mean to be a ‘pathfinder’?

As described in chapter 1, the 2022 TCPDP stated that the Scottish Government would work with the pathfinders to “refine, test, adapt and scale different approaches to provide person-centred solutions”. In practice, both pathfinders have taken an iterative approach. Both their activities and intermediate outcomes have evolved over time, even if the stated ultimate goal – reducing child poverty – has remained the same. The framing of the pathfinders around testing and adapting arguably distinguishes them from more ‘standard’ funded interventions, which may have more strictly defined goals and timelines from the outset.

Stakeholders in both pathfinders emphasised the value of the learning arising from the pathfinders. However, taking a more iterative approach has wide-ranging implications, not only for monitoring and evaluation, but also for how pathfinders achieve and sustain buy-in from partners; which partners are (or should be) involved as pathfinders evolve; how to communicate what the pathfinder is doing and trying to achieve; and how key internal and external stakeholders may view progress. For projects that plan to take a similar approach, it will be important to reflect on these implications at the outset and on an ongoing basis.

As has been noted, the Dundee and Glasgow pathfinders differed substantially in both scale and focus. In particular, the scope of system change aims and the resource dedicated to this differed between the two. This raises questions about the balance within ‘pathfinder’ type projects between developing and rigorously testing specific interventions and focusing on achieving wider system change. It also raises a question about whether diverse or changing local models can or should be evaluated against a common, overarching goal.

Questions for consideration:

• How wide is the pathfinder’s system change ambition? What is the balance between testing an intervention(s) and focusing on system change?
• Assuming the ultimate outcome is reducing child poverty, how much flexibility are partners prepared to accept around intermediate outcomes?
• If intermediate outcomes need to evolve over the course of the intervention, who needs to be involved in agreeing this, and how do you maintain buy-in if outcomes change?
• What evidence will count as learning?
• What are the expected timeframes for achieving intermediate and ultimate outcomes? Are these expectations clearly evidenced, articulated and shared across all partners and key stakeholders? How much flexibility will partners accept around timeframes?
• When, how and by whom should all of the above be reviewed, particularly where initiatives are specifically set up as ‘test and learn’ projects?

Making a difference to families on low incomes

The evidence on impacts for families from both pathfinders is more qualitative than quantitative at this stage. However, qualitative interviews strongly support the potential of person-centred, relational, holistic and flexible approaches to support families in moving from initial crisis towards longer-term outcomes. Such longer-term outcomes may include employment, or steps towards employment, or helping families to reach a more stable position in terms of their financial, emotional and social wellbeing. This evaluation also reinforces the benefits – both for families and for services – of increased networking and partnerships. This enabled partners to reach families they would not otherwise reach (for example, through having better connections with partners who already reach different groups) and supported families to access services they otherwise may not have known about (or trusted).

However, while the evidence of potential impacts on families is extremely positive, the work of both pathfinders raises a number of questions and challenges that may need clearer answers if this way of working is to be ‘scaled up’ much more widely. These are discussed in turn below.

Targeted vs broad approaches to tackling child poverty

Both Dundee and Glasgow have developed approaches that involve offering support for citizens who need it, while also finding ways of specifically identifying and reaching out to parents on low incomes, to link them to this support. There was a strong view among some stakeholders that this was the right approach. It avoids stigmatising low-income families by setting ‘criteria on who is and is not eligible for support; enables the pathfinders to have a wider impact on poverty across the two cities; and avoids other citizens missing out on much needed support. However, interviews (and data on attendance at the drop-in from Dundee) also highlighted potential implications in terms of managing demand and resources, and the extent to which interventions are able, in practice, to focus on families as efficiently and effectively as possible.

Questions for consideration:

• Who is the core target group for the service/intervention? How much flexibility is there around who can actually access support? Are there/should there be eligibility criteria for particular elements (e.g. more intensive support)?
• What are potential and actual implications of the approach adopted (whether more open or more targeted) for demand and resources, on the one hand, and for stigma and missed opportunities, on the other?

Meaning of ‘no wrong door’

The concept of ‘no wrong door’ was referred to across both pathfinders as a crucial principle they wanted to embody. However, it was apparent from interviews and project documentation that this phrase is understood in slightly different ways by different stakeholders. Meanings mentioned included:

• Joining up services/citizens being able to reach all the help they need through whatever service they access (even if that involves onward referrals)
• Citizens only having to tell their story once
• Holistic/person-centred support using case management approach – addressing clients’ full range of support needs, rather than focusing on individual problems in isolation
• Being open ended in terms of time; clients are supported until all their needs are addressed
• Being open in terms of who can access services (linking to the point about targeted vs. broad approaches, above).

While these elements may all be important features of an effective model of family support, there are also potential tensions between some of them in practice. In particular, the experience of the Dundee pathfinder and the level of demand experienced at the Linlathen drop-in highlights a potential conflict between being open in terms of who can access services, and having the capacity to offer more intensive, holistic, person-centred support to a particular demographic and/or geographic cohort.

Questions for consideration:

• What are the defining features of the approach to working with families that the project/programme is taking?
• What, if any, tensions might arise between different elements of this approach and how will these be navigated in practice?
• What is the best way of describing the approach to working with families, to avoid potential confusion?

Setting goals and measuring outcomes

The experience of the pathfinder in Dundee underlines the difficulty of knowing how far it will be possible to support a target cohort of families towards ‘hard’ outcomes (like employment or income improvements) over a specific time period, before engaging with them to better understand their needs. Very early learning from the DoCs also suggests that the families these Glasgow test projects hope to support may have more complex needs than initially anticipated. This highlights the challenge of setting specific targets related to child poverty drivers (particularly employment and benefit income) without fully understanding the population a project or service aims to support.

One option that should be considered for future pathfinders or similar projects would be to agree outcomes, target populations and targets only after an initial period of data analysis. For example, this could include identifying the numbers of people on benefits with ‘no work-related requirements’, which may limit the extent to which they can benefit from further ‘benefit maximisation’ work or engage with employability activities, as well as direct engagement with families to better understand their needs. This would enable projects to plan a more realistic set of goals for the families they are supporting, both individually and collectively. It could also help support agreement on realistic timelines for different goals.

Question for consideration:

• When and how should any expectations or targets around family outcomes and target populations, from projects that take a person-centred approach, be set and what data and engagement is needed to inform them?

Avoiding dependency in relationship-based support

The ultimate aim of person-centred, holistic support in both pathfinders was to empower families to be able to navigate the system more easily and access the support they need more independently. As discussed in previous chapters, although there were some indications of families gaining confidence in dealing with other services, the extent to which this goal was being achieved was not yet clear. It is arguable that achieving this goal is dependent on the system changing first, to become easier to navigate independently. However, interviews with some parents also highlighted that there may be a preference for coming back to key workers with whom they have built strong trusting relationships, even when they know how to access support via other routes. This highlights risks of dependency, especially given that key workers may not be based in the same place, with the same resources, over the medium to long-term. This is an issue Dundee is engaging with now, as it develops plans to expand the pathfinder approach to other wards and considers moving the existing team between different locations.

Question for consideration:

• How do projects balance holistic, relational support with avoiding dependency, so that people are empowered to use support more independently, especially if funding is not indefinite and individual support workers may not be permanently located in the same place-based services?

Key gaps in wider services

Stakeholders in both Glasgow and Dundee identified gaps in support for families around health and wellbeing. While there has been some progress in engaging the NHS and other partners who can support in this area, it continued to be viewed as a key gap in terms of being able to support families towards more positive outcomes. For families with complex health barriers, it was hard to see how projects like the pathfinders can support them towards employment without clients being able to access additional, specific support with these health issues.

Resourcing and capacity within the NHS in general, and mental health services in particular, are of course well known and not specific to Glasgow and Dundee (or, indeed, to Scotland). However, their significance for projects aiming to work with families on low incomes, particularly those with ‘no work-related requirements’, highlight the need to consider how to link with wider mental and physical health support from the outset.

Questions for consideration:

• How can the links between poverty programmes and health services be strengthened?
• How can gaps in families’ access to mental and physical health support be filled?

Delivering sustainable system change

While the Glasgow and Dundee pathfinders had different aims in terms of system change, stakeholders from both pathfinders strongly emphasised the importance of a multi-agency, partnership approach to delivering system change at any level. The findings for both also highlight the importance of:

• Getting the right partners on board at the outset
• Ensuring projects maintain the buy-in of existing and new partners
• Aligning aims and values between partners, and across different levels (delivery, strategic, senior)
• Bringing more partners in as a project identifies gaps, and revisiting whether the right partners are involved as the project/programme grows or evolves.

Both pathfinders also highlight common challenges that need to be considered when trying to implement system change, albeit these have played out in different ways in Dundee and Glasgow. These include:

• Where responsibility for driving system change sits, and the extent to which accountability and leadership are genuinely shared vs. concentrated in one service or organisation
• Issues relating to governance including ensuring the right level of oversight and strategic input for driving both operational delivery and system change
• General challenges around defining and explaining to others what ‘system change’ is; and how, and in what ways, pathfinders are seeking to influence it
• The resources (both time and people) available to drive system change thinking and activity
• Data sharing. Both pathfinders had spent a lot of time and energy trying to resolve data sharing issues that they felt were impacting their ability to target and support families and to facilitate wider system change. Neither had yet been able to achieve what they had hoped in this regard.

Questions for consideration:

• Is there a clear, shared understanding of the type of system change that projects are trying to achieve?
• Are the resources and timescales allocated to projects appropriate for influencing or achieving sustainable system change?
• How can system change aims and plans be clearly explained to internal and external stakeholders?
• What role should the Scottish Government and other national partners play in resolving data sharing issues, to avoid individual local authorities and projects ‘reinventing the wheel’?

Learning for monitoring and evaluation

The preceding chapters of this report have already included various reflections on challenges around monitoring and evaluation. A separate learning paper will also be available on monitoring and evaluation. This chapter focuses on two key questions the Scottish Government and its partners may wish to consider in terms of monitoring and evaluation of future, similar projects/programmes.

What does it mean to evaluate a pathfinder?

A recurrent theme in this evaluation has been the iterative and evolving nature of the Dundee and Glasgow pathfinders. This is an intentional and deliberate feature. Neither had a definitive fixed timeline for achieving the kind of impacts the evaluation questions focused on at the time the evaluation was commissioned. In practice, this meant that some of the original evaluation questions were unanswerable within the evaluation timeframe. Where projects are evolving as they are being evaluated, a greater degree of flexibility around the balance between formative and exploratory evaluation and more formal impact evaluation may be required. At the same time, if there are key elements that funders really want to understand the impact or value for money of, this also needs to be explicitly built into plans for monitoring data collection from the outset to ensure appropriate before and after measures are collected. Careful consideration also needs to be given to how long it is likely to take for these impacts to be achieved, and how this aligns with timelines for evaluation.

What are different partners’ expectations of the level or type of evidence from pathfinders?

It has been apparent from interviews and conversations with stakeholders in both Glasgow and Dundee that different partners (and sometimes different stakeholders within the same partner organisation) have had different expectations in terms of what types of data and evidence they expected to see from the pathfinders, and when. This was reflected in differing expectations both of this external evaluation and of the monitoring and evaluation activities being carried out or commissioned by the pathfinders themselves. Differing opinions have been apparent on the appropriate balance between: qualitative and quantitative evidence; reflection and learning and ‘hard’ evidence of outcomes; internal and external evidence; and data on families or the priority family groups, as distinct from data on citizens in general. There is arguably a need for further reflection on this, and on the implications for both internal and external monitoring and evaluation requirements, as plans for future evaluation of both the current pathfinders and new partnerships are developed.