Fairer Futures Partnership Programme: evaluation strategy

Challenges and risks

It will inevitably be challenging to deliver a fully comprehensive programme of evaluation for the Fairer Futures Partnerships, given pressure on resources, and a rapidly evolving policy landscape, including further expansion of partnerships across 2025/26, as well as ongoing developments in related policy areas. That is why the next phase of the evaluation will put in place a framework for longer-term monitoring of outcomes that can be applied to new partnerships as the programme expands.

Decisions to undertake any further detailed evaluation work alongside this will be made at the appropriate point, and be based on whether this is realistic, proportionate and adds value to the existing evidence base. As noted, in addition to Scottish Government research, the evaluation approach will also need to draw on a range of other evidence sources, and to this end, over the next year we will support capacity building among partners for improved data collection, analysis and (self-) evaluation in order to better evidence progress towards outcomes.

Drawing on learning from the first wave of evaluations, there are also a range of methodological and practical challenges to robust and impactful evaluation of the FFP. First and foremost, the FFP has a number of characteristics that make it a complex intervention. These include:

• The FFPs are testing approaches, rather than rolling out a standard delivery model, and so, by design, are changing and evolving over time, in response to ongoing learning and local need.
• They have a wide and evolving range of stakeholders and actors involved.
• They interact with a wide range of related policies and interventions, including other place-based initiatives operating within the same local areas.
• They are embedded within wider systems, such that activities and outcomes will be influenced by a wide range of external factors (including benefit levels and entitlements, labour market constraints, cost of living challenges, local capacity and gaps in wider support services, etc).
• There is variability in activities across the partnerships, with different approaches being tested, and also variability in individual support journeys, which are tailored to individuals’ needs in line with person-centred principles.

In combination, these characteristics make evaluation difficult. For example, it is difficult to establish what the boundaries of the FFPs are, what the baseline position is and to attribute any changes seen to the FFPs themselves rather than to other factors. This requires an evaluation approach that is cognisant of complexity^[11].

Other practical challenges include:

• Many of the outcomes set out in the Theory of Change will only be fully achieved over the longer term – beyond a single evaluation timeframe - in particular, transformational changes to support systems will take time to achieve and embed.
• It will be difficult to identify indicators and data sources for the full range of outcomes in the ToC; for example, it is often more difficult to measure ‘softer’, less tangible outcomes, but which are known to be important in embedding change, for example shifts in public service culture.
• The availability, sufficiency and quality of data available to local partnerships is likely to have limitations, and partner organisations may have limited capacity for making improvements, making establishing impact and assessing value for money very difficult.
• It may be difficult to assess the extent to which local findings and learning are scalable, adaptable to other areas (especially to areas with different demographic characteristics) and sustainable or cost-effective over the longer term.
• Given the rapidly expanding programme, and ongoing policy development, it may be difficult to deliver findings from the evaluation in a timely way to have the maximum impact on policy and practice.
• The ongoing availability of funding for both delivery and evaluation is not guaranteed and is subject to ministerial priorities.

These are common challenges in many government evaluations. Learning from the previous phase of evaluation work, some of the ways we will seek to address these challenges include:

• Working closely with policy colleagues and partners to ensure that the evaluation approach is attuned to the evolving programme and key policy questions and needs.
• Building additional time into evaluation timelines for extended scoping work and for reviewing the Theory of Change at multiple points, to account for the evolution of the programme and local initiatives.
• Communicating expectations around evaluation with partners from the outset and providing additional support, through both the national evaluation and the learning programme, including supporting local partners to be as explicit as possible about their theory of change, their desired outcomes and the likely timeframes for achieving these.
• Working collaboratively and openly with partners on improving ways of evidencing impacts, including exploring the use of new or innovative data sources, such as administrative data, viewing this as a shared and exploratory journey.
• Taking a methodological approach and considering research questions that are sensitive to the wider ‘ecosystem’ in which initiatives are embedded and which may impact the extent to which they are able to be embedded and sustained.
• Collaborating effectively with a wide range of analytical colleagues and experts to ensure that methodological learning from evaluating complex, place-based initiatives is widely shared, discussed and debated.