Whole Family Wellbeing Funding (WFWF) - year 1 process evaluation: final report

2 Evaluation approach and reporting

Aims and approach

In September 2022, the Scottish Government commissioned IFF Research to undertake a process evaluation of the implementation of Year 1, of Elements 1 and 2 of the WFWF. A process evaluation explores how an initiative (in this case the WFWF) was designed and delivered. It typically explores questions such as: was the initiative delivered as intended, what worked well or less well (for whom and why), what could be improved, and what can be learnt for future initiatives. The aim of the evaluation was to gather evidence from the first year of implementation to inform future Scottish Government policy and practice in the CSPPs. Parallel research of Element 2’s specific approach was undertaken by Rocket Science and Blake Stevenson, and evidence from Element 3 activity will be integrated into an impact evaluation from 2024.

The overarching aims and associated research questions of this Year 1 evaluation were to:

1. Provide an overview of types of activity the Element 1 and 2 funding is being used for.

a. What activities, audiences and topics are CSPPs targeting (or not targeting) their funding towards?

b. How does funding used compare with existing provision?

2. Understand the CSPPs’ views on how Element 1 and 2 funding has been used:

a. To what extent and how did the approach taken by the Scottish Government, in terms of allocation of funding, mechanisms of distribution and support, contribute to CSPPs’ ability to scale up family support services and drive system change?

b. How were funding decisions taken within CSPPs?

c. Which audiences were consulted, and in what ways? (e.g. delivery staff, wider partners, families and young people).

d. Explore experiences of design, set-up, structure, practice, workforce, partnership, managing change.

e. How is performance monitored (by whom, when by, targets, mitigations) to demonstrate impact?

3. Understand children, young people and families’ experiences of family support services that have received Element 1 and 2 funding, and the extent to which this has achieved short-term outcomes (as far as possible).

a. To what extent and how has the funded activity achieved intended short-term outcomes related to service delivery?

b. Were there any unintended outcomes for service delivery?

c. To what extent and how has the funded activity achieved intended short-term outcomes related to the experiences of children, young people and their families?

d. Were there any unintended outcomes for children, young people and their families?

4. Provide evidence for policy and practice to inform future improvement of the WFWF and whole system transformational change.

a. What conditions are necessary for consolidating improvements in the funded CSPPs?

b. What are recommendations for 2023-2026 funding and practice?

Figure 1 below summarises the evaluation approach and timescales.

Figure 1 WFWF Year 1 Evaluation approach overview

Scoping and set-up: September – November 2022

1. Inception & set-up

2. Initial plans & theory of change review

3. Info webinar for case studies

Wave 1: November 2022 – February 2023

1. Strategic interviews

2. Manager interviews

3. Annual reports guidance

4. Interim report

Wave 2: April – August 2023

1. Frontline staff groups

2. CYPF interviews

3. Follow-up manager interviews

4. Annual reports analysis

Final analysis & outputs: August – December 2023

1. Evidence triangulation

2. Findings presentation

3. Final & summary report

4. Infographic & blog

Scoping and set-up

An initial scoping and set-up phase focused on revising the existing logic model and establishing an evaluation framework (presented in Annex 1), both of which helped to govern evaluation direction and inform research tool development. To produce these evaluation tools the evaluation team reviewed strategic documentation and CSPPs initial plans - documents outlining CSPPs’ existing approach to holistic whole family support, planned activity for WFWF, anticipated outcomes for the first year and intended monitoring activities (further detail of how the plans were developed is included in Section 3 and the analysis approach is described in Annex 5).

Initial plans were received from 30 CSPPs, the contents of which were entered into an analytical framework which is a table structured by the research themes to help prepare, consolidate, and organise data ready for analysis. Funding aims, target populations, activities, partnerships, stage of development, and expected outcomes were all included in the framework. The data was analysed to understand how CSPPs intended to use their funding. The information was used by the research team to help inform the design of other evaluation activities, including how to best monitor progress over time; CSP annual report; and the case study research materials (see below).

Qualitative case studies

Case studies were undertaken with six CSPPs (of 32 CSPPs) across two waves of fieldwork, between November 2022 and August 2023. Details of how case study CSPPs were selected, and their characteristics are presented in Annex 5: Case Study selection.

Wave 1 fieldwork included interviews with strategic leads and local WFWF leads (see Table 1). Strategic leads who were interviewed were typically directors/chief officers or senior managers within children, families and justice services. Local WFWF leads interviewed were typically within the social work team or service managers with some responsibility for WFWF (or an element of funded activity) in their CSPP. The interviews aimed to understand how services were intended to be delivered, to explore opportunities and challenges to delivery, and to contextualise the experiences reported by other staff and families through the Wave 2 fieldwork.

Wave 2 fieldwork included focus groups with frontline staff, interviews with children, young people and families accessing WFWF support and interviews with strategic leads. Frontline practitioner focus groups explored progress towards early outcomes, the mechanisms of change and practical lessons learned for bringing about longer-term outcomes and sustaining benefits. Interviews with children, young people, and families explored their experiences of WFWF design and delivery engagement, and WFWF support. Interviews with strategic staff explored their experiences of the funding over time and any outcomes they had seen in their CSPP to date.

Note: Fieldwork was completed with additional strategic leads and WFWF leads as it was agreed with CSPPs that multiple staff could attend the interviews to provide adequate coverage of interview topics. Fieldwork with frontline practitioners and children, young people, and families was completed with fewer participants due to one CSPP feeling they had not sufficiently progressed their delivery to take part.

Analysis of the case study data was a continuous process (during and after fieldwork periods, and between phases) and iterative, moving between the data, research objectives and emerging themes. Data from case study interviews was entered into an analysis framework structured by key research questions and themes. The findings were systematically summarised by the research team and then triangulated through analysis sessions where key findings were discussed, and emerging themes and insights were tested.

Secondary data analysis

The Scottish Government developed a template for CSPPs to provide updates on their WFWF plans. The completed template was intended for CSPPs to include as part of their annual CSP reports for 2022-23. The template aimed to gather information about CSPPs’ WFWF activities to date, evidence of progress towards outcomes, funding spent, and the key challenges and successes they had experienced in delivering their WFWF activity. Annual WFWF progress templates were returned by 21 of 30 CSPPs, within the timeframes for analysis. The information provided in the reports was synthesised into a framework and analysed using thematic analysis to draw out key themes aligned to each of the relevant research questions.

Evaluation considerations

The WFWF and evaluation were undertaken within an ever-changing landscape. The ongoing post-pandemic recovery, and the cost-of-living crisis impacted family needs and services during the period of the evaluation. Also, in March 2023, Scotland had a change in First Minister and Cabinet responsibilities, which may have influenced policy decisions.

CSPPs were all at different stages of planning and delivering WFWF activities during the period of this evaluation. This meant their level of engagement with some audiences (especially with children, young people, and families) varied. In some CSPPs, frontline practitioners did not feel able to comment or provide observations on the WFWF, because activities were only in early stages. Moreover, children, young people and families’ engagement in design was low in some areas, so it was not possible to include their contributions in the evaluation. Data collection was adapted, by mutual agreement, to align with the stage of delivery more closely in one case study area, as activities with frontline practitioners and children and families had not yet begun.

Secondary analysis was limited by the availability and quality of data provided by CSPPs in their initial plans, and CSP annual report templates. Initial plans were provided by 30 CSPPs, and CSP annual reports were returned by 21 CSPPs by the deadline or inclusion in evaluation analysis. The relatively low return rate for CSP annual reports meant that data, especially relating to progress towards early outcomes, was incomplete. Therefore, readers should not generalise conclusions across CSPPs. The quality and completeness of information provided in the CSP annual reports varied, and this limited the scope of analysis. The variation in the information provided included gaps in responses or missing data (e.g. proportion of funding spent); limited detail or explanation (e.g. activities delivered, consultation groups convened); limited or unclear evidence of progress towards outcomes (e.g. presented outcomes they expect to achieve instead of progress towards those); and limited information on lessons learned and future plans for implementation and impact.

Qualitative case study evidence is not intended to imply prevalence but rather to illustrate the range of experiences implementing WFWF and provide depth of understanding. It should be noted that findings from this process evaluation may not be generalisable beyond the particular case study areas explored in this research. In particular, the evidence from case study CSPPs indicated that partnerships with third sector organisations were relatively limited, and in some cases attempts at collaboration had had negative results on relationships. These views typically came from strategic leads and local WFWF leads, and the evaluation did not hear from all third sector partners involved. It is important to highlight that there are good examples of collaboration with the third sector within the wider CSPP landscape, for example case studies from Supporting the Third Sector highlight good practice of collaboration with CSPP partners (third sector being a CSPP partner; Children in Scotland, 2023).

This report is intended to provide readers with a sense of the overall journey of CSPPs in their first year of implementing WFWF. The findings presented here are not exhaustive. Further, time-bound findings and lessons learned were captured in local evaluations and reports, and in knowledge exchange activities facilitated by the Scottish Government.

Structure of this report

This report presents the findings of the Year 1 process evaluation structured around the primary research questions:

• Section 3 presents the initial views of CSPPs on the WFWF funding, their process for developing their initial plans, and their perceptions of the ongoing support they have received from Scottish Government.
• Section 4 reports the priorities outlined by CSPPs, and the approaches taken to developing those priorities such as reflection, consultation, data analysis and involvement of local partners.
• Section 5 includes analysis of the delivery of WFWF to date, such as progress and nature of activities to transform support, and the enablers and barriers experienced.
• Section 6 presents CSPPs plans for monitoring performance of the WFWF, and various challenges identified in meeting reporting requirements.
• Section 7 outlines the evidence demonstrating progress towards early desired outcomes.
• Section 8 provides conclusions from the first-year evaluation and recommendations developed by IFF for Scottish Government and for CSPPs.