Whole Family Wellbeing Funding (WFWF) - year 1 process evaluation: final report – annexes

Annex 5: Research approach

The Scottish Government commissioned IFF Research in September 2022 to undertake an evaluation of the implementation of Year 1 of Elements 1 and 2 of WFWF (a ‘process evaluation’^[5]) to ensure that lessons are learned to inform future policy and practice in the CSPPs.

Analysis of initial plans

Initial plans are the first documentation of how CSPPs intended to spend their WFWF allocation for 2022-23. Plans include detail on their current approach to whole family support, planned activity for WFWF, anticipated outcomes for the first year, and intended monitoring activities.

CSPPs’ initial plans vary greatly both in the level of detail included and in their maturity. This is expected and understandable, given the early stages of the WFWF. The Scottish Government recognised in the design of the WFWF that CSPPs are all in different places on their journey to delivering holistic whole family support.

Our analysis included 30 CSPP WFWF initial plans for Element 1 funding. Two initial plans had not been submitted to the Scottish Government in time for the plan analysis to be conducted.

After reading and synthesising the information into the initial plan framework (a tool designed to capture key information relating to the research questions), the following descriptive analysis was carried out:

• What are the characteristics of CSPPs receiving funding?
• Who are the targeted beneficiaries?
• What is the main purpose of funding?
• What are the secondary purposes of funding?
• What are the activity types funding will be used for?
• Transformation implementation status – first steps, early progress, substantial progress?
• Average, highest and lowest value requested?

For all of the above, whether/how this varies by CSPPs characteristics.

Analysis of CSP annual reports

As part of the local authority obligations under the Children and Young People (Scotland) Act 2014, Children’s Services Planning Partnership were required to prepare a Children’s Services Plan for 2020-2023. The CSP annual reports are a summary of progress for that given year. CSP annual reports for 2022-23 were submitted to the Scottish Government, who shared these with IFF in July/August 2023. Alongside the statutory CSP annual report, IFF and the Scottish Government worked together to develop a template specifically focussed on the WFWF. It was not mandatory to complete this and 21 CSPPs chose to do so.

On receipt of CSP annual reports, an internal review and quality check on the data was undertaken to ensure it is complete, consistent and ready for analysis. Once checked and approved, the data was entered into a bespoke framework for qualitative analysis.

Secondary analysis was limited by the availability and quality of data provided by CSPPs. 21 CSP annual reports were provided to IFF for analysis. The information provided in the CSP annual reports was limited in some places which reduced the scope of analysis that could be undertaken. Common challenges included:

• Gaps or missing data such as not providing figures for funding spent on activities.
• Lack of detail or explanation, such as not providing sufficient description of activities e.g. description of programmes or groups convened.
• Lack of or unclear evidence of progress towards outcomes. For example, failing to explain through what mechanisms an outcome had been achieved or the extent to which it had been achieved.
• Lack of reflection on lessons learned and future actions.

Case Study selection

Six CSPPs were selected to be case study areas in the Year 1 Process Evaluation. These were Glasgow City, East Ayrshire, North Ayrshire, Aberdeen City, Fife and South Lanarkshire. Of these, Glasgow City and East Ayrshire were also selected to participate in Element 2 collaborative support.

CSPP case studies were chosen to provide coverage across: funding amount, whether they participate in Element 2 collaborative support (East Ayrshire and Glasgow City both received Element 2 support), area type (predominantly urban or rural), population, deprivation rank, needs of the family population (defined by the rate of children on the child protection register), type of activities outlined in their initial plans, and holistic whole family support journey stage (whether CSPPs were at an early, moderate or advanced stage in transforming their support).

Characteristics of Case Studies

Table 1 below shows the profile of each case study CSPP.

Interviewed population by case study

The table below shows the numbers of interviews conducted with each audience within each case-study CSPP.

Children, young people and families participant profile

The table below shows the profile of children, young people and families who took part in qualitative interviews across the six CSPPs.

Qualitative data management and analysis

All discussions were recorded with consent, stored on IFF’s secure drive in a folder to which only designated team members had access, and written up thematically by the researcher using a bespoke analysis framework.

IFF’s qualitative analytical approach is informed by grounded theory and structured by the research questions but builds upwards from the views of participants. It is continuous (during and after fieldwork periods, and between phases) and iterative, moving between the data, research objectives and emerging themes.

The analysis framework was structured by key research questions and data entered into relevant cells including direct quotes and examples. It included columns for the researchers’ own interpretation and key conclusions. Data was then coded, looking for patterns by theme within and across interviews.

The analysis process consisted of two key elements. Firstly, a process-driven element using matrix mapping framework technique. Recordings of discussions coded and systematically summarised into an analytical framework organised by issue and theme. Secondly, an interpretative element focussed on identifying patterns within the data and undertaking sub-group analysis. Researcher analysis sessions, led by the director, during which the team came together to discuss and test emerging themes and insights, were conducted after each phase and used to support interpretation of the data.

All evidence sources were analysed in their own right; the analysis process then went on to compare and contrast the findings across evidence sources. During this, the quality of evidence was weighed up. Any inconsistencies between different data sources were explored and explained. Where there were competing findings by evidence source, stronger evidence was considered over evidence with gaps.