Family Wellbeing Partnership in Clackmannanshire: evaluation

2. Evaluation Design and Methodology

Overview of Evaluation Approach, Methodology and Steps

Step 1: Developing an evaluation approach

• Reviewing the literature
• Getting to know the FWP team and understanding their work
• Developing a Theory of Change
• Formulating hypotheses to guide the evaluation

Step 2: Creating the monitoring framework

• Creating an overarching data monitoring framework with outcomes, indicators and data sources
• Collating and mapping existing data sources
• Identifying the most appropriate data collection methods to gather evidence
• Developing a plan for data collection and engagement

Step 3: Collecting the data

• 19 process interviews with key FWP staff and stakeholders
• 31 one-to-one interviews with families receiving support from the FWP
• A data audit and identifying data gaps
• Survey of wider system partners (81 stakeholders were invited to participate and 15 responses received, resulting in a response rate of 18.5%)
• Workshop discussions with key policy makers at a national level
• Additional engagement sessions with families to help fill data gaps

Step 4: Reviewing findings in light of emerging literature and policy learning

• Second literature review around emerging findings
• Reviewing hypotheses in light of evidence collected
• Lesson drawing analysis and synthesis of key findings
• Continuous engagement with council and government partners
• Development of final report

2.1 Evaluation Framework

The evaluation framework for this project is comprised of the following elements:

1) The development of a Theory of Change (ToC) for the FWP, which sets out aims, activities, outputs, assumptions, outcomes and the pathways by which change is expected to occur, alongside a set of hypotheses which offer proposed explanations for why the FWP is leading to change . The Theory of Change and hypotheses are set out below in Section 2.2.

2) The development and implementation of a data monitoring framework to measure outcomes from the FWP, both for the evaluation and to support local partners in tracking user experiences and outcomes. This involved clearly defining key concepts and terms used in the monitoring framework, specifying how data is being collected and from which sources and specifying a sampling and engagement strategy for the purpose of the process and impact evaluation.

3) Conducting a process evaluation of the FWP, looking at implementation and delivery, including how the capabilities approach has guided changes to service design, delivery and support, and how the FWP is experienced by staff and service users.

4) Conducting an impact evaluation that has examined whether and how the FWP is working for families and leading to desired outcomes, and whether there is evidence in support of the hypotheses set out alongside the ToC.

5) A systems evaluation analysis, looking at the complex system dynamics around key causal system relationships.

6) A policy learning exercise, drawing out lessons for future improvements to the FWP in Clackmannanshire and lessons for other areas seeking to employ a FWP approach.

2.2 Theory of Change

We have developed a Theory of Change (ToC) to guide this evaluation project. This is a standard model for evaluating interventions in complex systems. A ToC provides a framework for thinking about how an intervention works, and how it leads to intended outcomes.

Our ToC is modelled through a set of steps, to understand how an intervention is carried out. The steps begin with the context in which the evaluation is taking place, then moves to ‘inputs’ or resources, through to the activities involved, and finally outcomes and vision. Our focus is on understanding an intervention rather than seeking to prove correlation. We have structured our ToC as follows:

Context

The first step in the ToC is to assess the context within which the FWP approach is being implemented. Here, we consider four factors: the policy context, the socioeconomic context, the financial context, and local need in Clackmannanshire.

Inputs

The second step in the ToC is to identify the inputs, that is, the resources that have gone into the FWP. Here, we consider the funding for the FWP approach in Clackmannanshire, training, community engagement, collaboration and leadership approaches.

Activities

The next step is to capture the activities undertaken as part of the FWP approach which we analyse through the four workstreams.

Outputs

We then consider the outputs of the activities at two levels: at the individual/family/community level and at the service/system level.

Assumptions

It is important to identify key assumptions that enable or prevent the activities/outputs having the desired effect (outcomes). Here, we focus on the understanding of system boundaries, policy framing, political legitimacy, system leadership and innovative funding mix.

Outcomes

The next step is to consider the outcomes resulting from the FWP. We consider short, medium and long-term outcomes. Importantly, we divide outcomes into two categories. First, we sought to understand the wellbeing of individuals, families and communities as a result of FWP interventions (the focus of our impact evaluation). Here, we sought to explore (i) how families experience the FWP interventions, and (ii) the outcomes for individual, family and community wellbeing. Second, we sought to understand the transformation of public service delivery as a result of the FWP in Clackmannanshire. This outcome of ‘public services delivery’ is therefore the focus of our process evaluation. Here, we sought to understand the structures, governance, procedures and policies of the FWP project and to assess (i) how the wellbeing and capabilities approach of the FWP is guiding changes in the delivery of public services and (ii) how these changes are experienced by staff and partners.

Vision

Finally, we consider the vision the FWP is trying to achieve in reducing poverty and inequality and creating more resilient communities and the risks and opportunities in achieving this vision.

Table 1: Theory of Change Framework

Context

Policy Context: Complex policy environment, commitment to tackling child poverty & improve wellbeing through prevention.

Socioeconomic Context

Impacts of Covid, cost of living crisis, child poverty, inequality.

Financial Context

Public sector budget constraints, non-recurring FWP funding.

Local Need

Need to build public trust between local communities and services, work with third sector.

Inputs

Funding: From Scottish Government, philanthropy, reallocation of council funding.

Training: In VBL and trauma-informed practice.

Community Engagement: Staff time & investment in initial engagement with communities to understand needs.

Collaboration: Staff / partner time to develop initial collaboration with schools, social work, police, NHS, third sector organisations..

Leadership Approaches: Support to pursue new integrated approaches to service delivery & model leadership behaviours that support system change.

Activities

Community Around The School (CATS):

Work experience “tasters”, university trips, DoE programmes, adult learning and literacy programmes, family cooking sessions, cultural events, school mentoring programmes, breakfast clubs.

Child Wellbeing Partnership (CWP): Breakfast and after-school clubs, holiday provision, food services, transport, access to practical advice on benefits.

Safeguarding Through Rapid Intervention (STRIVE): Coordinated family support with interventions that respond to individual needs, in schools, homes, health centres.

Enhancing Employability: Training and pre-employability projects with flexible work opportunities for parents entering the labour market.

Provide holistic support through educational, childcare, and employability services aiming to meet both immediate and long-term needs.

FWP ensures that once a family engages with one service, they are seamlessly connected to others, promoting comprehensive, relational support rather than crisis-driven responses.

Outputs

Individuals/Families/Communities

• No. families accepting support.
• Length of engagement.
• No. families attending events.
• No. visits with health, statutory, other services.
• No. and type of referrals to housing, employment, education, training, services & benefits.
• No. community resources, including family support, childcare, shared spaces, and the use of these.
• FWP targets families, particularly those facing multiple disadvantages, such as lone-parent families, ethnic minority families, and families in areas of high deprivation like Alloa South and East

Service/System

• No. partners engaged in FWP.
• No. sessions / events run across workstreams.
• New partner collab. agreements.
• Data sharing agreements.
• New policies, guidance and training for orgs.
• No. new FWP services developed.
• Partner engagement in evaluation framework.

Assumptions

Understanding System Boundaries: Awareness of system, diverse perspectives, community needs.

Policy Framing: Clear policy and framing to scaffold services, policies and budgets.

Political Legitimacy: Early political support to ensure operational feasibility and community trust.

System leadership: Strong trust and permission from leaders at all levels to do things differently and to learn and adapt.

An innovative funding mix: To align with national and local policy ambitions.

ST Outcomes (up to 6m)

Individuals/Families/ Communities

• Easier access to the right support for individuals.
• Awareness of benefits, grants and support.
• Ability to budget.
• Awareness of housing standards & requirements.
• Awareness of learning and employment opportunities.
• Increased trust in services.
• Skills and confidence developed through access to integrated services.
• Immediate / basic needs of families met, including debt, food, energy, housing.
• Family participation in some community activities.
• Families are able to manage their own health and wellbeing.
• ➜Families have an increased level of trust in services.
• Prevention of crises.
• Reduced dependency on crisis interventions.

MT Outcomes (6-36 m)

Individuals/Families/ Communities

• Improved confidence and hope for the future.
• Increased engagement with a variety of services.
• Willingness to engage in new social interactions.
• Individuals and families have more social relationships.
• Individuals and families report an improved sense of support.
• Improved attendance at school, educational outcomes.
• Effort to stay engaged with services over time.
• Improvements in underlying issues, i.e. financial stability, mental health, addressing employability barriers through childcare, transport.
• Individuals develop greater agency and resilience.
• Families engage more actively within communities.
• Families report improved wellbeing, including mental, social, financial and emotional wellbeing.
• Individuals and families are better able to manage their money.

LT Outcomes (4+ yrs)

Individuals/Families/ Communities

• Sustained personal agency & empowerment.
• Improved life outcomes, including employability and vocational training.
• Individuals have greater resources and resilience.
• Children grow up in more stable, supportive environments.
• Families experience long-term financial stability through access to employment and training opportunities.
• Individuals & families have a stronger sense of belonging and become active in the community.
• Families report improved wellbeing, including mental, social, financial and emotional wellbeing and are better able to maintain their wellbeing.

Vision

Poverty and inequality are directly and consistently addressed and reduced through inclusive, preventative, relational and sustainable public services.

A resilient community ecosystem is developed, with the agency and voice of residents directing the design and operations of services, ensuring sustainable community connection and lifelong impacts.

The table above sets out the abridged Theory of Change (ToC) Framework for this evaluation project. The full narrative ToC can be found at Annex B.

2.2.1 Explaining why the FWP has effected change: hypotheses

As we saw in the Theory of Change framework, the FWP involves a range of inputs (resources) and activities (things the FWP does) that aim to affect outcomes for families and communities.

Our next line of inquiry is: how is the FWP in Clackmannanshire affecting these outcomes for people, and why? Why do certain activities lead to certain changes? To help us answer these questions, we developed a series of research hypotheses — that is, some proposed explanations for why the FWP is leading to change — to help guide our evaluation project. We developed these hypotheses at the start of the project, gathered evidence throughout the project, and sought to evaluate whether these hypotheses had been supported through the data collected at the end.

These hypotheses have been important in guiding the data collection and analysis for this project and in structuring the analysis for the workstream case studies below.

Hypotheses

We have developed 16 hypotheses to theorise the impact of the FWP on the key outcomes (people’s wellbeing and public services). These hypotheses are grouped into three themes, which capture how the FWP affects (i) individual, family and community wellbeing, (ii) changes in public service delivery and (iii) overall learning outcomes.

Individual, family and community wellbeing

H1 The person-centred, place-based approach to family wellbeing enables the formation of trusting relationships between families and FWP staff, allowing families to access the support they have needed previously.

H2 The integration of support services within the FWP results in improvements in the wellbeing of children and families, evidenced by increased school attendance, better educational outcomes, improved wellbeing indicators and reduced dependency on acute services.

H3 The FWP approach allows individuals and families to more easily access services and engage with them over time, improving their confidence and willingness to engage with services and to explore new educational, training and skills development opportunities.

H4 Individuals have greater resources, knowledge and resilience to make decisions and shape their futures based on their evolving capabilities, values, life goals and aspirations, which leads to improved life outcomes, including employability, education, and financial stability.

H5 The development of CATS leads to more confident children and young people, with an awareness of different sources of support and the opportunities available to them and a strong sense of their individual strengths and skills to thrive in life.

H6 The removal of barriers to vocational training and tailored employment opportunities, through childcare support, transportation and other services, leads to higher employment rates, greater financial stability for families, a reduction in poverty and the increased employability of individuals.

H7 Individuals and families actively participate in FWP activities, leading to higher levels of social cohesion, reduced feelings of isolation, and a stronger sense of belonging among residents.

H8 As power and agency are transferred to children, families and the frontline staff who support them, and there is more active involvement of citizens in decision-making processes, there is a greater sense of community empowerment. This ensures that services are responsive to local community needs, leading to higher satisfaction and effective outcomes.

Public service delivery transformation

H9 There is a shift to a more preventative, integrated, and multidisciplinary model of public services, with more partners adopting this model, which results in a reduction in public service crisis interventions, improved efficiency in service delivery, and a more timely and coordinated level of support for families.

H10 Through its preventative and person-centred approach, FWP staff and partners learn how to work more effectively with families and communities, to build trust, and to meet their needs.

H11 The strong trust, permission and values-based messages from leaders at all levels for staff to do things differently, in developing new joint working arrangements across services, leads to improved cross-agency communication, more empowered staff, a greater focus on reflective practice, and the development of shared values and commitments to outcomes.

H12 Ambitious and preventative policy approaches to reducing child poverty and improving collective wellbeing, implemented through efficient and integrated public services, leads to improved data sharing, deeper collaborations across services, greater policy coherence and financial alignment across funding streams.

H13 The development of a fully integrated, preventative FWP approach with robust levels of leadership and commitment by all service delivery partners creates a sustainable and flexible delivery model that can evolve and adapt to community needs into the future.

H14 Person-centred, inclusive, and equitable public service approaches to reducing poverty and inequality in Clackmannanshire are adopted by a greater number of partners, becoming the norm and creating sustainable systems change.

Lesson learning

H15 Clackmannanshire becomes a national model for public service transformation where public services are preventative, integrative and co-designed with the community.

H16 The flourishing of individuals, families and communities in Clackmannanshire and the development of new ways to support their wellbeing and reduce poverty provides a model for other communities around Scotland facing similar challenges and opportunities.

Assumptions, risks and opportunities

We also anticipated that there would be intervening factors that affect the relationship between the FWP intervention and its intended outcomes for public service transformation and individual, family and community wellbeing.

These assumptions and risks can act as enablers of, or obstacles to, change. They can therefore be seen as both risks to, and opportunities for, the FWP in Clackmannanshire. This is not an exhaustive list, however, we assume the following factors may have an impact:

Factor 1: Changing policy context (with potential changes in support for whole-family approaches)

Factor 2: Levels of long-term funding available (with potential changes in funding available to support the ongoing programme of FWP work)

Factor 3: External political, social, cultural and economic changes that affect the preconditions for transforming FWP activities into outcomes. This could include significant changes to policy in health, education, housing or other areas that affect family wellbeing, such as a change in the two-child benefit cap; changes in population health, such as new pandemics or as a result of the regulation of unhealthy foods that could improve health; systemic changes in community relations, such as changes in levels of community cohesion/hate crimes; and macroeconomic changes that could affect (positively or negatively) interest rates and the job market, such as the impacts of wars on oil prices and a cost-of-living crisis.

Approach

Throughout this project, evidence was gathered through a range of methods to determine whether the hypotheses were supported.

This evaluation project adopted a complex systems approach to understanding the impact of the Family Wellbeing Partnerships (FWP) on service delivery and family wellbeing.

We employed a case study methodology to explore the implementation and impacts of each of the four workstreams (STRIVE, Child Wellbeing Partnership, Enhancing Employability and CATS) on the wellbeing of people who have received FWP support. The case study method enables a greater understanding of the complexity of context in which the place-based FWP interventions have taken place, a full exploration of how these workstream activities have affected different parts of the FWP system (e.g. managers, practitioners and organisations working with the FWP) and their interrelationships and how the system has adapted. These findings are set out in Chapters 3 to 6.

As with all complex human systems, understanding the FWP is best approached through a living systems lens, rather than a mechanistic or linear model. “In contrast to machine-like systems, living systems are dynamic, relational, and capable of continuous learning and adaptation”.^[23]

Drawing on these perspectives, alongside the work stream case studies the evaluation also uses System Evaluation Analysis to map out features and relationships within the wider system of family support within Clackmannanshire. This approach supports an exploration of the underlying values, power dynamics, and assumptions embedded in how interventions are designed and delivered - recognising that the social systems within which the FWP operates are not static but constantly co-evolving in response to internal and external interactions.

By working with a living/ system frame, the evaluation is able to identify critical process touchpoints - areas where interventions have supported or constrained system change. As Margaret Wheatley highlights these “touchpoints are not merely technical levers, but moments of interaction where relationships, trust, and meaning-making shift. They illuminate how change happens (or stalls) not as the result of isolated actions, but through patterns of connection and feedback across the systems”^[24]. These findings are elaborated in Chapter 7, which highlights the places where the FWP’s design and delivery intersected most powerfully with the wider system to foster or hinder transformation.

2.3 Data Monitoring Framework

For this evaluation, a data monitoring framework was constructed to track the progress, effectiveness and outcomes of the FWP at both a workstream and system level. The framework is also designed to foster a culture of continuous learning and improvements by supporting local partners to fill data gaps and measure future outcomes. Below, we provide an easy-read visual of how the data monitoring framework was organised and how it may be used going forward:

Table 2: Data Monitoring Table Structure

Theory of Change Outcome	Indicators (measure outcome)	Data collected by the FWP workstream?	Data Source	Data Collection Tool	Data collected by the Local Authority?	Local Authority Outcomes (alignment)	Local Authority indicators
Short, medium or long-term outcome	E.g. no of participants or partners involved in specific actions/ activities	Yes/No	E.g. reports, case studies, evaluations, other documents	E.g. Surveys, interviews, focus groups, literature reviews, sign-ups, testimonials	Yes/No	Key objectives for LA, E.g. reducing child poverty rate, increasing employment	E.g. no of participants or partners involved in specific actions/ activities

In terms of service user outcomes, the monitoring framework was designed to enable a granular understanding of the characteristics of the people involved in the FWP, how they are engaging with services and support, and what outcomes are being achieved. The framework also gives an additional understanding of how the capabilities approach is being implemented and how it is leading to any outcomes for those involved. The framework builds upon the foundation of the SIP evaluation framework^[25] which set out an approach to measuring change in individuals’ capabilities across two dimensions:

• ‘Being’ capabilities, including measures such as confidence, self-awareness and mental wellbeing
• ‘Doing’ capabilities, which result from the improvements in the ‘being’ capabilities, such as moving towards employment or forming positive social relationships

At the system level, the framework supports the measurement of changes in service design and delivery and identifies indicators to measure system change goals, as set out in the ToC.

The data monitoring framework was developed using the following steps:

1. Identification of key outcomes and deliverables for each of the four workstreams

2. Discussions with key stakeholders to map current data sources, outcomes that had already been measured, and indicators to assess further outcomes

3. Desk-based mapping of agreed outcomes (based on the ToC) and indicators to data sources and project data collection methods

4. Continued engagement with stakeholders to ensure continuous improvement and longevity of the data monitoring framework

It is anticipated that the data monitoring framework will be used by the FWP as an ongoing tool to support monitoring and evaluation work. Limitations are considered later in the chapter.

2.4 Evaluation Methods

Below, we describe our approach to engaging with potential participants in the evaluation project.

Stakeholder mapping

During the initial stages of this work, we developed stakeholder maps that set out the key partners and participants who have been involved in the FWP since 2020. This mapping created an overall picture of the different actors involved in the development of the FWP, provided a range of people to draw on when sampling for fieldwork and helped determine the boundary of the system being evaluated. A summary of the key partners is illustrated in each of the four workstream chapters (Chapters 3 to 6).

Sampling strategy

The process and impact evaluations ran simultaneously in order to reduce the burden on participants and mitigate consultation fatigue. We used a purposive sampling strategy to interview project and council staff and a mixed sampling strategy (purposive and opportunistic) to interview individuals and families using FWP services. This included, for instance, identifying families to interview in advance through the guidance of workstream project staff as well as holding ‘drop-in’ interviews after school. This mixed approach was used to maximise opportunities to speak to families at times and in places convenient to them. The sampling strategy ensured a diverse range of views from different family types (as shown in table 3), work streams and staff types.

Engagement strategy

As suggested by our sampling strategy, we sought to maximise our opportunities for engagement across all levels of participation in the FWP. A key aim of the process evaluation was to speak to staff members at both the system and service level, whilst for the impact evaluation our focus was on speaking to families and individuals who had accessed FWP services. We opted to use primarily one-to-one semi-structured interviews rather than focus groups, as they allowed for more in-depth and intimate discussions about their experiences of the FWP. Participants were asked to complete an informed consent form prior to interviews taking place (see Annex D).

Process Evaluation

The focus of the process evaluation was to explore how FWP services were designed, implemented, and delivered from the inception of the FWP to the present day (2020-2024). The evaluation assessed whether services were delivered as intended, aligned with the ToC, and if they resulted in anticipated outputs. This stage involved extensive fieldwork in Clackmannanshire, including weekly trips to different parts of the council to assess the four workstreams with a focus on identifying both successes and areas for improvement. The process evaluation included the following research questions:

Key questions regarding service user experiences:

• How are families recruited and what are the enablers and barriers to engagement?
• What needs, aspirations, and experiences do families report regarding FWP services?
• To what extent have priority groups been reached?
• How do families perceive FWP support compared to previous interventions, and do they feel it is effectively meeting their needs?
• What does the support journey look like, and how can delivery be improved?

Key questions regarding systems change and public services delivery:

• What system changes does FWP aim to achieve, and how are these implemented?
• How effectively has the ‘capabilities approach’ been embedded in service design?
• Has the partnership been implemented as planned?
• What has worked well, and where is improvement needed in service delivery and collaboration?
• Have values, cultures, and behaviours changed across departments?
• How sustainable are the observed changes in service delivery?

Impact Evaluation

The aim of the impact evaluation was to assess the FWP’s influence on family outcomes, focusing on wellbeing, capabilities, and any changes in the systems and services that support them. This stage overlapped with the process evaluation, using a mixed-methods approach to evaluate outcomes and align findings with the Theory of Change. The impact evaluation included the following research questions:

• What positive outcomes or changes in capabilities have participants experienced?
• Which groups benefited most (or least) from the initiatives?
• How have the workstreams collectively supported family outcomes?
• What aspects of FWP support have been most effective?
• Do participants anticipate long-term benefits from FWP engagement?

Methods and Data Collection

A range of primarily qualitative research tools and methods were used to collect the data required for the evaluation. Data was collected for each of the four case study workstreams (STRIVE, the Child Wellbeing Partnership, Community Around The School, and Enhancing Employability). We sought to collect comparable data, where possible, across each of the workstreams.

In total, 31 families and 19 stakeholders were interviewed across the FWP workstreams with questions designed to evaluate both process and impact outcomes. This involved the following data collection methods:

1. Document review: collection and analysis of workstream-related documentation such as delivery plans, recruitment strategies, needs assessments, resourcing details, and user satisfaction reports. Gaps in data collection were noted, particularly gaps in anonymised satisfaction data and consistent baseline metrics. Any existing evidence of changes in service delivery, family satisfaction and well-being outcomes was also collected.

2. Research interviews - stakeholders: we invited 29 stakeholders to participate in a one-to-one semi-structured research interview and 19 agreed to take part. This included practitioners, service staff, and senior leaders involved in FWP delivery. The interview covered both process and impact evaluation questions.

3. Research interviews - service users: 36 families were highlighted by FWP and invited to participate in interviews and 31 agreed to take part in semi-structured interviews. Participants were interviewed either face to face or via a phone call.

4. Survey: we issued an electronic survey in January 2025 to 81 wider staff within the council and third sector and statutory partners associated with the FWP, including service delivery staff, partners and senior leaders, to capture perspectives on implementation challenges, collaboration, and service impacts. In total, 15 responses were received, resulting in a response rate of 18.5%. As previously stated, the data collected through this survey is being used as qualitative only due to the lack of a representative sample.

Table 3: Participant involvement in evaluation of the FWP (across workstreams)^[26]

Types of participants	Numbers of participants
Single parents	10
Dual parents	10
Families with children under 2	4
Young People (under 18)	7
Managers involved in FWP	11
Practitioners involved in FWP (Council and third sector)	6
Other FWP partners	2

2.5 Data Quality and Limitations

An overview of the data collected for this evaluation is presented in Tables 4 and 5 below.

Table 4: Process Evaluation Data Sources

Qualitative interviews with stakeholders - 19

Monitoring data - Analysis of workstream case notes, quarterly operational updates, reports

Table 5: Impact Evaluation Data Sources

Qualitative interviews with families - 31

Stakeholder survey - 15 responses

Case studies - Case studies of individuals receiving support gathered by workstream or conducted by previous evaluators after time of receipt of support.

Monitoring data - Analysis of workstream case notes, quarterly operational updates, reports

Wider trends - Data on wider trends relevant to FWP activities (employability, school attendance, probable suicides, domestic abuse, and homelessness) is drawn upon to illustrate Clackmannanshire-wide trends since the onset of FWP.

2.5.1 Strengths and limitations of the data

As demonstrated in the table above, this evaluation project collected qualitative evidence from multiple sources (service users, project staff, council staff) using different methods (interviews, surveys, literature review) to develop an evidence base for our findings. We used this data to evidence progress against the ToC outcomes for each of the four FWP workstreams. However there is very limited quantitative data available that tracks outcomes for individuals who have come into contact with FWP services, which means that we are not able to state the extent of impact the FWP might be having for families, although the qualitative data shows a range of ways that the FWP is impacting.

Moreover, there is also very limited data available (qualitative or quantitative) that captures longer term outcomes for families. While short-term trends can be observed through population level data in relation to some key indicators with relevance to FWP workstreams, such as school attendance and attainment rates, it is difficult to ascertain longer-term trends as the FWP is only in its fourth year. The lack of data on long-term outcomes makes it difficult to categorically state whether the FWP has made a positive long-term impact on the wellbeing of individuals and families or has led, for instance, to a significant increase in employment rates or a reduction in child poverty.

Finally, there is a lack of evidence regarding whether particular programmes have led to measurable outcomes at the level of the whole community. In particular, the project found that the measure of ‘community wellbeing’ can be challenging to define, quantify and measure. Although it is clear that some individuals who have taken part in FWP activities are reporting a large improvement in their own wellbeing, it is difficult to assess whether this is having an impact at a community level, or transferring into wider social relationships.

2.5.2 Strengths and limitations to the data monitoring framework

In its early stages, the FWP adopted an approach to data and monitoring which prioritised understanding local needs and capturing nuanced impacts, borne out of a sense that many of the FWP’s impacts as experienced by local families may resist easy quantification, and a feeling that the challenge of starting from a low baseline for many indicators made comparisons to national averages an unhelpful measure of progress. Hence the data monitoring framework developed for the evaluation has been retrospectively applied to the current data sources available within Clackmannanshire.

There are strengths and limitations to this approach. On the one hand, our relationships with key staff members within the FWP and wider Local Authority have allowed us to collect and review a wide range of data sources, some of which are both historic and unique (e.g. 2002 trauma survey, and the Mental Health and Wellbeing Survey which ran yearly from 2020 - 2024) and by working closely with FWP staff, we believe the data monitoring framework can be used for future monitoring efforts and tracking against specific outcomes.

On the other hand, it is evident that there are many data gaps in the framework, especially in relation to evidencing long term outcomes. Whilst some short and medium term outcomes are able to be evidenced through the various data sources collected by the FWP work streams, evidencing long term outcomes has proved challenging. This may be due to the challenges of monitoring long term outcomes (for example the difficulties of primary data collection), and the fact that some workstream activities have only been operational for 2-3 years. Despite this, however, there is the opportunity to further develop data monitoring efforts to target key long term outcomes which are not currently collected at service and workstream level, such as measuring material improvements (such as benefits accessed or employment opportunities) and to further strengthen the amalgamation of data sources relating to, for example, mental health and wellbeing outcomes over a longer period of time to capture the holistic, whole-systems approach being taken by the FWP.

2.6 Summary

This chapter has set out our overall approach and methodology for undertaking the evaluation of the FWP. A range of primarily qualitative methods were used to gather data for the evaluation, including the collection and analysis of workstream-related documentation, interviews with key FWP staff and families receiving support from the FWP, and a survey of wider FWP stakeholders. A data monitoring framework was also developed to measure outcomes of the FWP for the evaluation and support local partners in tracking user experiences and outcomes. Challenges were faced in applying this data monitoring framework retrospectively, and there were significant gaps in data.

Although the FWP has increasingly been viewed as an overall way of working rather than a set of specific projects, the evaluation has focused primarily on the four main workstreams that were initiated in 2020 and tracks their progress through to 2024. Each workstream chapter begins with high-level process evaluation findings, examining implementation and function in practice. It identifies key mechanisms, activities, and contextual factors behind successes and challenges. Each chapter also includes an impact evaluation using the Theory of Change framework to assess effects on service users and partnerships. Following this, Chapter 7, using systems analysis, explores the FWP overall approach in greater depth, examining its interconnectedness and interdependencies across the system. Chapter 8 then synthesises key findings; Chapter 9 presents key policy learning; and Chapter 10 concludes.