Five Family Payments: evaluation

Methodology

This chapter provides an overview of the evaluation approach for the Five Family Payments. It introduces the logic model and research questions, and gives a summary of the different data sources used to evaluate policy outcomes.

Overview of evaluation design and logic model

The Five Family Payments policy evaluation is based on a theory of change logic model. The model shows the mechanisms whereby interventions (such as Scottish Child Payment, Best Start Grant, and Best Start Foods) have a chain of immediate, short-term, and medium-term outcomes. These outcomes are linked with the overall aims of the benefits. Specifically these are to (a) help people on low incomes with the costs of supporting a family, (b) reduce financial pressures on low income families at key transition points in their children’s lives, and (c) to provide financial support for low income families to access affordable, nutritious food at the point of need. If the outcomes are met, they can contribute to long-term Scottish Government policy impacts.

The Scottish Government’s long-term policy impacts (e.g. reduced child poverty) will take time to determine, and may require access to robust quantitative data that is not currently available. These will be affected by a range of interventions in addition to the Five Family Payments, making it difficult to measure and attribute changes specifically to these benefits. However, achievement of short-term and medium-term policy outcomes (e.g. increased child-related spend and reduced material deprivation) could reasonably be expected to contribute to these wider outcomes.

The logic model for the Five Family Payments is shown in Figure 1 below. Note that while some of the logic model outcomes relate to Scottish Child Payment, the three Best Start Grants (Pregnancy and Baby Payment, Early Learning Payment, and School Age Payment) and Best Start Foods, others relate to only one or two of these benefits. Immediate outcomes relate to the delivery process – i.e. promoting and administering the benefits. The short-term and medium-term outcomes relate to policy outcomes, which are linked with the intended aims of the Five Family Payments – i.e. the impact of receiving payments.

Immediate outcomes

• The benefit is well promoted (All benefits)
• The benefit and its eligibility criteria are well understood (All benefits)
• The benefit is taken up (All benefits)
• Making an application is clear and easy (All benefits)
• Applications are processed in a timely manner (All benefits)
• Awareness is raised about other forms of support (All benefits)
• Clients feel they have been treated with dignity, fairness and respect (All benefits)
• Card reduces stigma and is easy to use (Best Start Foods)
• Card provides access to a range of retailers (Best Start Foods)
• Card provides access to a range of healthy foods (Best Start Foods)

Short-term outcomes

• Increased child-related spend (All benefits)
• Reduced pressure on household finances (All benefits)
• Reduced money-related stress (All benefits)
• Children able to participate in social and educational opportunities (Scottish Child Payment, Best Start Grant)
• Improved position of main carers within households (Scottish Child Payment)
• Grant reaches people at key transition points in child’s life (Best Start Grant)
• Families financially supported at key transition points (Best Start Grant)
• Healthy foods are more affordable (Best Start Foods)
• Mothers and children eat more healthy foods (Best Start Foods)
• Supports healthier shopping habits and meal planning (Best Start Foods)

Medium-term outcomes

• Improved health and wellbeing (All benefits)
• Reduced incidence of material deprivation (All benefits)
• Positive impact on Scottish economy (All benefits)
• Reduced incidence of debt (All benefits)
• Reduced incidence of food insecurity (Scottish Child Payment, Best Start Foods)
• Reduced barriers to education and labour market (Scottish Child Payment)
• Increased healthy eating behaviours (Best Start Foods)

Long-term outcomes and impacts

The long-term impacts in the logic model (reduced child poverty, reduced inequality of outcomes for children, reduced incidence of social exclusion, and reduced health inequalities) relate not only to the Five Family Payments but to the wider government outcomes for children and families, and are influenced by all social security interventions, as well as other interventions designed to support families with children across the Scottish Government. As such, the Five Family Payments will play an important, but not exclusive, role in contributing to these.

Evaluation questions

Below are the questions that informed the evaluation design:

1. To what extent have the Five Family Payments achieved their immediate, short-term and medium-term outcomes?

a. How well have the Five Family Payments been promoted and administered?

b. What impact are the Five Family Payments having on families and children?

c. How do outcomes vary for families and children with different demographic or household characteristics (particularly the priority families at risk of poverty)?

2. What evidence is there of the Five Family Payments contributing to the long-term government policy impacts of (a) reduced child poverty, (b) reduced inequality of outcomes for children, (c) reduced incidence of social exclusion, and (d) reduced health inequalities.

3. What issues have emerged from the evaluation findings that could inform future policy development?

Summary of data sources

In accordance with the evaluation strategy^[16], the evidence used in this report has been drawn from multiple data sources. Data sources that are used extensively are described below. Other sources that are used less frequently are named and cited in the findings section. These include reports by the Scottish Government and also by external organisations.

Bespoke commissioned research

ScotCen was commissioned by the Scottish Government to conduct mixed methods research with Five Family Payments recipients and stakeholders who support low income families. This involved (a) a survey of recipients, followed by (b) interviews with recipients and stakeholders. The full commissioned research report from ScotCen, with more detailed findings and further methodological information, is available at Annex A.

The commissioned research largely focused on the impact of receiving the Five Family Payments. This provided evidence on short-term and medium-term policy outcomes set out in the logic model above. However, it also explored factors relating to the delivery of the benefits, such as the application process and possible barriers to take-up. This provided a degree of evidence on immediate process outcomes in the logic model.

Survey of Five Family Payments recipients

The survey of Five Family Payments recipients was conducted online between October and November 2024. Its purpose was to determine (a) the extent to which the payments are helping families and children in relation to factors such as debt, material deprivation, food security, and health and wellbeing, and (b) whether certain subgroups of recipients experience different outcomes, based on key demographic and household characteristics.

The survey questionnaire was emailed to 51,500 Five Family Payments recipients across Scotland, in all Local Authority areas. A total of 3,922 recipients responded to the survey, of whom 3,407 received Scottish Child Payment, 523 received Best Start Foods, and 1,922 had received at least one Best Start Grant payment.^[17] The survey data was weighted to compensate for an overrepresentation of older respondents (aged over 35) relative to the general population of Five Family Payments recipients. The analysis was performed on weighted data, and included the following statistical procedures:

• Frequency analysis, to show how all respondents answered each question in the survey
• Crosstabulation analysis, to show how different subgroups of respondents answered each question in the survey
• Significance testing, to determine if differences between subgroups of respondents are representative of real differences in the population, with regards to how they answered the questions in the survey.^[18]

The subgroup analysis focused on the priority families who are most at risk of poverty, the Scottish Index of Multiple Deprivation (SIMD)^[19], and how long people had been receiving the benefits. More detail on the subgroups is shown in Table 1. It should be noted that base sizes (i.e. response numbers) were too small to conduct subgroup analysis for mothers aged under 25 – i.e. the sixth priority family group.

In addition to the analysis described above, which was conducted as part of the bespoke research contract (and reported in full at Annex A), further analysis of the survey data was subsequently carried out by Scottish Government researchers. This was done to contextualise (and aid interpretation of) key trends that emerged from the initial subgroup analysis. It involved the same statistical procedures outlined above. It focused on: (a) an analysis of a supplementary question in the survey about the financial situation of respondents, and (b) an examination of the relationship between the subgroups in Table 1. Results of the further analysis are presented in Appendix A, and discussed in the ‘discussion of progress towards Five Family Payments outcomes’ section towards the end of the Findings chapter.

Please note the following technical points about how the commissioned survey findings are presented in this report:

• The number of respondents providing a valid answer to each individual question/statement varied slightly, within the ranges shown.
• While the analysis was performed on weighted data, the reported number of responses to each survey question (as shown in tables, graphs, and in the commentary) refers to the unweighted base of respondents
• Most results are presented to zero decimal places. ‘0%’ should therefore be interpreted to mean less than 0.5%. If no responses were given then this is denoted by ‘-’.
• ‘Not applicable’ responses to questions have not been included in graphs and tables, unless otherwise stated. This means that results in tables and graphs may not equal 100%
• Subgroup differences are only reported in cases where they are statistically significant, unless otherwise stated.

Interviews with Five Family Payments recipients and stakeholders

The qualitative interviews with Five Family Payments recipients and stakeholders were conducted between October 2024 and January 2025. Their purpose was to complement the survey findings, by providing rich detail on the impact of the payments, in relation to factors such as debt, material deprivation, food security, and health and wellbeing. They also explored aspects of delivery with recipients and stakeholders, such as the application process and barriers which may be impacting take-up of the benefits.

A total of 33 Five Family Payments recipients took part in the interviews. These participants had completed the survey questionnaire, and were sampled on a diverse range of characteristics, including being in a priority family most at risk of poverty, SIMD, and length of benefits receipt. A total of five stakeholders from organisations that support low-income families took part in interviews. All interviews were conducted online or on the telephone. The transcribed data was anonymised and then analysed using NVivo.

Official Statistics

Social Security Scotland collects information on applications, payments, and clients in the process of delivering the benefits. Some of this information is published online as Official Statistics. The following Official Statistics publications are used as sources of evidence in this report:

• Scottish Child Payment statistics to 31 March 2025
• Best Start Grant and Best Start Foods statistics to 31 March 2025
• Social Security Scotland client diversity and equalities analysis from April 2023 to March 2024

The first two of these publications are used most frequently, and are referred to in the findings chapter as ‘Official Statistics’. The client diversity and equalities publication, which is also an Official Statistics publication, is named in full when it is cited in the findings chapter. Please also note the following technical points about how Official Statistics from all of these publications are presented throughout this report:

• Figures may not sum because (a) they have been rounded for disclosure control and/or (b) cases where information on applications or applicants are unknown have not been presented
• Where stated, secondary analysis has been conducted on rounded figures from published Official Statistics
• Most results are presented to zero decimal places. ‘0%’ should therefore be interpreted to mean less than 0.5%. If no responses were given then this is denoted by ‘-’.

Social Security Scotland Five Family Payments Client Survey

The Five Family Payments Client Survey (hereafter referred to as the ‘Client Survey’) is sent to everyone who has applied for one or more of Scottish Child Payment, Best Start Grant or Best Start Foods, and received a decision on that application. It collects equalities and socio-economic information from respondents, and asks about their experience of Social Security Scotland, including applying for and receiving benefits. As such, the Client Survey provided evidence on immediate process outcomes set out in the logic model above, relating to promoting and administering the benefits.

The results reported here are based on an analysis of 3,914 Client Survey responses collected between 1 April 2023 to 31 March 2024. The analysis included the following statistical procedures:

• Frequency analysis, to show how all respondents answered each question in the survey
• Crosstabulation analysis, to show how different subgroups of respondents answered each question in the survey.

The subgroup analysis focused on the priority families most at risk of poverty^[20], SIMD, and application outcome. More detail about the subgroups is shown in Table 2.

Please note the following technical points about how Client Survey findings are presented throughout this report:

• The number of respondents providing a valid answer to each individual question/statement varied slightly, within the ranges shown
• Most results are presented to zero decimal places. ‘0%’ should therefore be interpreted to mean less than 0.5%. If no responses were given then this is denoted by ‘-’
• Results denoted by ‘#’ have been suppressed due to disclosure control
• ‘Not applicable’ responses to questions have not been included in graphs and tables, unless otherwise stated. This means that results in tables and graphs may not equal 100%
• Subgroup differences are only reported in cases where the difference between the subgroups is more than 5 percentage points.

Limitations

This section explains what can and cannot be determined from the available data sources, and how this influences the extent to which conclusions can be drawn about the impact of the Five Family Payments.

Role of commissioned research: The evaluation is largely based on findings from mixed methods research commissioned by the Scottish Government. The survey provides robust, representative and generalisable high-level data on the impact of the Five Family Payments on recipients and their families. The large sample size also enabled a subgroup analysis, making it possible to identify differential outcomes for recipients based on key demographic and household characteristics, such as being in a priority family most at risk of poverty. However, the small number of survey respondents aged under 25 meant it was not possible to include the priority family ‘young mothers under 25’ as a subgroup in the analysis. Findings from the commissioned research are also based on the participants’ perceptions of impact, rather than objective measures of impact. This is a common limitation of self-report research.

Role of Social Security Scotland research: The Client Survey statistics contained in this report are based on up to 3,914 responses from clients who applied for at least one of the Five Family Payments between 1 April 2023 and 31 March 2024 (and the number of respondents providing a valid answer to each individual question/statement varies within the ranges shown throughout the report). Although the survey results provide insight into the views and experiences of a substantial number of Social Security Scotland clients, it should be cautioned that this represents a small fraction of Five Family Payments applicants during that time period, and views are drawn from a self-selecting sample of applicants. No weighting had been applied to counteract potential response bias. Therefore, it cannot be assumed that the results represent the views of Five Family Payments clients as a whole.

Lack of population survey data: Population surveys were considered as a source of evidence for this evaluation. This is because some population surveys, including those underpinning Scotland’s National Performance Framework indicators (e.g. Family Resources Survey), contain measures that align with medium-term policy outcomes in the Five Family Payments logic model, on topics such as debt and food security. However, population surveys were not used for the following reasons:

1. Until Scottish Child Payment was rolled out to children aged under 16 in November 2022, the Five Family Payments were only available to families with young children, which is a small subset of Scotland’s population. The number of population survey respondents receiving the Five Family Payments before November 2022 was therefore too small for a robust analysis.

2. At the time of writing, over two years has passed since Scottish Child Payment was rolled out to children under 16 in November 2022. However, relevant population surveys have only published data up to early 2023 (except the Family Resources Survey, which has published data to early 2024). To reasonably assess the impact of Scottish Child Payment with population surveys, it would be necessary to wait until more data has been published to (a) ensure enough eligible respondents have claimed Scottish Child Payment, and (b) allow any impacts of the benefit to take effect.

It should also be noted that, even with sufficient data, the extent to which population surveys can be used to evaluate specific policies (such as the Scottish Child Payment) is limited, as it is difficult to isolate the target group and the direct impact of a policy in wider survey data. As such, any future analysis of population surveys would only provide indirect evidence of impact, and need to be considered alongside other sources of evidence.

Long-term impacts will take time and additional data to determine: Understanding the true impact of the Five Family Payments would involve measuring their lasting contribution to wider Scottish Government outcomes. Doing so requires suitable time to have passed, and would involve isolating the influence of the benefits from other contributing factors, such as wider social security benefits and government interventions designed to support families with children. A step in this direction would be to link data on people claiming Scottish benefits with socioeconomic data on the same people held by Department of Work and Pensions, HM Revenue & Customs and the National Health Service. This would make it possible to make a more objective assessment of the impact of Scottish benefits on claimants’ income, employment and health. Data linkage is a key part of long-term social security policy evaluation, and exploration of data linkage is ongoing. However, the linkage process is time consuming and resource intensive to set up, due to the inter-organisational collaboration required, stringent data security and privacy procedures, and system design and development. More detail on options for extended policy evaluation is provided in the evaluation strategy. ^[21]

Research Advisory Group

A Research Advisory Group (RAG) was formed in early 2024 to guide and support key decisions in the delivery of the Five Family Payments evaluation, to ensure it achieved its aims and objectives. The RAG membership consisted of Scottish Government analysts, officials from Public Health Scotland, and a representative from academia. The RAG members offered a range of analytical perspectives from subject and methodological experts, helping to increase the transparency and robustness of the research. The group convened four times between March 2024 and May 2025.