Five Family Payments evaluation: annex A - mixed methods research

2. Methodology

2.1. Research aims and objectives

The overall aim of the research was to assess the experience and impact of Five Family Payments (FFP), both individually and collectively, on families and children receiving one or more of the payments. The research will assess the performance of the FFP against their short and medium-term outcomes outlined in Appendix A. A secondary aim of the research was to explore the awareness, take-up and impact of the FFP among seldom heard groups, particularly priority groups of low-income families at increased risk of being marginalised from the social security system.

To address these aims, the following key research objectives were proposed to provide evidence of the extent to which:

• FFP have improved financial outcomes for families.
• FFP have reduced material deprivation experienced by families.
• FFP have improved health and wellbeing outcomes for children and families.
• FFP have helped reduce barriers to education and the labour market.
• Recent efforts to engage marginalised groups have been effective, exploring whether and how they have led to greater awareness of the FFP and encouraged take-up of FFP.
• Changes to the eligibility and payment levels of FFP have impacted on the financial, material deprivation, health and wellbeing, educational and employment outcomes of families receiving FFP.
• The cost-of-living crisis has impacted on the overall finances of households receiving FFP, and the use and impact of FFP on families.

The research explored both the collective and individual impact of the FFP to allow for comparisons across groups based on benefit type, length of time receiving payments and demographic characteristics, with a particular focus on the families at risk of poverty; lone parents, families with a disabled child or adult, larger families (with three or more children), families from minority ethnic backgrounds, families with a child under one year, and families in which the mother is aged under 25 years^[4].

Research design

To address the research aims and objectives, ScotCen conducted an online survey of FFP clients, in-depth interviews with 33 clients who had completed the survey and interviews with five stakeholders who work with families eligible for FFP.

The online survey was conducted between 9^th October and 8^th November 2024. Client and stakeholder interviews took place between 28^th October 2024 and 27^th January 2025.

To ensure the research was designed and implemented in an ethical manner, an application was submitted to National Centre for Social Research (NatCen) Research Ethics Committee (REC). Ethical approval for the research was granted by NatCen REC on 30^th August 2024.

2.1.1. Survey of clients

The web questionnaire was developed in collaboration with the Scottish Government. The survey included pre-existing questions from the Social Security Scotland Client Survey, as well as newly developed questions to address the aims and objectives of this evaluation. The survey took approximately 15 minutes to complete.

Sampling and recruitment

A stratified random sample of 51,500 FFP clients with a successful application between 1st April 2022 and 31st August 2024 was drawn by the Scottish Government for invitation to participate in the survey. To draw this sample, the list of all eligible applicants was sorted first by local authority and within local authority by the Scottish Index of Multiple Deprivation (SIMD) and age. After generating a random starting point, cases were systematically selected from the list in a manner which ensured sample size for each local authority was proportionate to the overall number of FFP clients in that area area whilst also reflecting the spread of area deprivation and client age.

Sampled participants from the client database received an email^[5], sent by the Scottish Government, inviting them to participate in the survey, issued on 9^th October 2024. The email provided an overview of the study, the web survey link and a link to a participant information page.

Six days after the first invitation was issued, sampled participants were sent a first reminder email. After a further nine days, a second reminder email was issued. The survey closed on 8^th November. The achieved sample is outlined in Section 2.4.

Weighting

Aggregate demographic information (local authority, SIMD and age) for the total eligible population of clients and the sample of 51,500 drawn for the survey was shared with ScotCen so it could be compared with our achieved sample to assess its representativeness. The achieved sample under-represented younger clients aged under 35,and over-represented those aged 45 and over. The data were calibrated to the population figures to compensate for this. When the survey weight was applied, the age profile of the weighted achieved sample matched the age profile of the applicant population.

2.1.2. Interviews with FFP clients and stakeholders

Sampling and recruitment

The primary mode of recruitment for the in-depth interviews with FFP clients was asking those who had completed the survey to provide their contact details if they consented to take part in a follow-up interview about their experiences of receiving the FFP. Due to the large response to the survey and significant interest in taking part in a qualitative interview, it was possible to sample on a number of key criteria of interest, including:

• Age group
• Receipt of SCP, BSF and BSG
• Length of receipt of FFP
• Local Authority
• SIMD
• Employment status
• Kinship carers
• Key priority families most at risk of poverty (households with a disabled family member, families from minority ethnic backgrounds, child under 1 year, families with three or more children, lone parents and parents aged under 25 years)

The qualitative sample of clients is summarised in Section 2.5.

Selected individuals who provided their contact details were contacted by email to invite them to partipate in an interview. A list of organisations who offer support, was also attached to the invitation. Those interested in proceeding to interview agreed a suitable time and date for the interview to take place.

The five stakeholders were recruited from a range of organisations which aim to support low-income families. Over 20 individuals were contacted, seven expressed an interest in taking part in an interview, and five completed an interview.

Conducting the research

FFP clients who took part in an interview were given the choice of a telephone, video or in-person interview. All FFP client participants selected either a telephone or video interview. All participants were given a £40 Love2Shop voucher as a thank you for their time and for sharing their experiences with the study.

Stakeholders who took part in an interview were given the choice of doing so via telephone or video. All participants opted for a video interview.

With the consent of participants, all client and stakeholder interviews were audio recorded and transcribed for ease of analysis.

Research materials

The interview participant information sheets and topic guides were developed in consultation with and agreed with the Scottish Government. The client topic interview guide covered a range of topics, including accessing FFP, the use of FFP as well as their perceived impact, and any modifications that could be made to how FFP are delivered in the future. The aim of the interviews were to provide context to survey findings and to explore the impact of FFP on families in more detail. Interviews with stakeholders explored their views and experiences of supporting families to take up FFP, including any barriers or challenges encountered. Additionally, stakeholder interviews explored the perceived impact on families who do and do not receive FFP.

2.1.3. Analysis

Survey

All analysis of the survey data was undertaken in SPSS using the weighted data. Frequencies and crosstabulations were conducted, with the crosstabulations displaying overall results for key measures and the data for specific sub-groups, allowing comparison between them.

Statistical significance testing was carried out on the survey data to estimate how confident we can be that the differences between sub-groups present in the survey data are representative of real differences in the population, given the amount of uncertainty that we are prepared to accept in our sample. All statistical testing was done using logistic regression to a significance level of 95%^[6]. The statistical tests were two-sided taking into account the survey weights. Differences which are not statistically significant are generally not reported in the text unless it is considered noteworthy that no such difference is observed in the data.

Sub-group analysis was conducted for priority families at increased risk of poverty, SIMD and length of benefit receipt. This made it possible to understand the outcomes for priority family groups such as lone parents, families with three or more children, families with a disabled adult and/or child, families with a child under 1 year old, and minority ethnic families. The groups were defined as follows:

• Household composition: One parent/carer household / two or more parent/carer household.
• Family size: Households with one or two children under the age of 16 / households with three or more children under the age of 16.
• Disability: Households with disabled family member(s) / households without a disabled family member.
• Child age: Households with a child under 1 year old / households without a child under 1 year old.
• Ethnicity: Respondents from minority ethnic backgrounds / white ethnic backgrounds.
• Length of benefit receipt: In receipt of payment up to 12 months / in receipt of payment for over 12 months.
• SIMD: SIMD quintiles, where 1 is 20% most deprived areas, and 5 is 20% least deprived area.

The base sizes were too small to conduct sub-group analysis for mothers aged under 25 years (the sixth priority family group) or for client gender. Other sub-group analysis is possible but was out of scope for this project.

This report includes tables and charts to display survey data. Not applicable (n/a) responses are not included, therefore percentages may not equal 100%. However, a full set of data tables are available that include these responses.

Client and stakeholder interviews

The transcribed qualitative interview data were managed and analysed using qualitative analysis software NVivo. All qualitative data were anonymised for analysis and reporting. Analysis involved several stages. First, the key topics and issues which emerged from the research objectives were identified through familiarisation with the transcripts. A draft analytical framework was drawn up and an internal meeting was held to agree the initial coding framework. The analytical framework was then set up in NVivo and piloted with a few transcripts by two members of the research team and any amendments to the framework made. Each transcript was coded, so that all the data on a particular theme could be viewed together. Through reviewing the coded data, the full range of views were systematically mapped, and the accounts of different participants compared and contrasted. Emergent patterns and explanations for individuals and categories of respondent holding particular views were also explored and tested.

2.2. Strengths and limitations of the research

There were a number of strengths and limitations to the research.

2.2.1. Strengths

The study used a mixed method design. Combining a random probability survey with follow-up qualitative research enabled this study to capture more nuanced and in-depth information on the impact of FFP. The survey provided robust, representative and generalisable high-level data across Scotland of the impact of receiving FFP on families health and wellbeing, child spend, work and training and financial wellbeing, while the interviews captured rich, in-depth data which provide insights into the experiences and impacts of FFP on clients and their families.

Furthermore, with the inclusion of stakeholders who support families on low-incomes, the study was able to explore barriers in the take-up of FFP, adding to our understanding of those eligible for, but not claiming, FFP.

A large achieved sample size for the client survey enabled analysis by a variety of subgroups. This allows a deeper exploration of the experiences and impact on priority families including lone parents, families with a disabled family member and large families. This also enabled further exploration of experiences in the qualitative interviews.

2.2.2. Limitations

Despite the large sample size invited to participate in the survey, the number of mothers aged under 25 years who responded to the survey was too low to conduct analysis on this particular sub-group of clients. The qualitative interviews with clients sought to bridge this gap and while parents in this age group did participate, those aged under 20 years chose not to engage in the qualitative research. Stakeholders working with young parents were involved in the research and were able to provide further insight. The weighted sample overall is representative of client age.

As noted above, the survey response was large enough to enable sub-group analysis of most of the priority families providing keen insight into the views and behaviours of families with different characteristics. However, the data also demonstrate that many priority families share one or more of the priority characteristics – for example, where lone parents also have a disabled family member or large family. The analysis undertaken does not control for these interconnected characteristics meaning it is difficult to determine which individual factor is driving the relationship with views and behaviours. Thus for some sub-groups (for example ethnicity), multivariable regression analysis would be advisable. This allows an exploration of the independent relationship between a single variable (e.g. ethnicity) and an outcome measure (e.g. improved mental health) whilst controlling for multiple other variables (e.g. lone parent status or having a disabled family member). In so doing, it can provide some insight into the relative independent weight that different factors have on individuals’ views, behaviours and circumstances. The budget and timescale of this project did not enable regression analysis to be conducted.

Finally, the focus of the study was on the impact FFP had on those who received them. Therefore, this study does not provide insight into direct experiences of those who have not received any FFP, although eligible. However, as noted above, interviews with stakeholders explored some of the barriers to take-up of FFP.

2.3. Participant demographics

2.3.1. Survey participant demographics

In total, 3,922 current recipients of FFP completed the survey. Of these 3,407 were in receipt of SCP, 523 were in receipt of BSF, and 1922 had received at least one BSG. These are the unweighted figures for those in receipt of each of the Five Family Payments, which will serve as the unweighted bases throughout the report. All analysis was done on weighted data, as described above in section 2.2.3, therefore the percentages reported are all weighted and will not match exactly with the unweighted figures. There is some additional fluctuation in unweighted base size due to responses such as ‘Don’t know’ and ‘Prefer not to say’ not being included in the figures for analysis.

Household composition

The majority of survey respondents (91%) had at least one child under the age of 16 living in their household. The remaining respondents (9%) did not, but were pregnant.

Those who had at least one child under the age of 16 living in their household were asked how many children live in their household. Eighty-three percent of respondents had either one child (47%) or two children (36%) living in their household. Seventeen percent of respondents had three or more children living in their household.

The majority of respondents (97%) said they were the parent/step-parent/guardian of the child(ren) living in their household, and 2% were kinship carers which included sibling, grandparent or other relative. One percent of respondents said their relationship to the children in their household was ‘other non-relative’.

Around two-thirds (67%) of respondents said they lived in a one parent/carer household where they did not live with anyone else who has a parenting or caring role for the child/ren in the household. Under one-third (30%) of respondents said they lived in a two parent/carer household where they lived with someone who has a parenting or caring role for the child/ren in the household.

Those who selected 'other' (3%) as the best description of their household, provided further information on their household. There were those living in intergenerational households, living with their parents, grandparents or siblings. In some circumstances these family members provided some caring support, while in others it was not clear from the description if they played any caring role. Some respondents shared that they were living with older or disabled relatives that required their care. There were also respondents who selected 'other' to describe the caring responsibilities of their families. There were respondents who said they lived alone with their children but another adult from another household shared parenting/caring responsibilities. There were also respondents who described living with a partner who wasn't biologically related to their children but had a caring role. Respondents also described living with a partner who they cared for. Finally, there were respondents who described that they were kinship carers.

Respondents were also asked, who had the main caring responsibilities for children under the age of 16 in their household. Around two-thirds (67%) of respondents said they alone had caring responsibilities for children in their household, and one-third (33%) said they shared caring responsibilities with someone else. Note that, due to percentages being rounded, the figure for ‘someone else in the household alone’ appears as 0%; 0.4% of respondents (n=13) selected this answer.

Household income

The reported annual household income of respondents varied from less than £5,200 per year up to £51,999 per year. Around two-fifths (43%) of respondents had an annual household income of under £15,600. One-third (31%) of respondents had an annual household income between £15,600-£36,399. Around one-quarter (23%) of respondents prefered not to share their annual household income.

Note that, due to percentages being rounded, the figures for ‘£52,000 to £77,999’ and ‘£78,000 or more’ appear as 0%; 0.4% of respondents (n=15) and 0.1% of respondents (n=3) respondents selected these answers, respectively.

Rurality

The majority of respondents (85%) lived in urban areas, with the remaining 15% living in rural areas.

SIMD

Around two-thirds of respondents (67%) lived in the 40% most deprived areas of Scotland.

Gender

The majority of respondents described their gender identity as ‘woman’ (89%), with 1 in 10 (10%) describing their gender identity as ‘man’. One percent of respondents prefered not to say. Note that due to percentages being rounded, the figure for the the ‘other’ category appears as 0%; 0.2% of respondents (n=8) selected this answer.

Age

The majority of respondents were aged between 25 and 54 (91%), with the largest proportion aged between 35-44 (42%). A small proportion of respondents were aged 20-24 (5%) and aged 55 and over (3%). Note that due to percentages being rounded, the figures for the age groups ‘under 16’, ’16-19’ and ‘65 or over’ appear as 0%; 0.07% (n=3), 0.1% (n=5) and 0.8% (n=32) of respondents selected these answers, respectively.

Ethnicity

The majority of respondents reported that their ethnic group was white (85%). Twelve percent of respondents reported their ethnic group was Asian (5%), African (4%), mixed or multiple ethnic groups (1%) and other ethnic group (2%). Three percent of respondents preferred not to say. Note that due to percentages being rounded, the figure for the ethnic group ‘Caribbean or Black’ appears as 0%; 0.1% (n=5) of respondents selected this answer.

Employment and training

Around half of respondents were employed either part-time (33%), full-time (15%) or self employed (6%) at the time of the survey. Around 1 in 20 respondents said they were unemployed and seeking work (6%), unable to work due to short-term illness or injury (4%), or studying (5%) either in further or higher education (4%) or in school (1%). A further 10% of respondents reported that they were permanently sick or disabled and 1% permanently retired from work.

Note that due to percentages being rounded, the figures for the categories ‘government work and training scheme’ and ‘don’t know’ appear as 0%; 0.2% (n=9) and 0.04% (n=1) of respondents selected these answers, respectively.

Health

All respondents were asked if they had a physical or mental health condition or illness lasting or expected to last 12 months or more. Forty-seven percent said ‘yes’, and 46% said ‘no’. The remaining 7% preferred not to say.

Those who reported having a physical or mental health condition or illness lasting or expected to last 12 months or more were asked if these conditions or illnesses reduced their ability to carry out day-to-day activities. The majority of respondents (89%) said it did ‘a lot’ (45%) or ‘a little’ (44%). Fewer than 1 in 10 respondents (8%) reported that their ability to carry out day-to-day activities was ‘not at all’ reduced by their health conditions or illnesses.

All respondents were asked: ‘Does anyone else in your household have a physical or mental health condition or illness lasting, or expected to last, 12 months or more?’. While over one-half (55%) of respondents said ‘no’, however, around two-fifths (39%) said ‘yes’, this was the case. Around one-fifth (19%) of households had one or more child aged under 16, and 15% of households had one or more adults, with a physical or mental health condition or illness lasting, or expected to last, 12 months or more. Five percent of households had both adults and children in their household with a physical or mental health condition or illness lasting, or expected to last, 12 months or more.

Further survey respondent demographics can be found in Appendix B.

2.3.2. Qualitative participant demographics

Thirty-three people took part in an interview about their experience of receiving one or more of the Five Family Payments (FFP) (Table 2.12).

All 33 participants were currently in receipt of the Scottish Child Payment (SCP) at the time of the interview, eight of whom had said their payment had stopped at least once. Twenty-four participants had been receiving SCP for over 12 months.

Thirteen participants were also currently receiving Best Start Foods (BSF), with additional participants having received it when a child was under the age of three. All 13 participants in receipt of BSF had been receiving it for more than 12 months.

Twenty-two participants had received one of more of the Best Start Grant payments (BSG). Ten had received the pregnancy and baby payment, 11 had received the early learning payment and seven had received the school aged payment.

Thirteen participants received all FFP.

Priority family groups

As part of the evaluation, the Scottish Government identified a number of priority family groups most at risk of poverty. At least five participants from each of the priority family groups were interviewed (Table 2.13).

Almost two-thirds of participants (n=21) were a lone parent/carer and/or lived in a household where at least one adult or child was disabled.

Five kinship carers also participated in an interview.

Age of participants

The qualitative interviews included parents and carers from a wide range of ages (Table 2.14).

Over one-third (n=13) of participants were aged 45 and over. This was largely as a result of the groups of interest, particularly kinship carers, and large families with three or more children. The Scottish Government were also interested in the views and experiences of households with children aged 5-15 where parents and carers tended to be older.

Employment and education status

Qualitative participants included those in paid employment, in full or part-time education, full-time carers, those unable to work due to ill health and those looking for work (Table 2.4).

Approximately one-half (n=17) of participants were in paid employment, with around one-third unemployed (n=5) or unable to work due to ill health (n=5). Five participants were currently in education or training.

Location

The majority (n=21) of qualitative participants lived in the 40% most deprived areas of Scotland (Table 2.16).

Participants were engaged from across Scotland. The majority (n=21) lived in cities and large towns, with just over one-third (n=12) living in rural locations.

Stakeholders

Five stakeholders took part in qualitative interviews. Stakeholders represented a variety of organisations including food banks and community initiatives that provided benefits advice, general support, or support targeted priority families most at risk of poverty (for example, lone parents or families with a disabled family member). Stakeholders’ roles included a welfare rights advisor, financial inclusion officer, community connector and specialist project worker.