Fair Start Scotland Evaluation Report - Year 6: Phone Survey with Service Participants - March 2026

Technical Appendix

Survey approach

The Fair Start Scotland (FSS) Wave 5 telephone survey was carried out in June and July 2025. This included new participants who had not previously been interviewed, as well as longitudinal participants who had previously been interviewed at Wave 4 in December 2022 and January 2023.

Sampling

The sample was made up of two distinct groups:

1. New sample of participants who joined the FSS service in 2023-24 (between October 2023 and March 2024)

2. Longitudinal sample who joined the FSS service in 2021-2022 (between July 2021 and June 2022) and took part in the Wave 4 survey

For the new sample, IFF were provided with data consisting of all starts on the FSS service between April 2023 and March 2024, which totalled 13,053 records before data cleaning and 12,910 records after data cleaning^[44]. The decision was made to select a sample from those FSS participants who joined in Quarter (Q)3 and Q4 (Oct 2023 – March 2024) of Year 6 of service delivery rather than from the whole year. The main reasons for this were:

• the length of time between respondents joining FSS and completing the survey would be more similar to previous waves of this research, thereby enhancing the comparability of findings between waves (any differences over time are more likely to be due to actual changes in attitudes and outcomes)
• the length of time between respondents joining FSS and completing the survey would be shorter, promoting recall and promoting the accuracy of the data captured by the survey

A comparison of the key socio-demographic characteristics of those who started in Q1 and Q2 and those who started in Q3 and Q4 of Year 6 was conducted, in order to verify whether the Q3 and Q4 sample would be representative of the Year 6 cohort. This revealed some differences in profile, however none were significant apart from ethnicity. For ethnicity, more people from minority ethnic groups joined in Q3 and Q4 (20 per cent of all FSS starters) than in Q1 and Q2 (16 per cent). On balance, the benefits of this sampling approach were deemed to outweigh this difference.

After sample records with no contact details or duplicate contact details were excluded, 6,655 records (among those who started between October 2023 and March 2024) were drawn. The sample was then drawn in proportion with the distribution of participants by Lot (the nine FSS contract areas across Scotland) across all starts between October 2023 and March 2024. Opt-outs and records matching longitudinal contact details were removed, and from the remaining records, 2,375 useable records were drawn based on the expected response rate of 20 per cent and the target of 475 responses overall, as shown in Table 2 below.

For the longitudinal sample, all participants from the 2021-2022 cohort who had taken part in the Wave 4 survey were sampled for Wave 5. This totalled 750 respondents, as shown in Table 2. Once those who had declined further contact and opt-outs were removed, 627 useable records remained.

Fieldwork

Participants were sent an advance letter two weeks prior to the fieldwork to notify them of the research and offer them the opportunity to decline to take part. They were able to do this via email, or by calling a telephone voicemail service. Telephone fieldwork was conducted between 3^rd June and 11^th July 2025. Among the 2023-24 cohort, 102 participants declined to participate (4 per cent of the starting sample). Forty-two of the 2021-22 cohort declined to participate (7 per cent of the starting sample). The average survey length for the 2023-24 cohort respondents was 26 minutes 6 seconds while for the 2021-22 respondents was 11 minutes 28 seconds. An ‘assisted’ interview was offered in the most common minority languages spoken in the UK (e.g. French, German, Spanish, Polish, Bengali, Gujarati, Punjabi, Urdu, Yaruba and Hindi). In an assisted interview, the interviewer administers the questionnaire in English, but provides assistance in the respondent’s first language where necessary to aid understanding.

The survey was programmed into Unicom survey software, which interviewers used to call respondents and enter their survey responses.

The project aimed for 700 interviews in total. Due to a lower response rate than expected for the new sample, 669 interviews were completed, made up of 469 from the new sample and 200 from the longitudinal sample. A full breakdown of sample outcomes are shown in Table 2 and Table 3 below.

The survey included fewer questions for the longitudinal sample (2021-2022 cohort) than the 2023-24 new sample. One reason for this is that these respondents had previously completed a full survey at Wave 4, therefore there was no need to ask duplicate questions in areas that were unlikely to have changed, such as their employment history prior to FSS or their demographic information.

Where ‘Refused’ is reported, this option was not read out to participants alongside the other options during the survey, but rather interviewers coded this response when participants were unwilling to provide an answer.

Table 2: Wave 5 new sample outcomes (2023-24 cohort)

Analysis approach

The survey data was checked and processed using SPSS^[45], verbatim responses were fully coded for analysis purposes, then combined with the prompted options from the survey and organised into data tables. The data was weighted, significance testing was undertaken, and differences between subgroups identified.

Weighting

As with previous waves, a Random Iterative Method (RIM) weight was applied to the 2023-24 cohort data. A RIM weight treats each variable and question as an individual entity to assure each data point and demographic is accurately represented. This helps ensure that the survey data is representative of the relevant population on demographic characteristics which could reasonably be expected to affect survey responses. The RIM weight applied to the 2023-24 cohort was based on age, gender and Lot. It was used to correct any differences in demographic profile compared to the profile of all Q3 and Q4 2023-24 FSS starters, and to correct for any non-response bias.

A RIM weight based on age, gender and Lot was also applied to the longitudinal 2021-22 cohort data to ensure the data was in line with the 2021-22 population proportions. A further RIM weight based on the 2021-22 cohort’s employment status at Wave 4 was applied to correct for any non-response bias.

In order to directly compare the 200 longitudinal respondents to their previous responses, a subset of the Wave 4 2021-22 cohort data was selected, including only those respondents who also participated in Wave 5 survey. A RIM weight based on age, gender and Lot was applied to this data to ensure proportions were in line with the population proportions of 2021-22 FSS starters.

Table 4 and Table 5 below show the overall sample, the number of records drawn, the number of surveys achieved, and the weighted proportions from each Lot.

Table 5: 2021-22 cohort

Effect of weighting

The data has been weighted to ensure that it is representative of the target population. As weighting should only ever lead to minor corrections in the data the impact of weighting on significance tests should be minimal^[46]. The tables therefore apply significance testing to weighted data.

For the fresh sample of the 2023-24 cohort, the effective sample size, following weighting, was 420 (from an unweighted sample of 469).

Significance testing

As part of our analysis processes, data tables were created which tabulate the data question by question against key measures of interest, such as the subgroups listed in the tables below. Within these tables we have applied significance testing to make it easier to identify which relationships are significantly different from a null hypothesis (that there is no relationship between the data observed), and that we can therefore conclude that a relationship does exist; accepting a five per cent chance of being wrong (the typical level of confidence applied when interpreting statistical significance).

The statistical significance tests used within our tables are two-tailed z-tests. The same z-test was used for comparing the data between waves. As the significance testing was not integrated to the data tables for data comparison between waves, a significance testing tool was used for wave-on-wave analysis.

It should be noted that while reporting on statistical significance is important, effect size is an important measure of the strength of the relationship between two variables in a population. Effect sizes were not calculated in this research. Differences between groups may therefore be statistically significant (particularly where there are large sample sizes), but with very small effect sizes, thus indicating limited practical implications. This should be taken into consideration while interpreting the findings.

Margins of error

For statistics reported on the total base of 469, 2023-24 cohort participants, the maximum standard error (at the 95% confidence interval) is +/- 4.5%. This is higher than it was in the previous wave (+/- 3.6%) because the sample size is smaller (469 compared to 750 in the previous wave). In practice, this means that for a survey finding of 50%, the true population figure lies between 45.5% and 54.5%. For the 2021-22 cohort (200 participants), the maximum standard error (at the 95% confidence interval) is +/- 6.9%. Where comparisons between waves are made, based on the total number of responses for each wave, the maximum standard error is +/-6.0%.

Subgroup comparisons

The following subgroups were analysed across the data, and are shown in Table 7 for the 2023-24 cohort and Table 8 for the 2021-22 cohort. Some of the subgroups were based on data provided by Scottish Government (SG) while others were based on data collected during the survey, as specified in Table 6 below.

*Please note that some subgroups may sum to 99% or 101% within the demographic category due to rounding of percentages **The ethnicity category does not sum to 100% of respondents because some preferred not to answer this question. For re-joiners this is because some respondents said they could not remember

Table 8: 2021-22 cohort subgroups

*Please note that some subgroups may sum to 99% or 101% within the demographic category due to rounding of percentages **The ethnicity category does not sum to 100% of respondents because some preferred not to answer this question

Wave on wave comparisons

As shown in Table 1 the main report, there was a methodological difference between the first four waves of the survey (where the 2018, 2019, 2020 and 2021-22 cohorts were interviewed) compared to Wave 5 (where 2023-24 cohort was interviewed). This difference related to the estimated average length of time that had passed between participants first joining FSS and completing the survey. Whilst at Waves 1 to 4 this length of time amounted to around one year, at Wave 5 this was around a year and a half.

Due to this, comparisons between the 2023-24 cohort and the other cohorts are not methodologically reliable for some survey measures. Measures which can reasonably be expected to be affected by a longer time gap between starting FSS support and completing the survey are shown in Table 9 below, alongside researchers’ rationale on why comparisons cannot reliably be made for these measures.

Table 10 below presents measures which are not expected to be substantially affected by the period of time between participants starting FSS and completing the survey, alongside the researchers’ rationale.