Publication - Research and analysis

Fair Start Scotland - evaluation report 3: year two - overview

Published: 9 Nov 2020

Third report in a series of evaluation reports of Fair Start Scotland employability services which covers the second full year of service delivery (April 2019 - March 2020) and summarises findings from a participant phone survey, local area case studies and analysis of management information.

76 page PDF

2.5 MB

76 page PDF

2.5 MB

Contents
Fair Start Scotland - evaluation report 3: year two - overview
Appendix 1: Evaluation Methodology

76 page PDF

2.5 MB

Appendix 1: Evaluation Methodology

Telephone survey

The Fair Start Scotland (FSS) Wave 2 telephone survey was carried out in May 2020 by IFF Research Ltd.

The sample was made up of two distinct groups:

  • New sample of participants who joined the FSS service in 2019 (between January 2019 and December 2019). IFF were provided with a sample of 11,828 participants, from which 1,782 were drawn for the survey.
  • Longitudinal sample who joined the FSS service in 2018 (between April and December 2018) and took part in the Wave 1 survey. All participants who took part in the survey and agreed to be contacted again for further research were included in the sample for Wave 2.

A total of 1,007 surveys were completed at Wave 2, made up of 607 from the new sample and 400 from the longitudinal sample, as shown in Table 1.1. An additional 10 respondents from the new sample took part in the survey but said that they had never received a service from FSS and were removed from the final data.

Table 1.1: Wave 2 sample drawn, and surveys completed by sample group
Sample group Starting sample available Sample drawn Responses achieved
2019 cohort (new sample) 11,828 1,797 607
2018 cohort (longitudinal sample) 940 940 400

For the new sample, IFF were provided with data consisting of all starts on the FSS service during this period. From this, 1,782 pieces of sample were drawn. Sample was drawn in proportion with the distribution of participants by lot, with Lots 4,7,8 and 9 slightly oversampled to ensure a minimum base for subgroup analysis of this region. Table 1.2 below shows the number of records drawn and the number of surveys achieved from each lot.

Table 1.2: 2019 cohort (new) sample drawn, and surveys completed per Lot
Lot All sample Sample drawn Unweighted responses Weighted responses
N % N % N % N %
Lot 1 Glasgow 2398 20.3% 300 16.7% 101 16.6% 123 20.2%
Lot 2 Lanarkshire 1804 15.3% 213 11.9% 71 11.7% 93 15.3%
Lot 3 Tayside 1161 9.8% 150 8.3% 48 7.9% 60 9.9%
Lot 4 Forth Valley 453 3.8% 150 8.3% 50 8.2% 23 3.8%
Lot 5 East 2568 21.7% 327 18.2% 112 18.5% 132 21.7%
Lot 6 South West 1648 13.9% 207 11.5% 71 11.7% 85 14.0%
Lot 7 North East 461 3.9% 150 8.3% 50 8.2% 24 3.9%
Lot 8 Highlands and Islands 527 4.5% 150 8.3% 50 8.2% 27 4.4%
Lot 9 West 808 6.8% 150 8.3% 54 8.9% 41 6.7%
Total 11,828 100% 1,797 100% 607 100% 608 100%

Participants were sent an advance letter two weeks prior to fieldwork to notify them of the research and offer them the opportunity to decline to take part. Telephone fieldwork was conducted between 27th April and 29th May, and 1,007 completed interviews were achieved. A full breakdown of sample outcomes is shown in Tables 1.3 and 1.4.

Table 1.3: 2019 cohort (new) sample outcomes
  Number Proportion of starting sample (%)
Total sample 1,797 100%
Opt outs 18 1%
Unusable (for example, wrong number) 124 7%
Contact attempted, no final outcome after minimum number of attempts 935 52%
Refusal 99 6%
Respondent stopped or screened out during survey 14 1%
Total surveys completed 607 34%

Table 1.4: 2018 cohort (longitudinal) sample outcomes

  Number Proportion of starting sample (%)
Total sample 940 100%
Opt outs 11 1%
Unusable (for example, number not in use) 21 2%
Contact attempted, no final outcome after minimum number of attempts 475 51%
Refusal 24 3%
Respondent stopped or screened out during survey 9 1%
Total surveys completed 400 43%

The survey included fewer questions for the 2018 longitudinal sample than the 2019 new sample. The average survey length for the 2018 respondents was 13 minutes 36 seconds, and for the 2019 respondents it was 21 minutes 21 seconds.

The data was checked, tabulated and verbatim responses were fully coded for analysis purposes. A rim weight based on age, gender and lot was applied to the 2019 data to bring the oversampled Lots 4,7,8 and 9 back in line with population proportions of 2019 FSS starters, and to correct for any non-response bias. A rim weight based on age, gender, lot and employment status was also applied to the 2018 data to correct for any non-response bias and bring the proportions in line with the Wave 1 weighted data, which reflected population proportions of 2018 FSS starters.

More detail on the breakdown of respondent profile by Lot, after weighting, is set out in the original report. See: Fair Start Scotland Evaluation Report 3: Participant phone survey (year 2).

Local Area Case Studies

The local area case study research focuses on developing case studies in 9 localities across Scotland over the three years, one in each contract Lot area, undertaken by Rocket Science UK Ltd and Blake Stevenson. The research involves carrying out the following tasks in each of the case study areas:

  • Conducting desk-based area analysis of the socio-economic and employment trends in each of the localities to understand the local labour market context that FSS is operating in
  • Analysing the management and performance data from FSS in each of the localities to understand the profile and numbers of participants and outcomes achieved in the area
  • Conducting interviews with participants in each locality to understand their experience of and views on FSS
  • This year this was complemented by with follow up interviews with 9 of those interviewed in the locality case studies last year – this included 3 from each of Alloa, Irvine and Wick. The purpose of these interviews was to explore the longer-term impact of the experience of FSS on participants.
  • Conducting interviews with employers in one locality to understand their experience of and views on FSS. Because of the impact of COVID-19 arranging employer interviews this year proved difficult as employers had appropriate staff on furlough or it was difficult to get hold of very busy staff on in HR roles, so the number of employer interviews was limited.
  • Conducting interviews with service staff including managers and frontline staff delivering FSS locally to understand their experience of and views on the service
  • Conducting interviews with staff in partners of FSS providers in the locality to understand their experience of and views on FSS
  • Conducting interviews with relevant other stakeholders in the area to understand their experience of and views on FSS.

FSS participants were contacted through a database of all service participants supplied by Scottish Government. All participants living in Drumchapel, Dundee, and Peterhead and Fraserburgh who had taken part in the service for at least 6 months were emailed, and an introductory letter to participants across all three areas was issued. This was followed up with a telephone call to arrange an interview, making a maximum of three attempts to contact each participant.

In total, 30 interviews with participants were conducted, along with 18 key workers who – between them – worked with 28 of these participants. In addition, 9 of those participants who were first interviewed last year – 3 from each of the three areas (Alloa, Irvine and Wick) – were re-interviewed.


Contact

Email: socialresearch@gov.scot