Individual Training Accounts: evaluation

Evaluation Methodology

Overview of the evaluation methods

The mixed methods evaluation was conducted between February and July 2022. The evaluation began with an inception meeting and scoping phase in February. The main data collection phase took place from March to early June. Analysis and reporting were undertaken throughout June and into early July 2022.

Inception and scoping

A scoping phase was undertaken to gain an understanding of the previous ILA programme, the changes made to transition into the ITA programme and wider use of training / learning accounts in other countries. Scoping activities were led by SQW and involved:

• Semi-structured interviews (1-1 and groups) with key stakeholders including Scottish Government, SDS, Chartered Institute of Personnel and Development (CIPD) Scotland, Scottish Training Federation (STF), and Scottish Local Authorities Economic Development Group (SLAED).
• High level literature review of seven key reports relating to individual learning or training accounts.
• Review of programme monitoring information from SDS covering 2017/18 to 2021/22.

Main data collection

ITA applicant and participant online survey

An online survey was set up to capture the views of people who had applied to / used the ITA programme in 2019/20 and 2021/22. All participants in those years were sent a link to the online survey. An incentive was offered to maximise the response rate and all who participated had the opportunity to be entered into a prize draw to win one of ten £50 vouchers.

• The overall participant sample population was 21,956 contacts (12,685 for 2019/20; and 9,271 for 2021/22). The participant survey was launched on 1 April 2022 and closed on 1 May 2022. There were 2,102 responses, a response rate of 9.6% (with 1,160 for 2019/20 (9%); and 942 for 2021/22 (10%)). Participants were sent two reminders during April to complete the survey.
• The overall applicant sample population was 1,757 contacts for 2021/22. Due to the way SDS' system temporarily stored applicant data, the applicant survey was distributed on a rolling basis and closed on 8 May 2022. There were 151 applicant responses, a 9% response rate. Applicants were sent one reminder during April to complete the survey.

The survey was developed by SQW and Progressive. Scottish Government officials commented and approved the draft in advance of the survey being launched. SDS provided the sample population contact details to Progressive who distributed the survey. Further detail on the focus group methodology and standalone findings can be found in Annex A.

Online Focus Groups with ITA participants

A series of eight online focus groups were conducted by Progressive with ITA participants who were recruited through the online survey. In total, 43 participants took part in the focus groups. Several groups were themed around sectors / occupations, with invitees selected accordingly. The topic guide was developed by SQW and Progressive. Scottish Government officials commented and approved the guide, and attended two sessions to observe. Further detail on the focus group methodology and standalone findings can be found in Annex B.

Interviews with Training Providers

Semi-structured interviews were undertaken with 13 training providers involved in delivering ITAs. They were selected on the basis of levels and types of activity, and location. The sample also covered general providers, providers new to delivering in 2021/22, providers that delivered at scale, and those whose delivery had increased / decreased over time. The topic guide was developed by SQW and approved by Scottish Government officials.

Analysis and reporting

SQW and Progressive worked together to analyse the strands of research. The quantitative survey data first underwent a computer edit of the data prior to analysis and where 'other' type questions were used, the responses were checked against the parent question for possible up-coding. A series of charts and tables summarising the headline findings from the survey were developed based on the themes of interest and further cross-tabs were created in Excel to query the data in different ways.

The qualitative data analysis used a consultative process that involved listening to the recordings and documenting the prevalence of themes by examining transcripts of recorded focus groups, noting the relative frequency with which different issues arose, as well as the intensity of their expression. A summary analysis document was produced in Word to organise the data and assess the prevalence of themes across the groups – a tally denoting how frequently each theme was discussed was appended to the data, enabling researchers to assess majority/minority experiences and views. The research team worked to fit the data together in a comprehensive and logical manner for the report writing stage.

A key findings and policy workshop was held with the Scottish Government and SDS in mid-July to discuss the overall findings and potential options for programme development in future.