Scotland's Migration Service: early insights evaluation

3. Evaluation approach

This evaluation sought to capture early evidence relating to the process, impact, and value for money of Scotland’s Migration Service. In spring 2024, Scottish Government analysts developed a monitoring and evaluation (M&E) strategy for SMS, which set out evaluation questions, key performance indicators (KPIs), and a theory of change. This strategy was developed in the development phase of the service, and prior to the interim launch. The M&E strategy was designed to measure the operation and impact of the service, and was also aligned to Digital Scotland Service Standard (DSSS) criteria, which all Scottish Government digital services are required to meet. As a living document, the strategy was refined during the initial months of interim service delivery to ensure it was effectively set up for measuring performance and impact of SMS.

The SMS theory of change articulates the pathway from service inputs and outputs to the short, medium, and long-term outcomes the service intends to achieve. It also illustrates the logic which underpins this pathway and how success at the activity, input, and output stage can in the longer term be reflected in successful contributions to the overarching strategic outcomes that SMS is expected to contribute to. This theory of change is presented overleaf and acts as a framework for this early insights evaluation.

When the theory of change was developed, a number of external factors were identified which were considered likely to influence and/or constrain the success of the service. These factors include:

• UK Government Legislation: Immigration is a reserved matter to UK Government, which means that any changes to legislation which may impact the ability of migrants to move to and work in Scotland is out of the Scottish Government’s control.
• News and media narratives: It is possible that narratives around migration in the media may influence people’s decisions to move to and settle in Scotland.
• Skills and experience: The level and relevance of migrants’ skills and experience to the skills gaps in Scotland’s labour market and specifically those sought by the employers that engage with Scotland’s Migration Service will also influence outcomes.
• Labour market: Related to the above, the number of available employment opportunities and the relevance and attractiveness of these is also an external factor which will influence the Scotland’s Migration Service outcomes set out in the theory of change

SMS Theory of Change

Key activities and inputs listed in Figure 1 are as follows; user research is carried out into information needs of people moving to Scotland and the barriers to moving to Scotland; content and advice is kept up-to-date; providing useful information for migrants and employers on website; providing easy to use and accessible content and services; implementing an effective communications strategy to advertise services; carrying out regular monitoring of the service; using user feedback to improve the service, providing useful reception support for people moving to Scotland through CAS; providing Seraphus tailored one-to-one support for employers and migrants, providing useful immigration one-to-many support for employers delivered by Seraphus

Figure 1 demonstrates that these activities and inputs are intended to result in the following outputs; an accessible digital platform which meets the needs of migrants and employers; consistent service experiences across different types of user; an improved service through continuous response to user feedback; the provision of advice and support to migrants and employers through the one-to-one service; the provision of advice and support to employers through the one-to-many service and the provision of products (e.g. tutorials) by Seraphus’ one-to-many service.

Figure 1 shows that SMS’ outputs are intended to result in the following short-term outcomes; more people are attracted to live in Scotland and supported to move; more people are supported to integrate and establish life in Scotland after moving; employers’ needs are met through SMS and an increased number of employers are supported. In turn, the figure shows that short-term outcomes are intended to contribute to the following medium-term outcomes; more people move to Scotland and successfully settle into communities; people stay longer after relocating; working-age people with the skills to support Scotland’s economic recovery are attracted to Scotland and more employers use the immigration system to effectively meet their needs. Figure 1 shows that these medium-term outcomes are intended to result in the following long-term outcomes; migrants make essential contributions to communities, culture, public services and the private sector ; more people settle permanently in Scotland; Scotland’s available talent pool is expanded and skills shortages in the economy are reduced.

Lastly, Figure 1 demonstrates that it is intended that these outcomes will feed into key aims of three broader national strategies. This includes;

(1) making Scotland an attractive and welcoming country (an aim of the Scottish Government’s Population Strategy)

(2) having a workforce that meets the demands of an ever-changing economy, society and making Scotland more prosperous and international competitive (aims from Scotland’s National Strategy for Economic Transformation), and

(3) Communities that are inclusive, empowered, resilient, and safe; thriving and innovative businesses, with quality jobs and fair work for everyone and globally competitive, entrepreneurial, inclusive and sustainable economy (aims from Scotland’s National Performance Framework)

Evaluation questions

This evaluation assessed early evidence to respond to the following evaluation questions:

• EQ1 (Process): To what extent has Scotland’s Migration Service been delivered as intended?
  • Is the information and advice provided by the service useful and up-to-date?
  • Is the service easy to use and accessible?
• EQ2 (Impact): To what extent is Scotland’s Migration Service achieving its intended outcomes?
  • To what extent are different groups of migrants experiencing and being impacted by the service in different ways?
  • Is there evidence to suggest that SMS makes it easier for migrants to move to Scotland and settle into their communities?
  • Do the services provided help employers use the immigration system more effectively and efficiently? Does the service assist employers in meeting their skills and labour needs?
  • To what extent do experiences of the service vary according to the type of employer using the service?
• EQ3 (Value for Money): To what extent does Scotland’s Migration Service deliver value for money?

This report presents findings in relation to each of these three main evaluation questions in the following chapters. Given the full service had been operational less than a year (November 2024-September 2025) when this evaluation was carried out, it was not possible to fully address the above evaluation questions. This was particularly the case in relation to EQ2 (to what extent SMS is achieving its intended outcomes), as many of these relate to longer term impact, e.g. supporting movement and settlement of migrants in Scotland. Instead, this report presents the initial evidence in relation to the above questions and makes recommendations based on these, including for future evaluation. It is limited in the conclusions it is able to draw.

Evaluation approach and methods

This evaluation was carried out by SG analysts in Central Analysis Division and the Office of the Chief Economic Adviser and was informed by the Magenta Book for Government Evaluation. The SMS theory of change set the framework for the evaluation, which drew primarily on existing monitoring and performance data. Given the short period the full service has been operational, data availability was limited and in many cases analysis drew on small sample sizes. The evaluation consisted of analysis of a mix of quantitative and qualitative data, including:

• service monitoring and performance data;
• user feedback data;
• narrative accounts of the delivery and implementation of the service;
• and small-scale qualitative research with service users and service providers.

Further detail on each of these evidence sources and the analysis of these is set out below. Key data issues and limitations are also outlined.

Sources of evidence

It should be noted that while some of the sources which the evaluation drew on (described below) contain data for every month since in April 2024 (the first month for which data was available, following the launch of the interim service at the end of March 2024), for other sources, data has only been captured since the full launch of the service in November 2024.

Appointment records from Citizens Advice Scotland and Seraphus

Project records from CAS include details of over 1,000 booked appointments with current and prospective migrants, out of which 804 went ahead. These records provide key characteristics of those attending appointments (including current location, age, sex, nationality, ethnicity, and reason for requesting appointments), as well as a summary of the advice provided.

Seraphus records cover close to 200 booked appointments, out of which 146 appointments went ahead and were attended by employers or investors, including organisations currently based in Scotland as well as those considering operating from Scotland in the future. Records include characteristics of employers and investors attending appointments (including their location, nationality and sector), as well as in indication of the subject of appointments and a summary of the advice provided.

It should be noted that statistics on appointments reported in this early insights evaluation, may differ slightly from those presented in quarterly SMS statistics publications^[30].

Web analytics data

The SMS web analytics dashboard captures website traffic for pages linked to the service, including pages on Scotland.Org and Scotland’s Migration Service for Employers and Investors The dashboard contains data for each month since the interim launch of the service in late March 2024, though the number of web pages which SMS oversees has significantly grown since this time.

Service accessibility data

Evidence on the accessibility of the digital service is drawn from an accessibility audit carried out by a third-party accessibility testing organisation in the Beta phase of the service. This audit was carried out in line with DSSS requirements, and commissioned to test the service’s compliance relative to WCAG 2.2 level AA, which is a standard all government services must meet in relation to accessibility for disabled users. The second indicator was based on a standardised readability assessment of the SMS’ online content, with a target for online content set at a reading age of 12. While the Scottish Government’s Service Manual recommends a reading age of between 9 and 11 years, this was not feasible for the service, given that some terms which had to be referenced as part of SMS’ online content (e.g. the names of specific visas) would automatically exceed this recommendation.

Service user feedback surveys

Service users were invited to fill out feedback surveys immediately after, and 3-6 months after, having an appointment with either CAS or Seraphus. These had a very low response rate, which is a common trend for feedback surveys and online surveys in general^[31]. The low response rate limited analysis and meant it was not feasible to test for statistical significance. Between September 2024 and August 2025, the proportion of service users completing feedback surveys^[32] was as follows:

• User feedback surveys completed by current or prospective migrants shortly after appointment: 3% (n=24)
• User feedback surveys completed by migrants 3-6 months after appointments: <1% (n=5)
• User feedback surveys completed by employers or investors shortly after appointment: 33% (n=37)
• User feedback surveys completed by employers or investors 3-6 months after appointments: 14% (n=16).

Qualitative research with users and service providers

Researchers from CAS carried out a small number of qualitative interviews and focus groups with service users and advisers for the purposes of this evaluation in September-October 2025. Focus groups were held with advisers from CAS (2 focus groups with 6 advisers) and from Seraphus (1 focus group with 3 advisers) to gather insights about their experiences of delivering the service and providing advice to migrants (CAS advisers) and employers and investors (Seraphus lawyers). Group discussions were prompted by questions linked to the type of users which advisers had interacted with, the most common advice and support provided, the extent to which advisers had felt able to support users and views on how the service could be improved in the future.

One-to-one interviews were held with employers (two interviews) and migrants who had relocated to Scotland at the time of their appointment (four interviews). Due to challenges with recruitment, eight migrant service users (six already living in Scotland [post-movers] and two looking to move to Scotland [pre-movers]) completed written feedback which answered the same questions asked in the four interviews. Interviews and written feedback with service users asked questions about the perceived impact of SMS and the user’s full SMS journey, including any barriers to accessing the service and suggested improvements.

Narrative accounts of SMS delivery and implementation

To inform the process component of the evaluation, the SMS Team was asked to submit a narrative account of work completed as part of the development, design and launch of the service and digital online platform. This overview was produced using a template developed by Scottish Government analysts, which asked the SMS Team to submit details of:

• The policy background to the service
• The extent to which the service had delivered intended outcomes
• The sources of evidence used to monitor service performance
• Challenges encountered and intended next steps for the service.

This was thematically analysed and triangulated with the other available qualitative and quantitative data on service delivery. This was then systematically assessed against the SMS theory of change to respond to EQ1.

Stakeholder feedback

To inform this evaluation, feedback on the service was gathered from eight stakeholders, including representatives from six universities (who engage with and represent migrant/international students) who submitted a joint response, as well as two employer and investor service users. Six additional organisations were approached for feedback but did not respond to this request.

Stakeholders who responded to the request for feedback were asked to comment on the perceived impact of the service on the group(s) that they represented and make recommendations for the future of the service. Identifying details have been limited in order to maintain the anonymity of stakeholders.

Analysis

Evidence described above was analysed and systematically assessed against the SMS theory of change. Descriptive statistical analysis was conducted to provide a summary of any data which could demonstrate the reach and impact of the service among migrants, employers, and investors. Quantitative data analysis was not tested for statistical significance due to small and unrepresentative sample sizes, which would have limited the reliability of inferential statistics. Using quantitative analysis, KPI data was analysed to provide mean, minimum and max responses.

Thematic analysis was carried out on the following evidence sources to identify key themes aligned with the evaluation questions and components of the Theory of Change:

• Usable open text responses from the migrant service user and employer service user feedback surveys
• Evidence from focus groups (CAS and Seraphus lawyers), interviews (employer and migrant service users), and the written responses to the interview questions (migrant service users).

The value for money chapter (Chapter 6) estimates the economic activity supported by the service by user group to date, focusing on migrants, employers, and investors using feedback survey data and assumptions about average migrant salaries to calculate gross value added (GVA) to the economy. This GVA estimate is then compared to service delivery costs to date to produce an illustrative benefit-to-cost ratio.

Limitations

There are several limitations of the data which should be noted when interpreting the findings in this report.

Potential bias

As set out above, analysis relied primarily on existing monitoring and feedback data, which is limited in its ability to capture the complexity of experiences of users and progress towards the nuanced SMS outcomes. Feedback data is particularly susceptible to bias due to the self-selected nature of samples and self-reported experiences. This means that the validity and reliability of results may be skewed, as individuals who have particularly strong views, either positive or negative, are often more likely to respond to feedback surveys than those who have had more routine experiences. As such, findings drawn from this data should be interpreted with caution, especially considering the small sample size highlighted above. Findings should also be interpreted in light of the external factors outlined in this chapter, as it has not been possible to isolate the influence of these.

Small sample sizes

Evidence of user experiences was gathered through surveys completed after users attended appointments and webinars, as well as through qualitative interviews. Both of these sources have small sample sizes which cannot be considered representative of SMS users. Between September 2024 and August 2025, a total of 29 migrants had completed a user feedback form on their experiences of appointments (5 completed 3-6 months after appointments), and 53 had been completed by employers and investors (16 completed 3-6 months after appointments).

Low uptake of invitations to participate in qualitative research meant that only four migrant interviews went ahead, with a further eight migrants responding to interview questions via MS Forms. Only two of these participants were migrant service users who had not relocated to Scotland at the time of their engagement with the service. Similarly, there was low uptake among employer and investor users, meaning only two interviews with employers went ahead and no investors were interviewed. Both employers represented Scottish NHS boards which impacted the ability of the qualitative evidence to explore experiences across different sectors. Regardless, this qualitative evidence is valuable in the rich insights it generated in terms of user experiences and the impact of the service.

The above sample limitations mean this evaluation has been constrained in terms of what conclusions it can draw on the impact of the service, as we were not able to establish a sample that was representative of the full user base. This has also meant it has not been possible to explore how user experiences may have differed based on socio-demographic characteristics or by type of organisation, as it has not been possible to break down findings linked to these variables in a way that avoids individual service users being identifiable in the final evaluation report.

Socio-demographic characteristics

Relatedly, while CAS intended to collect data on users’ socio-demographic characteristics for appointments with the service, this data was not collected in full for all users. In particular, the ethnicity of migrants booking appointments was unavailable for 59% of appointments. This therefore further limited the extent to which we could assess whether user experiences of the service varied by socio-demographic characteristics.

Data quality and availability

A number of data quality and availability issues were encountered during analysis which affected the ability of the evaluation to answer specific evaluation questions and to link evidence to some components of the SMS Theory of Change.

• Key Performance Metrics: The KPIs which informed the evaluation of the service were only available from January 2025. This resulted in only 8 months of KPI data for the employer and investor focused service, and only 7 months of KPI data for the migrant focused service (given that no migrants submitted feedback surveys in May 2025). The low number of data points available combined with the small feedback survey sample sizes limit the reliability of conclusions concerning how SMS met user needs.
• Web Analytics: The ability to draw conclusions from the web analytics data has been impacted by some missing data and inconsistencies in how data was recorded. There is a discrepancy where the total page views of the SMS dashboard do not align with the sum of individual pages across different parts of the platform due to the way data is indexed, however a consistent approach is adopted to analysis to minimise the impact of this. These data gaps and inconsistencies could have impacted the conclusions drawn about the impact and reach of the SMS digital content as figures should be considered an estimate rather than exact number.
• Changes in feedback survey questions: New questions on overall satisfaction and on health and wellbeing were added to the CAS feedback survey in May 2025. At the time of analysis, there were only four months of data which meant this evidence could not be used to inform the impact evaluation.
• Inconsistencies in variables: Inconsistencies in how variables were recorded in the monthly appointment records required significant data cleaning and harmonisation before analysis could begin. This included correcting formatting inconsistencies and coding for incomplete or missing data. Some variables were not recorded until August 2024 which included capturing how migrant service users discovered the service.

Long-term outcome measurement

Measurement of long-term outcomes is also out of scope for this evaluation, given the service has not been operational for long enough to allow for this. This means we have not been able to robustly assess whether the service has contributed to outcomes such as migrant retention in Scotland, reducing skill shortages in the Scottish economy, or the wider contributions of migrants to Scottish communities, culture, public and private sectors.

The above limitations have constrained the ability of this evaluation to respond to its evaluation questions, which is highlighted at relevant points throughout the report, and the results of this evaluation should be interpreted in light of the above limitations.