The Scottish Government’s Quality Assessment of the Office for National Statistics Labour Force Survey and Annual Population Survey data for Scotland

2 Introduction

2.1 Office for National Statistics Labour Force Survey and Annual Population Survey

The Annual Population Survey (APS) is an important source for Scottish Government to produce statistics and describe the Scottish Economy. It helps us to understand the performance of the Scottish and local economies, covering key metrics such as employment, unemployment and economic inactivity.

The survey is also used to inform policy decisions, such as the commitments within the Scottish Government’s Programme for Government on Economic Growth and Child Poverty. These statistics are also a vital component of measuring progress and helping local authorities tailor policy decisions to meet local need.

The APS and Labour Force Survey (LFS) are UK wide surveys. The LFS is an in-person survey collected by the Office for National Statistics (ONS) in Great Britain (GB) and the Northern Ireland Statistics and Research Agency in Northern Ireland. The ONS has the role of combining the GB and Northern Ireland data to produce monthly labour market statistics for the UK based on rolling quarterly data.

The APS combines results from the LFS with local LFS boosts where additional questionnaires are issued to try and increase the response rate. The Scottish Government funds the local LFS boost for Scotland. The APS provides estimates based on rolling annual data on a quarterly basis.

With a larger sample, the APS provides more survey responses which are used to produce estimates for local authorities and smaller groups of the population in Scotland that are not possible using the LFS alone.

While the APS allows us to provide more detailed breakdowns, the LFS is more timely and is used to measure headline estimates of employment, unemployment, and economic inactivity for Scotland on a monthly basis. Headline estimates by sex are also sourced from the LFS. However, we have paused commenting on these statistics in recent months due to concerns regarding the declining sample sizes seen up to Jan-Mar 2024 and which we discuss in more detail later in this paper (see Detailed Quality Assessment Results for Labour Market Trends).

2.2 Challenges in gathering survey data

The LFS and APS rely on the participation of households to gather data. Each quarter, a number of GB addresses (25,800 since Jan-Mar 2024) are sampled from the Postcode Address File owned by Royal Mail. Interviewers visit addresses to collect household responses throughout the quarter. In Scotland, a different approach is used north of the Caledonian Canal and more details can be found in the Labour Force Survey User Guides published by ONS.

Over the last several years, and particularly since the COVID-19 pandemic, the number of households responding to the survey has been falling. This change in response can be represented by the number of households/people responding to the survey (the achieved sample, or sample size).

Figure 1: Household responses have fallen in nearly all years since the inception of the APS in 2004. A sharp decline was seen at the onset of the COVID-19 pandemic with the number of household responses continuing to decrease until 2023

Annual Population Survey achieved sample for households in Scotland, Jan-Dec 2004 through to Jan-Dec 2024

The survey is designed to gather a representative sample of the population but when less people respond to the survey there is added uncertainty. We do not know who we are missing and whether they are similar or different to the people who have responded. This is relevant for people who are less inclined to engage with interviewers, those who are wary or have lost trust in government in recent years, and groups who make up a small proportion of the population, such as minority ethnic groups.

With the sample size reducing in both surveys until 2023, we face added difficulties in presenting the Scottish data and reduced confidence in the estimates obtained from the APS and LFS. This has an impact on what data can be reported.

2.2.1 Challenges for the Labour Force Survey collection

In response to the COVID-19 pandemic, the ONS increased the wave 1 LFS issued sample size to mitigate the drop-in response rates. In July 2023, they reverted the issued sample size to pre-pandemic levels.

Excluding imputation (where people not responding to a wave 2 interview have their previous response used again), the decrease led to an achieved response rate of 12.7% in Jul-Sep 2023 which was the lowest response rate in 10 years. As a consequence of such a low response rate and with concerns for the quality of the estimates, ONS paused the publication of any country or regional breakdowns in October 2023 (Jun-Aug 2023 period).

ONS developed a plan to address these concerns and reintroduce LFS estimates. The plan covered both data collection and methodological improvements. The LFS was reintroduced as the headline measure of labour market statistics in February 2024 (Oct-Dec 2023 estimates) following the use of an alternative experimental adjusted series until the LFS estimates could be reinstated. ONS published an article which detailed the reintroduction of the LFS estimates and the impact of reweighting the key indicators for the UK. Since its reintroduction, the LFS has been classed as official statistics in development. As a result, users have been made aware that the estimates will potentially have a wider degree of uncertainty and the limitations of the statistics will be explained in each publication.

Figure 2: Response rates in Great Britain for the Labour Force Survey have decreased significantly since the onset of the pandemic, falling to a 10 year low in Jul-Sep 2023. Despite mitigation work by ONS to improve response rates since then, the response rate is still around half the rate seen prior to the pandemic

Response rates for Great Britain (excluding imputation) together with issued sample size indexed to the Jan-Mar 2019 period, Jan-Mar 2019 to Jul-Sep 2024.

The ONS increased the issued sample in Jan-Mar 2024 back to the level in Apr-Jun 2023 (1.5 times the Jan-Mar 2019 level). ONS continue to make improvements based on their plan and reweighted the LFS estimates again in December 2024.

In response to the issues with the LFS, ONS conducted analysis to assess the impact on the estimates obtained from the APS. ONS concluded that the APS estimates are robust at national and headline regional level (which includes Scotland). However, the impact of falling sample sizes in addition to the APS estimates not accounting for the new population statistics from the UK Censuses, led the Office for Statistics Regulation (OSR) to temporarily suspend the accreditation of all APS-based outputs.

This means APS-based outputs do not currently meet the standards of trustworthiness, quality and value in the Code of Practice for Statistics that producers of official statistics should adhere to. This means we have reduced confidence in the estimates obtained from the APS data, which has an impact on our ability to produce quality statistics on the economy to meet users' needs.

2.3 The Scottish Perspective

We continue to support the ONS efforts to improve the quality and reduce the variability in the LFS and APS estimates. However, the need to pause the LFS outputs in October 2023 and the de-designation of APS-based outputs to official statistics in development have impacted on the statistics we can produce for Scotland.

Due to the LFS being a UK level survey, a reduction in the sample is to be expected when focussing on Scotland data alone. The reduction in sample sizes seen in the LFS since the pandemic has been more pronounced in the Scottish data than at the UK level. The APS was created to boost sample sizes in order to produce estimates for local authorities and smaller groups of the population that are not possible using the LFS alone. However, the APS has also seen a considerable reduction in sample sizes since the pandemic that is more pronounced in Scotland due to smaller overall sample sizes than at the UK level. Due to these falling sample sizes, the Scottish data from the LFS and APS is showing increased volatility and uncertainty, making it difficult to draw conclusions from the data.

This was highlighted by OSR in their response to ONS where they stated “ONS must ensure that its analysis of the quality of APS-derived statistics is sufficient to enable the Scottish Government and Welsh Government, and other producers of APS-derived statistics, to reach a view on the quality of their APS-derived statistics.”

The ONS recently stated that it is still appropriate to use the estimates from the survey however users should be aware of these issues, in particular, the increased uncertainty around these estimates in recent years. Although some analysis has been undertaken below the UK level, the focus of the ONS assessment is primarily on UK estimates. This does not inform the situation for estimates at or below the Scotland level in any detail, which is the data the Scottish Government rely on.

The majority of users of the Scottish Labour Market Statistics require data below the headline Scotland level. Specifically, users require high quality labour market data for Scotland broken down by Local Authority and protected characteristics as a minimum. In addition, many users of the Scottish data require intersectional data, for example, breakdowns by age and sex or by age and disability.

Based on the ongoing data quality issues and the uncertainty around the impact on the Scotland level data, the Scottish Government took the decision in November 2024 to stop publishing data below Scotland level, including for any ad hoc queries until a full quality assessment had been undertaken. The Scottish Government’s Chief Statistician released a statement explaining the impact of falling sample sizes on Scottish data and what we are doing about it.

The Scottish Government continue to work with Welsh and Northern Irish Governments and our colleagues in ONS to better understand the quality issues in the Scottish data.

In October 2024, ONS held an evidence gathering workshop with users as part of their quality assessment. We shared our required minimum viable sample sizes per data category with ONS. We advised ONS that to publish high quality Scottish Labour Market estimates we would require a sample of 30 or more individuals for each category e.g. minimum of 30 individuals with a disability and 30 individuals without a disability. As an example, if we were to achieve breakdowns by sex using 30 individuals per characteristic, we would require a total sample size of 57,600 individuals at Scotland level (1.67% of the population).

This compares to an achieved overall sample from the APS of approximately 14,000 individuals for Scotland in Jan-Dec 2023.

Users of Scottish Labour Force Survey or Annual Population Survey data

The Scottish Government is responsible for reporting on Labour Market Statistics for Scotland. There are a number of publications that are produced based on the LFS and APS data which will be described later in this paper. There are also other key outputs that we produce or supply data for, such as:

• Briefings on labour market statistics for Scotland released by ONS
• National Performance Framework (NPF) indicators
• National Strategy for Economic Transformation
• Fair Work Action Plan/Evidence Plan
• Wellbeing Economy Monitor
• Child Poverty indicators
• Gender Equality Index
• Scottish Economic Insights
• Scottish Economic Bulletins

As well as publications and frameworks, we receive a high volume of ad-hoc queries on the labour market that require LFS and APS data. These include requests for data, statistics or analysis from Scottish Government and UK Government policy colleagues, internal and external stakeholders, parliamentary questions, other public bodies such as the Scottish Fiscal Commission, and other users of the labour market statistics for Scotland. For example, policy teams in areas such as Health, Justice, Child Poverty, Housing, Trade, and Parental Support all use the Scottish Labour Market Statistics supplied by the LFS and APS to help shape and evaluate policies to improve the lives of the people of Scotland