Using financial data to enhance labour market statistics in Scotland

Summary

Declining response rates to traditional labour market surveys have increased uncertainty in official labour market statistics, particularly for sub-national and demographic breakdowns. In response, the Scottish Government commissioned a feasibility study to assess whether pseudonymised consumer transaction data from banks could provide an additional source of labour market insight for Scotland and complement existing statistics. The study used a dataset sourced from NatWest Banking Group and was delivered by Smart Data Foundry (SDF). It explored the potential for financial transaction data to complement existing survey-based measures of employment, unemployment and economic inactivity in Scotland.

Smart Data Foundry was an Edinburgh-based, not-for-profit organisation that specialised in using financial data securely and responsibly to support research and analysis for the public good. The work undertaken by SDF included curating the source data, undertaking the analytical work and producing the supporting documents in this report.

The research utilised a Scotland-specific dataset sourced from NatWest Banking Group comprising approximately 575,000 pseudonymised banking customers. Customers aged 18 and over represented around 13.3% of the Scottish population. The work was undertaken across four phases:

• assessment of data representativeness
• development of transaction-based classifications of labour market status
• population scaling and uncertainty analysis
• recommendations for ongoing implementation

As direct labour market status was not available within the source data, a novel classification framework was developed combining expert-defined behavioural rules with unsupervised k-means clustering of financial activity. Using 26 transaction-derived features, the approach identified distinct patterns of financial behaviour that aligned with broad labour market categories.

The scope of the study was a proof-of-concept and it successfully demonstrated that pseudonymised financial transaction data can be used to generate stable proxy estimates of employment, unemployment and economic inactivity at both national and sub-national levels, and for demographic groups. These findings have relevance beyond Scotland and could support the development of supplementary labour market statistics across the UK and other devolved administrations. Beyond labour market statistics, consumer transaction data may also provide valuable insights into a range of wider socioeconomic issues, including the cost of living, household financial resilience, debt and borrowing patterns, and other emerging policy questions that are not yet fully understood.

While this study suggests significant potential for future analytical applications, further work is needed to focus on refining the classification methodology, strengthening validation, and assessing performance over time and across population groups to further enhance the value of this data.

For now, it provides compelling evidence that administrative financial data could enhance statistical insights across a range of economic and social policy areas, by offering a scalable, timely, and regular source of insight alongside traditional survey methods.

Key findings

Sample

• the Scotland-specific dataset was sourced from NatWest Banking Group.  The dataset used in the analysis contained approximately 575,000 customers across all 32 local authorities and represented around 13.3% of the Scottish adult population (aged 18 and over)
• the sample was a customer dataset rather than a random sample of the Scottish population, although representativeness testing in phase 1 found broad coverage
• the representativeness assessment found broad coverage across geography, key demographic, and socioeconomic characteristics
• coverage was lower and uncertainty was greater for younger people, particularly those aged 18 to 24, and classification performance was weaker for some groups, including women and certain age groups
• coverage of “hard to reach” areas such as rural areas and island local authorities was well represented in the sample compared to traditional surveys

Methodology

• the feasibility study developed a hybrid method combining expert-designed rules with unsupervised clustering of financial behaviour. Three broad groups were identified via the clustering method and these aligned with the study’s expert-led classifications of “employment”, “economic inactivity”, and “unemployment or benefits-reliant”
• labour market status was inferred from financial behaviour rather than observed directly because customer-level labour market status information was not available in the data. Therefore, the estimates derived from the feasibility study are proxy classifications and are not direct measures of labour market status
• reweighting was used to account for uneven coverage by local authority, age band and sex. Scotland-level estimates were stable over the period examined, while uncertainty was greater for smaller geographical areas and younger age groups
• confidence intervals primarily reflected sampling variation. They did not capture uncertainty arising from customer selection, classification error, methodological choices or differences from official definitions
• outputs were quality assured against official statistics and reference data sourced from the National Records of Scotland, the Office for National Statistics, the Department for Work and Pensions, and NOMIS with findings reviewed and challenged by an expert steering group

Differences from official definitions

• the analysis covered customers aged 18 to 64 which differs from the age range most often used for official labour market estimates
• the transaction-based measure of “unemployed or benefits-reliant” used in this study is different from the International Labour Organization definition of unemployment. Financial transactions cannot directly show whether someone is actively seeking work or available to start work

Conclusions

• this feasibility study used a novel approach to explore the use of pseudonymised financial transaction data as a potential new source of labour market insight for Scotland
• the findings demonstrate that labour market status can be inferred from financial behaviour, producing stable proxies for employment, unemployment and economic inactivity at national and sub-national levels
• the study provides compelling proof of concept that administrative financial data could complement traditional survey-based labour market statistics, offering more timely and granular insight than is increasingly possible through surveys alone
• while the results are promising, further methodological development, validation and governance would be required before the outputs could be considered suitable for official statistical use
• beyond labour market measurement, the richness, scale and frequency of financial transaction data present significant opportunities to support wider economic and social policy analysis, including areas such as financial resilience, cost of living pressures and economic wellbeing