Scottish Household Survey 2023 Data Explorer update - Breach report

Circumstances of breach

The Scottish Household Survey Data Explorer was scheduled to be updated with the 2023 survey results at 09:30 on 2 December 2024. The update was delayed until 20:00 on 2 December 2024.

The delay was caused by a technical issue deploying the app to the Shiny server. The app had been extensively tested locally (i.e. run from an R script on a laptop), but had not been tested for deployment to a remote server. When deployed, it became apparent that the script did not load correctly. The reason for this is that it did not reference a package (Markdown) that required loading for deployment to the Shiny server. This was not an issue when run locally, as Markdown is in-built for R Studio. Locating this issue and re-loading the data was time-consuming.

Impact of breach

Scottish Household Survey 2023 results on the following topics were published elsewhere, as part of planned topic-specific reports, and so users already had access to them at 09:30 on 2 December 2024: satisfaction with public services, trust in public institutions, perceptions of local councils, transport, discrimination and harassment, housing, household finances, environment and volunteering.

Scottish Household Survey 2023 results on the following topics were not available until the Data Explorer update at 20:00 on 2 December 2024: neighbourhoods, internet, physical activity, culture, childcare.

The Scottish Household Survey mailbox received one query, from SportScotland, who required the 2023 physical activity results for an urgent briefing. We responded promptly to explain the problem, and emailed them the physical activity tables directly.

Corrective actions (taken or planned) to prevent re-occurrence

This breach has highlighted that it is not possible for updates to the Data Explorer to ‘go live’ at a precise, pre-announced time. This is because the code to update the Data Explorer takes around 45-120 minutes to run, and the exact time it will take to run cannot be predicted accurately.

Therefore, we will not use the Scottish Household Data Explorer for the first release of statistics in future. Statistics will be added to the Data Explorer only when they have already been released in an official statistics publication on the Scottish Government website. Unlike the Data Explorer, these publications can be scheduled in advance to go live at a specified time.

Nevertheless, to ensure Data Explorer updates are as timely as possible, we have created a password-protected alternative web address for the Scottish Household Survey Data Explorer, where we can test deployment to the Shiny server in advance of updating the publicly available version.