Statistics.gov.scot improvement project: alpha user research report

Research overview

Background and objectives

In early 2024, the Scottish Government’s Open Data Team chose to run an Agile Discovery to better understand the statistics.gov.scot service and the environment in which it operates. The Discovery consisted of three workstreams, being user research, technology and process, and policy and landscape.

In summary, the Discovery found there to be a strong and urgent case for change. The current statistics.gov.scot service was (and is) not meeting user needs, and the current technology contains significant risk in terms of cost, support and longevity.

The Statistics.gov.scot Improvement Project - Alpha represents the second step on the journey to improving the service. This consisted of two workstreams, being Users and Technology. During this Alpha, the Open Data Team have prototyped and tested solutions to shape and inform what a future state service could look like. The purpose of the Users Workstream was to understand how well the Alpha prototypes address the needs and expectations of the service’s current and potential user base, and how the service could be improved to better address these. This primarily involved usability and accessibility testing of prototypes with a range of users. The overarching research question was: how can the open data portal be redesigned and developed to better serve its users and support governmental goals?

The team used Atlassian Jira and Confluence to respectively manage and document the project. While this report should be taken as the summative and final output of the Users Workstream, the repository of information and outputs on Confluence and Jira offer further details of activities, draft outputs, and ways of working.

Methodology

The programme of user research took a mixed methods, phased approach, comprised of moderated usability and accessibility testing with users (29, 60-minute sessions over two rounds) and a moderated and unmoderated survey (primarily consisting of the System Usability Scale; 50 total responses at time of writing). Round 1 of user testing focussed on general usability, resulting in various small iterative improvements to the prototypes, before Round 2 of user testing, which introduced specific accessibility-focussed sessions alongside further general usability testing of a data publishing prototype. Prototypes were subject to light touch accessibility reviews prior to each round of testing.

Prior to recruitment and any data generation, the project team completed a Data Protection Impact Assessment (DPIA), Risk Assessment Register, and User Research Ethics Plan. The DPIA was reviewed and approved as appropriate, with feedback incorporated where given. The Ethics Plan was reviewed and approved by an SG User Research Lead, with feedback incorporated where given.

Data collection

User groups

The Discovery research identified a set of six user groups, which were reviewed prior to recruitment in Alpha, and maintained for recruitment purposes. These are:

• general citizens: someone who may be occasionally interested in what’s behind the headlines that affect them (inc. those who self-identify with low digital literacy)
• inquiring citizens: someone who maintains a keen interest in specific issues and may occasionally use statistics (e.g. charity, third sector or think-tank employee)
• commercial users: someone who is interested in specific datasets that are useful for achieving business objectives, working in the private sector (e.g. at an energy company or bank)
• technical/expert users: working with and talking about data are part of their daily lives. (e.g. academia, data journalist, developers)
• public sector/policy influencers: somehow who researches specific issues and uses statistics as evidence to inform others. (e.g. policy advisers, statisticians, councils, health boards)
• data publishers: someone who collates and provides data to be published on the site (both regular and infrequent)

General citizens, inquiring citizens, and commercial users were recruited by an external recruitment agency according to a brief (see ‘Recruitment and participation’ below for more information). This was specifically to address the gaps in participation noted during the Discovery phase, specifically being a lack of general and inquiring citizen users and users with accessibility needs who may or may not use assistive technologies. During Discovery, we found it challenging to recruit these users as they tended to be beyond out usual professional and day-to-day networks. Despite this, we found issues with the site that affected all participants (including expert users), and surmised that these would similarly affect users who we were unable to recruit. Filling these gaps in Alpha ensured we had a broad and inclusive coverage of all the above groups, including those with accessibility needs.

Technical/expert users, public sector/policy influencers, and data publishers were recruited using convenience snowball sampling, where known contacts (including participants from Discovery) were invited to participate through established networks, and asked to pass on the recruitment information to any other interested parties (see ‘Recruitment and participation’ below for more information).

Recruitment and participation

Recruitment and participation was managed by the project team and via an external recruitment agency (which was selected from a shortlist of three, based on a balance of cost and service). The recruitment agency was engaged in order to recruit additional general citizens, inquiring citizens, and commercial users, alongside participants with a variety of accessibility requirements, to test both the usability and accessibility of prototypes for a broad and inclusive range of users. The recruitment agency has a signed non-disclosure agreement and DPIA in place with Storm ID, and is experienced in recruiting participants for SG. Given a recruitment specification, the recruitment agency screened external participants and scheduled research sessions on behalf of the project team.

The project team recruited participants via screener questionnaires. These vary between internal (SG) and external (non-SG) participants, as we asked for more personal data from external participants, including some (special category) health data, specifically about health conditions/impairments/disability (if volunteered). We were keen to include external users of assistive technology (which may include disabled users) and users with low digital literacy within the research. This is why our screening questionnaire for external participants asked about health data, which is special category personal data. We also asked all participants for any access requirements for sessions, which can constitute special category personal data.

From internal SG participants, the only personal data we required was their name and email address for contact purposes (we also asked for their job title and their experience with the service, which is not personal data).

All participants were given information that would ordinarily feature in a privacy notice on an information sheet, by the SG Open Data Team and/or the user researcher at Storm ID. The information includes:

• Background to project
• Who is collecting the information (details of all project partners transparent to participant)
• Why the work is being undertaken (purpose and legal basis of processing)
• Details of the session (what will happen)
• How information will be collected (written and digital notes, wireframe sketches)
• What information we collect (anonymised written and digital notes)
• That data is stored securely in password protected files on Scottish Government OneDrive and Scottish Government Confluence with access limited by role.
• Details of what SG Open Data Team and Storm ID do or are planning to do with the information.
• That the participant providing personal data is completely voluntary and they will give their agreement/permission to be contacted based on the information that the participant provides
• Information on their rights (e.g., we keep what you tell us private, you can ask to see what you have told us, you can ask us to rectify errors, you can ask us to delete what we know about you)
• Who will have access to the information provided (including information on publication of anonymised research reports)
• Length of time data will be kept for.
• Access to participant information and access will be restricted based on role, i.e. only those who require access to the data from The Scottish Government Open Data Team and Storm ID will have access to the data.
• Contact information for research team and in the event of a complaint.

This was accompanied by a permission form that was completed by the participant (digitally, or verbally if required, with a witness from the project team), which displayed permission requests clearly and prominently and asked individuals to positively opt-in. Permission from the participants is used to comply with our ethical requirements and is not for processing personal data. This was made clear to participants during research activities.

As noted above, general citizens, inquiring citizens, and commercial users were recruited by the external recruitment agency according to a brief. Technical/expert users, public sector/policy influencers, and data publishers were recruited using convenience snowball sampling, where known contacts (including participants from Discovery) were invited to participate through established networks, and asked to pass on the recruitment information to any other interested parties. The project team distributed recruitment information and screeners through personal invitations to participate, and the following online channels:

• Better Data Community - Microsoft Teams channels posts and newsletters.
• SG Viva Engage:
  • Scottish Official Statistics – Community.
  • Digital Data and Technology Profession (DDAT) – Community.
  • Digital Directorate - Community.
• SG Digital Directorate Newsletters.
• Office of the Chief Statistician (OCS) Newsletters on 28/05/2025, 02/06/2025 and 14/07/25.
• Open Data Scotland Slack channel.
• Trust and Transparency bulletin.

These posts reached 3000+ potential participants, of which 31 were scheduled for user testing and 29 participated (2 cancellations). Specific details of participant roles and responsibilities are not included here to prevent identification. Participants can be roughly aligned with our user groups as follows:

• general citizens (5)
• inquiring citizens (5)
• commercial users (4)
• technical/expert users (3)
• public sector/policy influencers (4)
• data publishers (both regular and infrequent) (8)

There are some overlaps in role, but each participant is only counted once, based on their principal use of statistics.gov.scot e.g. those working with data in the public sector who are data publishers are counted as data publishers.

Six participants with accessibility needs were recruited specifically for Round 2, being two general citizens, two inquiring citizens, and two commercial users. To this end, the recruitment brief included the following specifications:

All [six] participants should identify as having one or more accessibility requirements based on the following:

• Cognitive impairment (e.g. dyslexia).
• Motor impairment (e.g. paralysis, muscular dystrophy).
• Visual impairment.
• Deafness or hearing loss.
• Neurodiversity (e.g. ADHD, autism).

This should include:

• At least one participant who primarily navigates the web by keyboard.
• At least one participant who uses a screenreader (e.g. Voiceover) whilst using the internet.
• At least one participant who uses screen magnification whilst using the internet.

Another participant with accessibility needs was recruited as part of the wider recruitment. Participants also self-assessed their digital confidence as part of initial screening, with all participants during both rounds reporting generally high digital confidence.

Across all recruitment activities, we aimed for a reasonable gender split and spread of ages, with a mix of geographical locations across Scotland.

Overview of data collected

Data collected for this project included:

• Recruitment data (name, email and phone number to arrange session timings and information to cater to access requirements, demographic information including age, local authority, gender, disability or long-term health condition, experience with the service and job title where applicable).
• Research data (handwritten notes, digital notes).
• Data collected via audio and video recordings (contextual information, experiences of public services, feedback to prototypes).

Research activities

During moderated user testing sessions, participants were asked to complete realistic tasks using one of the four prototypes as appropriate to the user type, before completing the System Usability Scale (SUS) questionnaire, generating quantitative data via Microsoft Forms. The SUS is a ten-item Likert scale developed by John Brooke in 1996, that has since been validated and used extensively in various contexts to provide a high-level subjective and comparative measure of usability. We used the established ‘website’ variant of the SUS, in which the word ‘website’ is substituted for the word ‘system’ in each statement.

Audio and video data was collected via screen recording, with notes taken during each session by a maximum of two additional notetakers. Broadly, these sessions focused on task completion, comprehension, navigation, and trust in the prototype. All data was securely stored on SG systems, e.g. OneDrive and Confluence, with access limited by role.

Further quantitative and qualitative data was generated through an unmoderated survey of a wider pool of potential participants. The anonymous questionnaire primarily consisted of the SUS, plus some basic demographic information and free text comments (all optional), via Microsoft Forms. The questionnaires were specific to each prototype, linked to from each prototype’s phase banner, and contained an instruction to find data using the prototype prior to completing the form. The links to the questionnaires went live on the prototypes at the start of Round 2 of user testing.

Data analysis

Qualitative data from user testing and surveys was subject to a relatively swift and pragmatic version of template analysis, whereby initial coding of a subset of data formed a template to guide the analysis of the remaining data, with the template adjusted as required to incorporate new codes and sub-themes. Quantitative data from surveys was subject to a basic statistical analysis, using the SUS Analysis Toolkit, with a minimum SUS score of 74.1 set for benchmarking. 74.1 was chosen as a benchmark based on research conducted by Bangor, Kortum and Miller (2009), Sauro and Lewis (2016), and Lewis and Sauro (2018). While there is some very minor variation in categorising scores, there is general agreement across these studies of SUS scoring that a score of 74.1 rates the system being assessed as Above Average (relative to industry benchmarks) and Good (on the adjective scale), with a letter grade of B (below this being a B-). While this represents a reasonable benchmark for a live website, it is worth noting that we are in Alpha, testing fairly advanced prototypes, and so achieving a score of 74.1 or above at this stage would arguably bode well for a fully developed site.

Following analyses of qualitative and quantitative data, findings were shared for review and discussion with the project team. This was an iterative process, with findings becoming refined and stabilised through each round of data collection and analysis. While the Discovery work was more exploratory and thematic in understanding user experiences and needs relative to the current platform, the emphasis in Alpha was on testing and refinement, and so analysis naturally tended to produce more practical and specific findings, primarily to inform user and technical requirements.

As in Discovery, the research session schedule and participation were very well-organised by the Open Data Team, and participants were highly engaged, participative, and talkative. Although this report has a single author, the Users Workstream of this project was very much a collaborative effort, and the contributions of the Open Data Team and SG more widely in observing and/or participating in sessions should not be underestimated. Further participation in sessions with users is to be encouraged, as it offers the team a chance to gain first-hand awareness of real-life challenges faced by users, so prompting richer discussions about how to create a user-centred experience. This was certainly evident throughout this project.

Having outlined the methods used to generate and analyse data, we now turn to the findings and recommendations generated through that analysis.