Statistics.gov.scot improvement project: alpha user research report

Appendix B (user testing scripts)

Round 1

General introduction (5 min)

Thanks for taking the time to speak with me today. I’m from Storm ID, and we’re working with the Scottish Government Open Data Team, to conduct research into the usability of a prototype statistics website.

[Optional] I have some colleagues on the call. They are here to observe the session and take notes. During the session, they will have their microphones and cameras off, so will be in the background as you and I chat. Are you happy for them to stay on the call to observe the session?

[Intros here or before ‘First impressions’]

If you’re comfortable with us doing so, we’d like to record this session. This would be recording the screen, audio, and video. We ask to do this so that we can go back and check the recording when writing our report on how the session went. We don’t share these recordings with anyone, and once the report is complete, the recordings will be deleted. If you’d rather we don’t record, we can still continue with the session.

Are you happy for me to turn on the recording? [Turn on recording]

The session will take up to 60 minutes.

This session is entirely voluntary and if at any point you’d like to take a break or stop the session you just need to let me know. You do not need to give a reason.

We’re going to ask you to share your screen, tell us what you think of the prototype, and complete a few tasks using the prototype – please just give each one your best attempt. Please talk us through your thinking as you work through each task, and don’t worry if you get stuck – that’s useful for us to see.

Sometimes I’ll ask things that seem really obvious, and I might seem to be asking the same thing repeatedly – please be patient with me and trust me, it is all useful.

We hope you’ll be comfortable giving us your honest opinion. Ultimately, feedback or comments gathered in these sessions will help us improve what we’re doing. I’m interested in how you find the interface and where it could be clearer. There are no right or wrong answers – we’re testing the prototype, not you.

At the end I’ll ask you to complete a very short survey – it should only take a few minutes.

Any comments and feedback you share with me will be used only for the purposes of this research. Direct quotes or opinions will be anonymised before we include them in our report. 

Do you have any questions for me before we begin? 

Participant to navigate to

Cobalt Open Data Portal Prototype

or

Emerald Open Data Portal Prototype

or

Cobalt Open Data Admin Portal Prototype

and share browser window.

First impressions (5 mins)

• Take a few moments to look at the homepage.
• What is your first impression?
• What do you notice first?
• What information seems most useful to you here?
• What do you expect you can do here?
• Is there anything of interest to you on just this page?
• Anything missing which you expected to see?
• What does [datasets, organisations, themes etc.] mean to you?

Specific tasks by user group follow.

General citizens and inquiring citizens

Task 1: Check the figures behind a news headline (finding data, searching) (5-10 mins).

Note: go back to the homepage.

Scenario: Imagine you came across a news headline claiming the number of police officers in Scotland is at a 'record low.’ Using the site, find the official data behind this headline to check if it’s accurate.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• How easy or difficult was it to find the data to verify the headline?
• Did the information you found match what you expected to see?
• Is there anything you would have preferred to see (e.g. a summary or graph) to help understand the headline?

Observation notes:

• Does the participant use the search bar or navigate through categories to find the data?
• Are they confused by any terminology or by the way in which results are presented?
• Do they notice details like dates or organisation name? Did these give them confidence?

[Optional] Task 2: Look up local COVID-19 case trends (finding, searching) (5-10 mins)

Note: go back to the homepage.

Scenario: You want to check recent COVID-19 case numbers in your local area. Try to find data that would help you think about this.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• How was that experience?
• Did you find the information you were looking for?
• What would you do next (e.g. look for more details, compare to other areas)?

Observation notes:

• Did the participant want to view the data or download it? Or did they ignore or miss this?
• If they attempted to visualise the data, then how did they get on? Was it helpful? Why or why not?
• Where there are any signs of frustration, e.g. repeatedly clicking or scrolling?

[Inquiring citizen] Task 3: Compare GDP between two industry sectors (finding, searching, viewing) (up to 10 mins)

Note: go back to the homepage.

Scenario: You want to compare the GDP of two industry sectors (for example, manufacturing and agriculture). Use the site to find data for each of two industries and see which one has a higher output or growth.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• How easy was it to find information for both sectors?
• Were you able to make a comparison from the data you found? What would have made this easier for you?
• Do you feel confident that you found the right data to answer your question? Why or why not? Was anything missing?

Observation notes:

• Did the participant find the data easily?
• Was there any difficulty in interpreting the data?
• Did the user seem happy enough throughout the process? Were there any particular sticking points for them?

Task 4: Find publisher contact details (5 mins)

Note: go back to the homepage.

Scenario: You are interested in contacting the publisher of the data to ask if they have found anything interesting in their analysis. Find out how you would do this.

• Where do they begin?
• What are they thinking as they look for this?
• Where would they typically expect to see contact information?

Follow-up questions:

• How would you typically expect to get in contact with a data publisher? (e.g. email, ‘phone, contact form etc.)
• Why might you get in touch with a data publisher?

Observation notes:

• Did the participant hesitate when looking for this information?

Task 5: Locate help or support (5 mins)

Scenario: Let’s say you need help while using the site – for instance, understanding how to find data. Show me how you would find help or support on the site.

Follow-up questions:

• Was it clear where you should go for help?
• Was the help you found useful?
• Was the help where you expected to find it?

Observation notes:

• How did the participant look for help? Did they see it in the top nav, or scroll to the footer etc.?
• Did the participant distinguish between contacting the Open Data Team and contacting a specific data publisher?

System Usability Scale

Ask participant to complete the SUS.

Discussion (5 min)

• How would you describe your overall experience using this site? (Any particular words come to mind?)
• What did you like best about the site, and what did you like least or find frustrating?
• Would you trust the data on this site is reliable? Why or why not?
• If you could change one thing about this site then what would it be?

[Thanks and close.]

Commercial users

Task 1: Forecast hotel occupancy (finding, searching, filtering, viewing, downloading) (10 mins)

Scenario: You are trying to forecast hotel occupancy rates as part of a post-COVID tourism recovery analysis. Using the site, find data on tourism or hotel occupancy in Scotland that could help you forecast future occupancy trends.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• How was that experience?
• Did the data you found seem sufficient for forecasting purposes?
• If you were doing this for work then would you feel confident using the data from this site? Why or why not?

Observation notes:

• Did the participant have a clear strategy, like searching for specific keywords or browsing by theme?
• Did they encounter any difficulties understanding the data?
• What were their instincts around downloading and/or analysing data?

Task 2: Review energy efficiency (finding, searching, filtering, viewing, downloading)(10 mins)

Scenario: Let’s say your company provides energy-efficiency services. You want to identify areas with poor energy efficiency to target your marketing. Use the site to find data about the energy efficiency of buildings in Scotland.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• How easy was it to find relevant data?
• Do you think the data you found would help in this scenario? If not, then what other information would you need?
• Was any terminology confusing?

Observation notes:

• Did the participant start with search, or browse by theme?
• Did the participant struggle with any terminology?
• Did the participant use filters to narrow down e.g. area?

Task 3: Find publisher contact details (5 mins)

Note: go back to the homepage.

Scenario: You are interested in contacting the publisher of the data to ask if they have found anything interesting in their analysis. Find out how you would do this.

• Where do they begin?
• What are they thinking as they look for this?
• Where would they typically expect to see contact information?

Follow-up questions:

• How would you typically expect to get in contact with a data publisher? (e.g. email, ‘phone, contact form etc.)
• Why might you get in touch with a data publisher?

Observation notes:

• Did the participant hesitate when looking for this information?

Task 4: Locate help or support (5 mins)

Scenario: Let’s say you need help while using the site – for instance, understanding how to find data. Show me how you would find help or support on the site.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• Was it clear where you should go for help?
• Was the help you found useful?
• Was the help where you expected to find it?

Observation notes:

• How did the participant look for help? Did they see it in the top nav, or scroll to the footer etc.?
• Did the participant distinguish between contacting the Open Data Team and contacting a specific data publisher?

System Usability Scale

Ask participant to complete the SUS.

Discussion (5 min)

• How would you describe your overall experience using this site? (Any particular words come to mind?)
• What did you like best about the site, and what did you like least or find frustrating?
• Would you trust the data on this site is reliable? Why or why not?
• If you could change one thing about this site then what would it be?

[Thanks and close.]

Technical/Expert users

Task 1: Pull carbon footprint data via API (finding, using API) (10 mins)

Scenario: You’re working on a project about carbon footprints, and you need to pull the latest carbon footprint data via the site’s API. Find a relevant dataset and figure out how you would retrieve that data using an API.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• How straightforward was it to find the dataset and API?
• Was the API presented as you would expect?
• Is there anything about the process that might slow you down?

Observation notes:

• Does the user appear comfortable with the terminology?
• Note any points where the user hesitates or backtracks - do they see the ‘API’ button immediately?
• Does the participant comment on data format (JSON, CSV, etc.) or any metadata? Is the layout and terminology meeting their expectations?

Task 2: Gather data-zone boundaries and lookup table (finding, searching, data formats) (10-15 mins)

Scenario: You need to create a map of statistical data, which requires geospatial boundary files and a lookup table for data zones in Scotland. Find a suitable dataset and the corresponding data zone lookup table (a CSV or similar that maps data zone codes to area names).

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• How was the experience of finding these?
• What would you do next?

Observation notes:

• Does the participant search for a specific dataset, broader terms like ‘data zone’ or ‘GeoJSON’, or do they browse and/or filter datasets?
• What does the participant do first on the dataset landing page?
• Does the participant find the map to view?

Task 3: Find publisher contact details (5 mins)

Note: go back to the homepage.

Scenario: You are interested in contacting the publisher of the data to ask if they have found anything interesting in their analysis. Find out how you would do this.

• Where do they begin?
• What are they thinking as they look for this?
• Where would they typically expect to see contact information?

Follow-up questions:

• How would you typically expect to get in contact with a data publisher? (e.g. email, ‘phone, contact form etc.)
• Why might you get in touch with a data publisher?

Observation notes:

• Did the participant hesitate when looking for this information?

Task 4: Locate help or support (5 mins)

Scenario: Let’s say you need help while using the site – for instance, understanding how to find data. Show me how you would find help or support on the site.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• Was it clear where you should go for help?
• Was the help you found useful?
• Was the help where you expected to find it?

Observation notes:

• How did the participant look for help? Did they see it in the top nav, or scroll to the footer etc.?
• Did the participant distinguish between contacting the Open Data Team and contacting a specific data publisher?

System Usability Scale

Ask participant to complete the SUS.

Discussion (5 mins)

• How would you describe your overall experience using this site? (Any particular words come to mind?)
• What did you like best about the site, and what did you like least or find frustrating?
• Would you trust the data on this site is reliable? Why or why not?
• If you could change one thing about this site then what would it be?

[Thanks and close.]

Public sector/policy influencers

Task 1: Brief minister on child obesity (finding, searching, filtering, visualising) (10 mins)

Scenario: You are preparing to brief a government minister on child obesity rates for an upcoming meeting. Use the site to find the latest statistics on child obesity in Scotland.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• How was that experience?
• Was it easy to understand the data?
• Would you feel confident citing this site’s data to a minister? Why or why not?

Observation notes:

• Did the participant prefer to search or browse to find the data? Were they successful?
• Did they focus on finding a single headline figure or try to gather additional context?
• Did they have any difficulty interpreting the data?

Task 2: Analyse ageing population (finding, searching, filtering, visualising) (10 mins)

Scenario: You’re working on a social care strategy for an ageing population. You need data to understand the scope of the issue, e.g. how fast the population is ageing, or what proportion of the population is over a certain age. Use the site to find statistics on the ageing population in Scotland.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• Did you find what you needed to help inform the strategy?
• How was the experience of finding this data?
• Is the data presented in a way that’s helpful to you?

Observation notes:

• If searching, then did the participant use specific terms (e.g. “ageing population”) or broader terms (e.g. “age”)?
• Did they make any comment on the details of the dataset, e.g. the date last updated?
• Did the participant hesitate at any points, e.g. when visualising data?

Task 3: Evaluate policy outcomes (finding, searching) (10 mins)

Scenario: You need to evaluate the outcomes of a carbon reduction policy – say the government aimed to cut the carbon footprint or emissions by a certain percentage. Use the site to find data on carbon emissions or carbon footprint trends in Scotland.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• Could you find what you needed?
• Did you find it easy to interpret the data?
• What would you do next?

Observation notes:

• Did the participant prefer to use search or navigate by themes?
• Did the participant look for charts or another visualisation?
• Did the participant pay any attention to descriptions and/or metadata? Did they appear to understand this? Did they find it useful?

Task 4: Find publisher contact details (5 mins)

Note: go back to the homepage.

Scenario: You are interested in contacting the publisher of the data to ask if they have found anything interesting in their analysis. Find out how you would do this.

• Where do they begin?
• What are they thinking as they look for this?
• Where would they typically expect to see contact information?

Follow-up questions:

• How would you typically expect to get in contact with a data publisher? (e.g. email, ‘phone, contact form etc.)
• Why might you get in touch with a data publisher?

Observation notes:

• Did the participant hesitate when looking for this information?

Task 5: Locate help or support (5 mins)

Scenario: Let’s say you need help while using the site – for instance, understanding how to find data. Show me how you would find help or support on the site.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• Was it clear where you should go for help?
• Was the help you found useful?
• Was the help where you expected to find it?

Observation notes:

• How did the participant look for help? Did they see it in the top nav, or scroll to the footer etc.?
• Did the participant distinguish between contacting the Open Data Team and contacting a specific data publisher?

System Usability Scale

Ask participant to complete the SUS.

Discussion (5 mins)

• How would you describe your overall experience using this site? (Any particular words come to mind?)
• What did you like best about the site, and what did you like least or find frustrating?
• Would you trust the data on this site is reliable? Why or why not?
• If you could change one thing about this site then what would it be?

[Thanks and close.]

Data publishers (regular and infrequent)

Task 1: Data upload (validation, publication) (15 mins)

Scenario: You have new data that needs to be published on the site (e.g. quarterly data, annual data, or one-off dataset). Show how you would add a new dataset to the platform.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• Was it clear how to start the process of adding a new dataset?
• Did anything confuse you or slow you down?
• How did this compare with any other similar systems you’ve used?

Observation notes:

• Did the participant find the process intuitive? Did any particular part seem more challenging?
• How do they handle metadata?
• Did they try to access help or guidance?

Task 2: Correct previous release error (maintenance) (10-15 mins)

Scenario: There’s a mistake in a dataset you published earlier. How would you correct or update an existing dataset on the site?

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions (after Task 2):

• How would you describe the process of updating an existing dataset
• How did you feel about version control or how users would know the data had been updated?
• What would you expect to happen next?

Observation notes:

• How does the user navigate to the dataset?
• How did they find the edit/update function? Where did they look for it first?
• Did they seem concerned about the update process in any way? Or were they confident it was all robust?

[Optional] Task 3: Respond to user feedback (5 mins)

Scenario: A user reached out with feedback that your dataset’s documentation was unclear. Show what you would do to respond to this feedback.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• Were you able to find where to update the details of the dataset?
• Would you like to be able to communicate with users directly using the site?
• Do you feel the site supports you in providing clear information to users? Why or why not?

Observation notes:

• Did the participant easily find out how to edit just the details of the dataset?
• Did the participant mention anything about help, guidelines or standards?

Task 4: Locate publisher help or support (5 mins)

Scenario: Let’s say you need help while using the site – for instance, understanding how to fix a validation error. Show me how you would find help or support for publishers on the site.

• Where do they begin?
• What are they thinking as they look for this?

Follow-up questions:

• Was it clear where you should go for help?
• Was the help you found useful?
• Was the help where you expected to find it?

Observation notes:

• How did the participant look for help? Did they see it in the top nav, or scroll to the footer etc.?
• Did the participant prefer to contact the team directly or use the help content?

System Usability Scale

Ask participant to complete the SUS:

Discussion (5 mins)

• How would you describe your overall experience using this site? (Any particular words come to mind?)
• What did you like best about the site, and what did you like least or find frustrating?
• Would you trust the publishing process to present your data correctly? Why or why not?
• If you could change one thing about this site then what would it be?

[Thanks and close.]

Round 2

General introduction (5 min)

Thanks for taking the time to speak with me today. I’m from Storm ID, and we’re working with the Scottish Government Open Data Team, to conduct research into the usability of a prototype statistics website.

[Optional] I have some colleagues on the call. They are here to observe the session and take notes. During the session, they will have their microphones and cameras off, so will be in the background as you and I chat. Are you happy for them to stay on the call to observe the session?

[Intros here or before ‘First impressions’]

If you’re comfortable with us doing so, we’d like to record this session. This would be recording the screen, audio, and video. We ask to do this so that we can go back and check the recording when writing our report on how the session went. We don’t share these recordings with anyone, and once the report is complete, the recordings will be deleted. If you’d rather we don’t record, we can still continue with the session.

Are you happy for me to turn on the recording? [Turn on recording]

The session will take up to 60 minutes.

This session is entirely voluntary and if at any point you’d like to take a break or stop the session you just need to let me know. You do not need to give a reason.

We’re going to ask you to share your screen, tell us what you think of the prototype, and complete a few tasks using the prototype – please just give each one your best attempt. Please talk us through your thinking as you work through each task, and don’t worry if you get stuck – that’s useful for us to see.

Sometimes I’ll ask things that seem really obvious, and I might seem to be asking the same thing repeatedly – please be patient with me and trust me, it is all useful.

We hope you’ll be comfortable giving us your honest opinion. Ultimately, feedback or comments gathered in these sessions will help us improve what we’re doing. I’m interested in how you find the interface and where it could be clearer. There are no right or wrong answers – we’re testing the prototype, not you.

Towards the end I’ll ask you to complete a very short survey.

[If required based on accessibility needs] If you would like, then I can help you with this by reading out the options and filling in the form on your behalf.

The survey asks you about your agreement or disagreement with the following statements:

• I think that I would like to use this website frequently.
• I found the website unnecessarily complex.
• I thought the website was easy to use.
• I think that I would need the support of a technical person to be able to use this website.
• I found the various functions on this website were well integrated.
• I thought there was too much inconsistency on this website.
• I would imagine that most people would learn to use this website very quickly.
• I found the website very awkward to use.
• I felt very confident using the website.
• I needed to learn a lot of things before I could get going with this website.

The choices for each statement are: Strongly disagree, Disagree, Neither agree nor disagree, Agree, or Strongly agree. [End accessibility needs section]

Any comments and feedback you share with me will be used only for the purposes of this research. Direct quotes or opinions will be anonymised before we include them in our report. 

Do you have any questions for me before we begin? 

Participant to navigate to

Cobalt Open Data Platform

or

Emerald Open Data Platform

or

Workflow Manager Prototype

and share browser window.

First impressions (5 mins)

• Take a few moments to browse the homepage.
• What is your first impression?
• What do you notice first?
• What information seems most useful to you here?
• What do you expect you can do here?
• Is there anything of interest to you on just this page?
• Anything missing which you would expect?
• What does [datasets, organisations, themes etc.] mean to you?

Specific tasks by user group follow.

Accessibility testing sessions (inc. general citizens, inquiring citizens, commercial users)

Task 1: Check the figures behind a news headline (finding data, searching) (5-10 mins).

Note: go back to the homepage.

Scenario: Imagine you came across a news headline about the differences between how much men and women are paid in Scotland. Using the site, find the official data behind this headline.

• Where do they begin?
• What are they thinking as they try to find this?

Follow-up questions:

• How easy or difficult was it to find the data to verify the headline?
• Did the information you found match what you expected?
• Is there anything else you would have wanted to help understand the headline?

Observation notes:

• Does the participant use the search bar or navigate through categories to find the data?
• Are they confused by any terminology or by the way in which results are presented?
• Do they notice details like dates or organisation name? Did these give them confidence?

[Optional] Task 2: Look up local COVID-19 case trends (finding, searching) (5-10 mins)

Note: go back to the homepage.

Scenario: You want to check recent COVID-19 case numbers in your local area. Try to find data that would help you think about this.

• Where do they begin?
• What are they thinking as they try to find this?

Follow-up questions:

• How was that experience?
• Did you find the information you were trying to find?
• What would you do next (e.g. try to find more details, compare to other areas)?

Observation notes:

• Did the participant want to view the data or download it? Or did they ignore or miss this?
• If they attempted to visualise the data, then how did they get on? Was it helpful? Why or why not?
• Where there are any signs of frustration, e.g. repeatedly clicking or scrolling?

[Optional] Task 3: Review energy efficiency (finding, searching, filtering, viewing, downloading)(10 mins)

Scenario: Let’s say your company provides energy-efficiency services. You want to identify areas with poor energy efficiency to target your marketing. Use the site to find data about the energy efficiency of buildings in Scotland.

• Where do they begin?
• What are they thinking as they try to find this?

Follow-up questions:

• How easy was it to find relevant data?
• Do you think the data you found would help in this scenario? If not, then what other information would you need?
• Was any terminology confusing?

Observation notes:

• Did the participant start with search, or browse by theme?
• Did the participant struggle with any terminology?
• Did the participant use filters to narrow down e.g. area?

Task 4: Find publisher contact details (5 mins)

Note: go back to the homepage.

Scenario: You are interested in contacting the publisher of the data to ask if they have found anything interesting in their analysis. Find out how you would do this.

• Where do they begin?
• What are they thinking as they try to find this?
• Where would they typically expect to find contact information?

Follow-up questions:

• How would you typically expect to get in contact with a data publisher? (e.g. email, ‘phone, contact form etc.)
• Why might you get in touch with a data publisher?

Observation notes:

• Did the participant hesitate when trying to find this information?

Task 5: Locate help or support (5 mins)

Scenario: Let’s say you need help while using the site – for instance, understanding how to find data. Show me how you would find help or support on the site.

Follow-up questions:

• Was it clear where you should go for help?
• Was the help you found useful?
• Was the help where you expected to find it?

Observation notes:

• How did the participant browse for help? Did they find it in the top nav, or scroll to the footer etc.?
• Did the participant distinguish between contacting the Open Data Team and contacting a specific data publisher?

System Usability Scale

Ask participant to complete the SUS.

Discussion (5 min)

• How would you describe your overall experience using this site? (Any particular words come to mind?)
• What did you like best about the site, and what did you like least or find frustrating?
• Would you trust the data on this site is reliable? Why or why not?
• If you could change one thing about this site then what would it be?

[Thanks and close.]

Data publishers (regular and infrequent)

Workflow Manager Prototype:

Task 1: Data upload (validation, publication) (15 mins)

Scenario: You have new NRS data that needs to be published on the site (e.g. quarterly data, annual data, or one-off dataset). Show how you would add a new dataset to the platform.

• Where do they begin?
• What are they thinking as they try to find this?
• At every stage of process
  • Is anything missing?
  • What do you expect to happen at next stage?

Follow-up questions:

• Was it clear how to start the process of adding a new dataset?
• Did anything confuse you or slow you down?
• How did this compare with any other similar systems you’ve used?

Observation notes:

• Did the participant find the process intuitive? Did any particular part seem more challenging?
• How do they handle metadata?
• Did they try to access help or guidance?

Task 2: Correct previous release error (maintenance) (10-15 mins)

Scenario: There’s a mistake in a dataset you published earlier. How would you correct or update an existing dataset on the site?

• Where do they begin?
• What are they thinking as they try to find this?

Follow-up questions (after Task 2):

• How would you describe the process of updating an existing dataset
• How did you feel about version control or how users would know the data had been updated?
• What would you expect to happen next?

Observation notes:

• How does the user navigate to the dataset?
• How did they find the edit/update function? Where did they try to find it first?
• Did they seem concerned about the update process in any way? Or were they confident it was all robust?

Task 3: Locate publisher help or support (5 mins)

Scenario: Let’s say you need help while using the site – for instance, understanding how to fix a validation error. Show me how you would find help or support for publishers on the site.

• Where do they begin?
• What are they thinking as they try to find this?

Follow-up questions:

• Was it clear where you should go for help?
• Was the help you found useful?
• Was the help where you expected to find it?

Observation notes:

• How did the participant try to find help? Did they find it in the top nav, or scroll to the footer etc.?
• Did the participant prefer to contact the team directly or use the help content?

System Usability Scale

Ask participant to complete the SUS:

Discussion (5 mins)

• How would you describe your overall experience using this site? (Any particular words come to mind?)
• What did you like best about the site, and what did you like least or find frustrating?
• Would you trust the publishing process to present your data correctly? Why or why not?
• If you could change one thing about this site then what would it be?

[Thanks and close.]