Scottish Health Survey: code of practice compliance

Quality Management Approach Statement

1. Summary

The Scottish Health Survey (SHeS) is an Accredited Official Statistics publication that provides data about the health of the population living in private households in Scotland. The SHeS series was established in 1995, it was repeated in 1998 and 2003 and has been carried out annually since 2008.

This document provides information on the quality of the SHeS statistics.

2. Data Quality

This section provides users with information about the quality of the data.

Standard five of the Standards for Official Statistics in the Code of Practice for Statistics recognises that producers must support a quality culture that promotes good practice and encourages learning and improvement, under the direction of the Chief Statistician/Head of Profession for Statistics – so that the public can have confidence that published statistics are produced by organisations that continuously improve quality standards.

The European Statistics Code of Practice and more specifically the Quality Assurance Framework identifies five dimensions of data quality:

• relevance
• accuracy and reliability
• timeliness and punctuality
• accessibility and clarity
• coherence and comparability

This framework is used to assess the quality of the SHeS statistical outputs.

2.1 Relevance

Relevance is the degree to which statistics meet the needs of users.

User need

The purpose of SHeS is to provide information at national level about the health of the population and the ways in which lifestyle factors are associated with health. This level of information is not available from administrative or operational databases, as hospitals and GPs are not able to collect detailed information about peoples’ lifestyles and health- related behaviours. In addition, it is crucial that the Scottish Government has information about the health of the population, including people who do not access health services regularly.

The specific aims of SHeS are:

• To estimate the prevalence of particular health conditions in Scotland.
• To estimate the prevalence of certain risk factors associated with these health conditions and to document the pattern of related health behaviours.
• To look at differences between regions and between subgroups of the population in the extent of their having these particular health conditions or risk factors, and to make comparisons with other national statistics for Scotland, England, Wales and Northern Ireland.
• To monitor trends in the population’s health and health related behaviour over time.
• To make a major contribution to informing policy decisions, monitoring and evaluating current policies and monitoring progress towards health targets.

Types of users of SHeS data

The following sets out some of the main groups of SHeS users:

Government departments, agencies and policy makers - to inform policy decisions, allocate resources and evaluate programmes:

• Office of the Chief Statistician
• Health and Social Care Analysis Division
• National Performance Framework Unit
• Population Health Strategy and Improvement Division
• Drugs Policy Division
• Mental health Directorate
• Chief Medical Officer Directorate
• Dentistry and Optometry Division
• Children and Families Analysis Division
• Tackling Child Poverty and Social Justice Division
• Unpaid Carers Unit
• Food Standards Scotland
• Public Health Scotland

Local government - to plan services and make funding decisions at the community level:

• Health Boards
• Local Authorities

Academics and researchers - for policy analysis and research and to track societal trends:

• Glasgow Centre for Population Health
• Medical Research Council (MRC)
• Scottish Health Action on Alcohol Problems
• Several Universities

Charity organisations and community groups - identify community needs, advocate for policy change, support funding applications, evaluate and improve services and support research:

• Alcohol Focus Scotland
• ASH Scotland
• Asthma and Lung UK Scotland
• BEAT (eating disorder charity)
• Crisis (homelessness charity)
• Paths for all
• The Scottish Commission for People with Learning Disabilities
• Independent Food Aid Network

Media and journalists - to report on key social issues, hold policymakers accountable and inform public debate.  And the general public to stay informed about key issues, such as health risk

2.2 Accuracy and Reliability

The degree of closeness between an estimate and the true value.

Sampling selection, error and bias

The survey sample is designed to be representative of the population living in Scotland. As it is not possible to survey every household in Scotland, a proportion of addresses are selected, or sampled, each year in such a way as to be as representative of the whole population as possible.

The Royal Mail’s small user Postcode Address File (PAF) is used as the sample frame for the address selection. The advantages of using the PAF are as follows:

• It has previously been used as the sample frame for Scottish Government surveys so previously recorded levels of ineligible addresses can be used to inform assumptions for the  sample design
• It has excellent coverage of addresses in Scotland
• The small user version excludes the majority of businesses.

The PAF does still include a number of ineligible addresses, such as small businesses, second homes, holiday rental accommodation and vacant properties. A review of the previous performance of individual surveys found that they each recorded fairly consistent levels of ineligible address for each local authority. This means that robust assumptions can be made for the expected levels of ineligible addresses in the sample size calculations.

While SHeS is designed to have as representative a sample as possible, it is not possible to obtain a fully representative sample as participation is on a voluntary basis. Therefore, statistical adjustments are undertaken to ensure that the data is representative of the population of Scotland, according to the most up-to-date population estimates. For more information on the weighting approach used, see chapter 1 of the latest technical report.

Data collection

The SHeS data is collected by trained interviewers.  Interviewers use computer assisted personal (CAPI) interviewing for interviews done in-home and computer assisted telephone interviewing (CATI) for the telephone interviews.

A large number of quality control measures are built into the survey at the data collection stage and thereafter, to monitor the quality of interviewer performance. Quality checks are carried out at 10% of productive households. These recalls check with the participants that interviewers followed the correct survey procedures when conducting the interview.

In addition to the above quality checking procedures, the computer program used by interviewers has in-built soft checks (which can be suppressed) and hard checks (which cannot be suppressed) associated with particular interview questions. When uncommon or unlikely answers are entered, or answers outside a predetermined range, these checks are triggered and appear as a warning message on the interviewers’ laptop. The interviewer is either encouraged to double check the entered response (a soft-check) or asked to change it (a hard-check). For example, when young children are weighed by having an adult hold them; the weight of the adult on their own was entered into the computer followed by the combined weight of the infant and adult. A hard check is used to ensure that the weight entered for the adult alone did not exceed the weight of the infant and adult combined. Soft-checks are similar to hard-checks, however they could be suppressed. For example, soft-checks are applied to height measurements; if an interviewer enters a respondent’s height to be in excess of 1.93 metres (6 feet 3 inches), a message appears asking the interviewer to confirm that this entry is correct. The interviewer can suppress the soft-check once they confirm that the height entry is not a mistake.

Response

Survey response is an important indicator of survey quality as non-response can introduce bias into survey estimates. At each selected household in the main sample, all adults and a maximum of two children are eligible for interview. When considering the household response rate, households classed as “responding” are those where at least one eligible person opted-in/consented to interview and was interviewed. 

In SHeS, the number of adults per productive address increases with decreasing levels of deprivation. Combined with the increasing response rate as deprivation decreases, this leads to a skew in the number of responding adults in each deprivation quintile. Additionally,  the age distribution of adult respondents is generally older than the population as a whole and as a result younger age groups are under-represented in the SHeS sample when compared to the NRS mid-year population estimates. The weighting strategy for the survey takes into account the above.

Data processing and analysis

All data input by the interviewers is then sent to the data management team. Extensive cross-checking validation routines are applied and the potential for data processing errors is regarded as low risk.

Data analysis is done using mainly automated processes which minimise the risk of potential manual errors. Results are examined alongside previous outputs to ensure that they can be used to assess trends and that any changes are reasonable. The data is also checked against other analysis (for example in the SHeS dashboard) to ensure consistency.

Age-standardisation

Age standardisation is used in some analysis in order to enable groups to be compared after adjusting for the effects of any differences in their age distributions.

Rounding

All data reported in the publication text and accompanying tables are rounded to ensure robustness. Proportions are rounded to the nearest integer and mean scores to one decimal place. Significance testing is done using unrounded data.

Suppression

Shorthand is used in the excel tables to indicate where bases are less than 30 and these results are not presented; [u] = low reliability. Some caution is also needed when interpreting results where bases are between 30-49.

2.3 Timeliness and Punctuality

Timeliness refers to the lapse of time between publication and the period to which the data refer. Punctuality refers to the gap between planned and actual publication dates.

The suggested month of publication is announced on the Scottish Government website some months ahead of publication. If there are any changes to the pre-announced date, this is announced alongside the reasons.

The Scottish Health Survey publication takes place roughly 9-10 months after the end of the reference period. Time is required after data collection for data processing, analysis and quality assurance, as well as preparation and dissemination of the final results and reports.

2.4 Accessibility and Clarity

Accessibility and clarity are about:

• the presentation of statistics in a clear and understandable format
• the release of statistics in a suitable and convenient manner
• availability and accessibility on an impartial basis with
• availability and accessibility of supporting metadata and guidance

The Scottish Health Survey results are used by a range of users and multiple formats are provided to meet different needs:

• The main report, presenting headline results, trends and narrative is provided in an accessible format (HTML) on the Scottish Health Survey website. The report is also available in PDF format.
• Data tables accompanying the report are available in MS Excel format with each release to allow users to access all the data referenced in the report for each topic. From 2025 onwards, the table format will be updated to follow accessibility guidelines.
• Trends at Scotland level (by sex, age, SIMD, equivalised income, long term conditions and urban/rural classification) and data for Health Boards and Local Authority areas are also presented in a R Shiny dashboard which was introduced in 2019. This provides as much of the SHeS data as possible, with the necessary disclosure controls applied.
• Aggregate SHeS data is also published on the open data platform in CSV format which allows users to access directly for their own purposes.
• Pseudonymised individual level data is also available to approved researchers on the UK Data Service for research projects.

Methodological information, specific quality issues and definitions are included in the technical report that is published alongside the main report every year. Metadata appropriate for users to consider comparability of statistics over time and any other caveat is included with all data releases, for example as notes on tables. This metadata is reviewed and updated as necessary with each publication.

We aim to use modern and accessible means of dissemination and communication of statistics and are committed to continual improvement. More information about general accessibility is available at: Accessibility - gov.scot (www.gov.scot).

2.5 Coherence and Comparability

Coherence is the degree to which data that are derived from different sources or methods, but refer to the same topic, are similar. Comparability covers how consistent these statistics are over time and domain – for example how comparable they are with those of other regions and countries.

The Health Survey for England (HSE), the Health Survey Northern Ireland and the National Survey for Wales publish similar results for some indicators:

Health Survey for England - NHS England Digital

Health Survey Northern Ireland | Northern Ireland Statistics and Research Agency

National Survey for Wales | GOV.WALES

The surveys are conducted separately and have different sampling methodologies, so data across the surveys are only partially comparable.

Guidance on the comparability of statistics across the UK is included in the introductory section of individual chapters in the main report. SHeS data presented as a timeseries is considered to be consistent enough over time to make reasonable comparisons apart from 2020. Fieldwork for SHeS 2020 was suspended in March 2020 and, whilst data for some of the key measures was collected via a telephone survey in August and September 2020, due to the different methodology, these results were published as experimental statistics and have not been included in time series analysis presented in the main report. Whilst the 2021 survey included most of the usual SHeS questions and key indicators, interviews were still conducted by telephone and this, along with a different approach to sampling, is likely to have impacted the responses received and thus comparability with the SHeS data from other survey years. The 2022 survey was a transitional year moving from pandemic approaches back to the usual SHeS methodology.

Information is available in the survey reports and documentation to ensure users are aware of these changes.  Additional commentary on impact and definitions is included in technical reports.

3. Tell us what you think

We are always interested to hear from our users about how our statistics are used, and how they can be improved.

​​​​​​3.1 Enquiries

For enquiries about this publication please contact:

Julie Landsberg
Health and Social Care Analysis

E-mail: scottishhealthsurvey@gov.scot

For general enquiries about Scottish Government statistics please contact:

Office of the Chief Statistician

e-mail: statistics.enquiries@gov.scot

More information about Scottish Government statistics is available on the Scottish Government website.

3.2 Join our mailing list

If you would like to receive notifications about statistical publications, or find out about consultations on our statistics please join the ScotStat mailing list.

3.3 Future publications

Details of future publications can be found on our forthcoming publications page.