We are testing a new beta website for gov.scot go to new site

Quality Assurance Procedures

Introduction

Quality Assurance is a fundamental activity of Statisticians in the Office of the Chief Economic Adviser (OCEA). The production of high quality statistics depends on having sound quality assurance procedures in place at all stages of the production process, from data capture or receipt, to the preparation of statistics for users.

Clear statements about the requirement on those producing official statistics are available on the UK Statistics Authority website:

Code of Practice for Official Statistics

http://www.statisticsauthority.gov.uk/assessment/code-of-practice/index.html

National Statistician's Guidance on Quality

http://www.statisticsauthority.gov.uk/national-statistician/guidance/index.html

Furthermore, more detailed guidance is available on the IMF web pages. See 3.5 - Data Quality Assurance Framework - The ESS Handbook for Quality Reports.

http://www1.unece.org/stat/platform/display/metis/Part+B+-+Metadata+Concepts%2C+Standards%2C+Models+and+Registries

The ESS Handbook for Quality Reports provides much more detailed guidelines and examples of quality reporting practices. A key objective is to promote harmonised quality reporting across statistical processes and across Member States and hence to facilitate cross-comparisons of processes and outputs.

Statisticians in OCEA attempt to work to the high professional standards set out in these documents, with particular emphasis on the six dimensions of the ESS Quality Framework: relevance, accuracy, timeliness and punctuality, accessibility and clarity, comparability, and coherence.

.

Overview of Quality Assurance procedures in OCEA

Relevance

Relevance is the degree to which statistical outputs meet current and potential users needs. It depends on whether all the statistics that are needed are produced and the extent to which concepts used (definitions, classifications etc.,) reflect user needs.

The seven statistical outputs produced by OCEA-Macroeconomic Statisticians (Government Expenditure and Revenue Scotland, Input-Output Tables and Multipliers, Gross Domestic Product, Index of Manufactured Exports, Global Connections Survey, Retail Sales Index for Scotland, and the Scottish National Account Project) are all highly relevant to the Scottish Government who need to understand the economic and fiscal position of the country in order to govern properly.

These seven outputs are all linked in many ways. The most essential output - the short term GDP measure - is dependent on good quality weights from the Input Output system and Retails Sales information from RSIS. The Input-Output system uses data from the GCS and GERS, which is part of the SNAP system. All products, as they are related to the measurement of economic growth, are therefore relevant to the Scottish Government's main purpose, to focus the Government and public services on creating a more successful country, with opportunities for all of Scotland to flourish, through increasing sustainable growth.

Meeting user needs is a key element in the annual planning process. Registered users are invited to comment on the yearly plans, but in practice, OCEA receive little direct feedback. The Scottish Economic Statistics Consultants Group, a Committee under the ScotStat umbrella, meets to discuss planning arrangements for the production of economic statistics in Scotland and offers advice on methodological issues. The membership comes from academia, the business sector, central and local government and also includes experts from the Office for National Statistics (on an ad-hoc basis). The Group represents key users of economic statistics in Scotland and its members generally have a very high media profile. Further details of the consultation and planning process may be found using the following link:

http://www.scotland.gov.uk/Topics/Statistics/Browse/Economy/ScotStat

.

Accuracy

The accuracy of statistical outputs in the general statistical sense is the degree of closeness of estimates to the true value.

Sampling Errors

Due to the numerous data sources used in the production of economic statistics for Scotland and the complexities in the range of methodologies adopted, it is impossible to produce reliable estimates of the standard errors of published estimates. However, OCEA produce revisions tables in all its main outputs to highlight where and why estimates have been revised.

For most of the OCEA products (GERS, I-O, GDP, IME, RSIS and SNAP) the data are collected by other government departments (mainly ONS, HMT and HMRC). Where the data are collected for UK purposes, but the Scottish extract is not sufficient for similar use in Scotland, top-up surveys are organised. The Scottish Government currently pays for survey boosts of the Monthly Business Inquires, the Business Register Employment Survey, the Annual Business Inquiry and the Labour Force Survey.

For the GCS, where most data are collected in a Scottish Government survey, probability sampling is used, and the sample is stratified in favour of larger companies and known exporters.

For all data processing outliers are investigated and removed where appropriate.

Coverage Errors

For all OCEAs statistical outputs where sample information is weighted to the total population, the Interdepartmental Business Register (IDBR) is used as a sampling frame.

OCEA currently receive quarterly excerpts of this relating to all local units operating in Scotland. As this is a dynamic, administrative source, it is not always accurate and up to date. OCEA statisticians make adjustments to the register for known inaccuracies.

More work should be carried out to establish a reliable business register for Scotland which can be adjusted back through time to allow better time-series analysis of weighted estimates on a comparable basis through time.

Measurement Errors

OCEA Statisticians carry out sense checks on all data collected locally and via other organisations. Measurement errors are routinely identified and appropriate action taken - follow up, deletion or imputation.

Non-response Errors

This is an issue in particular for the GCS. The questionnaires ask for details of exports of goods and services from Scotland to the Rest of the World and to the Rest of the UK. Many companies, especially large national and multinational companies, often have difficulty in isolation Scottish activities from UK-wide activities. Many companies supply a best guess where appropriate.

Processing Errors

These are rare. If the error results in a major change to a published figure, an erratum would be issues and the figures re-published. Users would be informed and a statement place on the website. This has not happened to date.

For minor errors, these would be routinely corrected at the next publication for regular statistics and details of the revisions would be explained, where appropriate, in the accompanying Statistics Publications Notice.

Modelling Assumptions and Associated Errors

OCEA adopt different models relating to the Scottish economy where full ranges of data are not available, or where data come from a variety of data sources, and these may be of varying quality.

The Input-Output Tables and Multipliers is a good example where significant value is added by combining supply and demand estimates, together with estimates of flows of goods and services between industries and final demand sectors, to form a more holistic analysis of the economy in Scotland for a particular year. This forms the basis of the multiplier analysis and moreover has been extended to form a carbon Assessment model.

Models, by their nature, involve some subjective judgement. Where possible, OCEA will explain to users what assumptions have been made and how these assumptions might impact on the analysis or end product.

The GERS methodology uses some modelling work in apportioning UK expenditure and revenues to Scotland. The detail underpinning these decisions is fully documented on the web and in the publication. A sensitivity analysis is also provided.

Seasonal Adjustment

For GDP, IME, RSIS and SNAP, outputs are produced on a seasonally adjusted basis. Component series are tested for seasonality. The X-12-ARIMA technique is used where appropriate to remove regular seasonal peaks and troughs so that the underlying trends and other features of the data are easier to identify. Further information about the seasonal adjustment of the GDP data can be found using the following link:

http://www.scotland.gov.uk/Topics/Statistics/Browse/Economy/papers/SES2006A3

Imputation

OCEA Statisticians often made professional decisions involving imputation, especially where missing values occur in samples which are weighted to form time series estimates.

Mistakes

Mistakes are rare, but when they occur, measures are taken to avoid similar mistakes in the future. These are more likely to occur when statistics are produced in a hurry or on an ad-hoc basis. Regular peer review of the main outputs limits the chance of mistakes of any significance being made.

Revisions

Planned revisions in OCEA follow standard, well-established and transparent procedures. All statistical outputs contain table of revisions to previously published figures, and the reason for the revisions is published in the Statistics Publication Notice.

OCEA have a published Revisions Policy, which is available using the following link:

http://www.scotland.gov.uk/Topics/Statistics/Browse/Economy/ScotStat/revisionspolicy

.

Timeliness and Punctuality

The Timeliness of statistical outputs is the length of time between the event or phenomenon they describe and their availability.

Punctuality is the time lag between the release date of data and the target date on which they were scheduled for release as announced in an official release calendar, laid down by the Regulations or previously agreed among partners.

OCEA publish outputs when they are ready. As most of the data used in the production of economic statistics are received from other government departments (Mainly ONS, HM Treasury and HMRC) there is often limited scope to produce outputs earlier.

Notable exceptions are: publication of the quarterly GDP estimates has been brought forward by a week; and, following the GERS review in 2008, the timetable of the annual GERS report was brought forward six months from December to June.

OCEA publish a timetable for future publications for the year ahead. This is available using the following link:

http://www.scotland.gov.uk/Topics/Statistics/Browse/Economy/ScotStat/ForthcomingPublications

.

Accessibility and Clarity

Accessibility and Clarity refer to the simplicity and ease which users can access statistics, with the appropriate supporting information and assistance.

Publications are available on line, and follow the standard accessibility guidelines. Publications are presented in a user friendly manner e.g. tables are available in spreadsheet format. OCEA carry out numerous bespoke analyses for users (especially in the academic community) where users need statistics below publication level.

Coherence

The Coherence of two or more statistical outputs refers to the degree to which the statistical processes by which they were generated used the same concepts - classifications, definitions, and target populations - and harmonised methods. Coherent statistical outputs have the potential to be validly combined and used jointly. Examples of joint use are where the statistical outputs refer to the same population, reference period and region but comprise different sets of data items (say, employment data and production data) but for different reference periods, regions or other domains.

Considerable effort is now made to ensure internal comparability with our own figures, especially on a time-series basis. For example, until 2009, the Input-Output tables for Scotland were published on an annual basis but no revisions were made to earlier years. In the latest publication, a time series of I-O tables was published from 1998 to 2004 on a consistent basis, allowing for methodology and definitional changes over time (e.g. the re-calculation of FISIM to be consistent with the UK National Accounts). The next publication, updating the tables to 2007, will also incorporate revisions back to 1998.

The next publication of GDP for 2010 Q2, due for release on 20 October, will use the 1998-2007 I-O tables for weights and constraining variables. The quarterly SNAP estimates will also be constrained to the annual I-O analysis for 1998 to 2007.

Comparability

Comparability is a special case of coherence and refers to the latter example above where the statistical outputs refer to the same data items and the aim of combining them is to make comparisons over time, or across regions, or across other domains.

OCEA have taken care to ensure that economic statistics produced for Scotland are presented on the same basis, where relevant, as those produced for the UK as a whole.

The same standards and definitions e.g. the Standard Industrial Classifications are used to ensure comparability.

It may not always be easy to ensure comparability on the same time-scale. For example, the ONS are moving from SIC 2003 to SIC 2007 in a piecemeal fashion. Whilst we will receive data on the new classification, the control totals we use will not be available until the Regional GVA estimates are compiled. This is the last product ONS will be converting.