Experimental Statistics: data being developed
The Scottish Government will release its statistics for evaluation as 'Experimental Statistics: data being developed'. We consider that this provides users with a clearer definition of experimental statistics, and informs users that they are still under review.
The statistics should be supported by a summary of the particular reasons why they are being released as experimental – for example, about changes to methods or coverage from existing series, or if they are new – what quality issues might affect the use made of them and how the statistics should be interpreted.
What are ‘Experimental Statistics’?
Experimental Statistics are a type of official statistic that are undergoing development. They are defined in the Code of Practice for Official Statistics as: 'new official statistics undergoing evaluation that are published in order to involve users and stakeholders in their development as a means to build in quality at an early stage'.
Principle 4, Practice 5 of the Code instructs producers to release experimental statistics to achieve continuous improvement in statistical processes; Protocol 1, Practice 5 states that producers should 'involve users in the evaluation of experimental statistics'. There is an emphasis across the Government Statistical Service (GSS) to consult users during the review of statistics, and to make experimental series available during this period to assist in the quality assurance, development and familiarisation of the statistics.
Our policy on dissemination is that all experimental statistics should be clearly labelled as ‘Experimental Statistics: data being developed’ on the individual website pages (HTML), and any associated downloadable formats such as PDF and spreadsheets.
The Scottish Government will pre-announce the publication of its experimental statistics as far in advance as possible. However, due to the changing nature of these statistics, it may not always be possible to provide a release date twelve months in advance; this is mainly to ensure that the statistics are released as soon as is deemed appropriate. The Scottish Government will always seek to minimise any changes to the announced publication date of the statistics and will keep users informed throughout, in accordance with the Code of Practice for Official Statistics.
The application of the ‘Experimental Statistics: data being developed’ label is at the discretion of the responsible statistician. Defining what is experimental is largely a matter of judgement, but will typically arise when statistics are new and still subject to testing in terms of their ability to meet users’ needs.
The statistics in question might cover one or more of the areas outlined above, and there are a number of aims to labelling statistics as experimental; these include:
- Consultation: the Scottish Government would like to get informed feedback from users and potential users of the statistics;
- Familiarisation: where the experimental series are alternative versions of the format and content of existing series, and the Scottish Government may wish to help users become accustomed to new presentations or methods;
- Use: experimental series can provide useful information for users as long as their nature and limitations are well explained and understood
When do ‘experimental statistics’ stop being experimental?
Again, the removal of the Experimental classification is likely to be a matter of judgement, but will take into account factors such as:
- When it is judged that the statistical methods are relatively stable
- When coverage reaches a good level
- When user feedback indicates that these statistics are useful and credible
- When the defined development phase has ended
- When it is judged that the statistics meet the quality standards of National Statistics, following assessment by the UK Statistics Authority
The following text should be included alongside the statistics, to aid users’ understanding:
Experimental Statistics: data being developed
These statistics are currently being developed and have been published to involve users and stakeholders in their development, and to build in quality and understanding at an early stage.
Compliance with the Code of Practice for Official Statistics
The Code promotes and supports the release of experimental statistics to involve users in their development at an early stage; however, it is likely that the statistics will not be fully compliant in all areas due to their nature as ‘data being developed’.
The Scottish Government releases experimental statistics to engage with users and understand their needs as part of developing Principle 1 of the Code. The statistics may also be released to help develop methods and improvements in quality under Principle 4 of the Code, and it is important that these developments are fully discussed alongside the statistics. The statistics should always be supported by appropriate guidance and commentary to inform users about their strengths and weaknesses, particularly in relation to use under Principle 8 of the Code.
It is also important to note that there are a number of practices where full compliance will be expected for all experimental statistics released, these include:
Principle 2 – Impartiality and Objectivity;
Principle 3 – Integrity;
Principle 5 – Confidentiality;
Principle 6 – Proportionate Burden;
Principle 7 – Resources, and;
Protocol 3 – The use of administrative sources.