Joined up data for better decisions: A strategy for improving data access and analysis

Section 2: Background

In early 2012 we published the consultation document, A Scotland-wide Data Linkage Framework for Statistics and Research, which set out the aims, benefits and challenges to data linkage alongside a draft set of 'Guiding Principles' and suggested functions and objectives for a Data Sharing and Linking Service (previously named the National Data Linkage Centre) and a Privacy Advisory Committee.

In parallel to the written consultation process we had a number of meetings and discussions with stakeholders, sought advice from international experts, commissioned a series of public deliberative events and engaged with discussions of the Administrative Data Task Force.

Benefits of Data Linkage

In the consultation paper we asked for views on the benefits of data linkage and the barriers to more data linkage. Most of the responses concurred with the summary given in the consultation paper or added further detail or examples. Furthermore, all participants involved in the deliberative research events recognised the potential benefits of data linkage.

It is beyond the scope of this strategy document to restate or provide a comprehensive overview of the benefits of data linkage. Rather, we recognise that the benefits of data linkage are many and can, broadly speaking, be summarised as:

• Speeding up cycles of improvement through the delivery of a higher quality cross-sectoral evidence base to inform public policy and strategic planning, spending and delivery decisions.
• Maximising the value of existing data to develop efficient and reliable methods of producing statistics, including better statistics at sub-national level.
• Allowing relatively low cost longitudinal research to be conducted both retrospectively and prospectively.
• Increasing the capacity to robustly evaluate programmes, by providing the potential to answer far more sophisticated research questions than otherwise possible.
• Improving the quality and consistency of data, through general feed-back loops following linkage activities.

Barriers to Data Linkage

In terms of issues relating to incomplete data, or data that cannot be linked; capacity for secure exchange and access to data; and the capacity of public sector organisations to analyse and make use of linked data, consultation responses generally concurred with the summary given in the consultation paper or added further detail or examples of these challenges.

It is clear that in order to make better use of administrative data it will be necessary in many areas to first improve the quality and consistency of the administrative data systems that exist across the public sector in Scotland. Where this has been done, for example in relation to school and pupil data through the ScotXed Programme, substantial benefits to all are evident.

Likewise, building on and expanding from the existing high quality facilities for secure exchange and data access in Scotland, and further increasing the strong analytical skills and expertise to analyse and make use of linked data, are important components.

In the consultation paper we also suggested "Uncertainty about the legalities and public acceptability of data linkage". Responses provided examples and evidence of this, and it is clear that there is considerable variation in the interpretation of the legal and regulatory environment, and that data custodians are often unsure whether they can legally and appropriately make data available for linkages.

In addition though, the consultation responses, along side the broader range of views we have heard, have been helpful in beginning to unpack a range of challenges, ethical considerations and risks related to data linkage.

Public Interest and Public Acceptability

It is necessary that data linkage activity is conducted in the public interest, and that it is acceptable to the public. In order to deliver a data linkage framework that encourages and enables both of these things, we will engender an approach of transparency and public engagement in all aspects of the framework.

The consultation responses and deliberative events with the public showed that the scope for commercial gain arising from data linkage activity is a cause for concern amongst some.

This strategy is for the public, private and voluntary sectors in Scotland. The Guiding Principles are clear that public benefit should be at the forefront of data controllers' deliberations about whether or not to permit sharing and linkage of data.

In further developing the framework, to inform the deliberations and guidance of the Privacy Advisory Committee and the policies and procedures of the Data Linking Service, work with members of the public and other stakeholders to fully explore these issues will continue. We will facilitate a dialogue between members of the public and both public and private sectors about the appropriateness, the concerns, the benefits and the risks involved when the private sector is involved in the use and analysis of data collected and held by public bodies.

The Scottish Government does not intend to "sell" linked data or to encourage other public bodies to do so.

Privacy and Consent

Protecting identity and privacy in line with legal requirements (including but not limited to the Human Rights Act and the Data Protection Act) underpins this Framework and the Guiding Principles.

Whenever data are accessed, analysed, shared and linked there is some risk to privacy, however small that may be in most cases. The main risks are data loss (through accident or malice), or statistical disclosure (the potential to identify an individual within a dataset by their unique or rare combination of characteristics). It is therefore vital that in encouraging more use of data, and data linkage in particular, these risks need to be at the forefront of considerations and their mitigation must be a high priority.

The risks to privacy must be carefully considered and managed both where data subjects have given consent for a specific data usage and where alternative governance mechanisms (e.g. access granted by the data controller) are used. Consent of data subjects for any specific usage is the 'gold standard' governance mechanism, and is a major concern of many members of the public, data controllers and researchers. Consent is not, however, a legal requirement for data sharing or data linkage to occur, and there are other ways to protect the rights of data subjects.

The European Commission is proposing new regulation to protect individuals relating to the processing of personal data and the free movement of such data. The ability to collect, analyse and transfer personal data is critical to data linkage for statistics and research. The EU Regulation is still under negotiation and the Scottish Government is liaising with the Ministry of Justice as the negotiations proceed. The regulations will be carefully examined and appropriately applied.

Archiving and Transparency

Several consultation responses raised concern about the potential for the Data Linkage Service to amass data about individuals. Likewise concern was raised that any organisation, now or in the future, may hold excessive quantities of linked data. The principles relating to separation of functions will address this by encouraging separation of the indexing, linking and analysis functions, as well as the time-limited storage of linked data. Decisions on handling, storage and linkage of data will continue to sit within the established concepts of Public Interest, Governance and Public Transparency. This will place a clear responsibility on individuals and organisations engaged in this work to do so in an open manner which is consistent with the principles and responsive to evolving concerns of the public.

There is no intention to create a single archive, or store, of excessive quantities of linked data, or a single organisation that controls all data linkage in Scotland.

By establishing facilities whereby cross-sectoral linkage can be achieved but personal identifiers (e.g. names, address, or persistent identifiers such as Community Health Index number) are transferred and held separately from any 'characteristic' data, the Data Linkage Service will increase the capacity within Scotland for optimal methods of data linkage.

By establishing a set of principles in which the importance of transparency is central, and a Privacy Advisory Committee with representation from members of the public, we aim to develop a system of checks and balances that prevents any one organisation from gaining too much personal data.