Independent review – Independent advisory group on new and emerging technologies in policing: final report

3. Research evidence and emerging technologies in policing

This chapter focuses on the role of research evidence in relation to the adoption and implementation of emerging technologies in policing. This chapter draws largely from chapters 2 (and 4) of the workstream 2 report (Buchanan et al., 2023), with some content derived from the Stirling Research Commission (Connon et al., 2023) and other workstream reports, e.g. Campbell et al. (2023). Whilst the opinions and experiences of the public and stakeholders may also be of interest these are considered elsewhere, e.g. Chapter 6, whereas this chapter focuses on research evidence. It covers the use of research evidence in supporting decision making and evaluation, innovation, consultation and engagement and provides suggestions for further research.

Existing approaches to use of research evidence:

It is crucial to understand the full impact of emerging technology to enable appropriate decisions to be made before its implementation in Scottish policing.

The first section of the report draws on chapter 2 of the workstream 2 report (Buchanan et al., 2023) and considers existing approaches taken by Police Scotland to the use of research evidence for the consideration, adoption and implementation of emerging technologies. It provides an overview of the governance that is followed during the consideration of new and emerging technology prior to any subsequent adoption, following the pathway of technology adoption from initial idea/concept to Business-as-usual adoption of the technology.

Governance for technology adoption pathways:

Police Scotland and the Scottish Police Authority (SPA) have a governance process in place which has been maturing to maximise appropriate governance, oversight and scrutiny (see Figure 2.1 Buchanan et al. 2023: 5). This includes the recent introduction of a joint Memorandum of Understanding (see Chapter 8 below) between the two organisations, which aims to ensure early visibility and oversight of any new strategy, policy or practice under consideration by Police Scotland. Neither organisation has a specific board for consideration of new and emerging technology or research evidence, but the most likely governance route for emerging technologies would be through Police Scotland's Change Board or Demand, Design and Resources Board, into the Strategic Leadership Board and then on to the SPA Resources Committee and SPA Board (see Figure 2.2, Buchanan et al., 2023: 7).

SPA's function in oversight of change enables questioning of new initiatives. It is vital to understand ethical aspects of technologies and their potential use in policing prior to adoption (i.e. at the stage of considering need for change). It is also essential to understand the societal and cultural aspects of the need for change, and of the technology and its deployment and to consider the required socio-technical innovation. It is acknowledged that all these aspects need to be analysed and evaluated together, particularly since the operational technologies deployed e.g. with the aim of identifying a suspect population, can aggravate inequality and distrust by drawing from in-built bias in data, technologies, practices and institutions. Unintended bias or inequality would be one of the aspects examined through the proposed sixth ethics and human rights case process (see Chapter 8). Furthermore, it is important to note that where a proposal to process personal data presents a likely high risk to the rights and freedoms to individuals there is an existing legal obligation to assess the risks to the rights and freedoms of data subjects and the measures envisaged to address those risk in a Data Protection Impact Assessment (DPIA) under data protection law.

Stage 1 is when the need for change is identified and considered. As a project can be initiated from any business area (Operational or Corporate). If the latter it will be approved by a Director, but if Operational Stage 2 is likely to involve the preparation of a brief for one of the Local Management Boards (Local Policing, Crime and Operational Support or DCC Designate). If a change is approved in principle by one of the boards, the lead will be asked to prepare a Potential Project Assessment (PPA) to present to the Demand Management Board where it may be deemed Business as Usual (BAU) or Project. If the latter an Initial Business Case (IBC) will be completed to bring together all of the key information needed to initiate the project. If it is of substantial size and scale a Programme Brief will also be required.

The IBC includes aspects such as the case for change, project plan, controls and communication plan and should include the benefits and dis-benefits of the proposed change. Crucially, 'benefits should be quantifiable wherever possible and be based on robust research and evidence' (Buchanan et al., 2023: 8). However, as it stands there is no mandated requirement for scientific research to be included as part of the IBC. Nonetheless, this seems to be changing as the SPA have stated that having dip sampled recent IBCs in 2022 it was 'common for there to be a strong reference made to existing scientific literature or evidence (with a particular focus on evidencing the benefits of proposed technology)'. In my view as Chair, some basic level of information about the research evidence base should be required for all business cases and it is important to ensure that the assessment be balanced and include evidence about 'dis-benefits' as well. This is not to say that new empirical research needs to be conducted (which can take significant time), but that the existing evidence base should be taken into account (see key consideration 3.1 below).

Stage 3 involves consideration of the IBC by Change Board of the benefits, impact, timescales, cost or resource implications etc. If approved a Full Business Case (FBC) will be prepared and considered by the relevant Programme Board and Change Board using the Investment Governance Framework (see Buchanan et al., 2023 Appendix A).

Whilst the business case template used by Police Scotland lends itself to the inclusion of research and evidence, particularly in the 'case for change' section, as Chair I note that the business case template does not explicitly mention research and evidence. The description includes a 'qualitative assessment of the status quo and benefits the change/proposal will bring' and encourages the inclusion of 'quantitative data and scientific standards' (including for use as baseline in any potential future evaluation). Detail of the current business case structure may be found in Table 2.1 (Buchanan et al., 2023: 10). Indeed, the dip sampling of business cases found that although it was common for there to be reference made to existing literature or evidence of benefits there was a lack of scientific evidence and research provided to support the case for change.

Therefore, an opportunity exists to strengthen the business case process by mandating for the inclusion of evidence to substantiate the case for change. In my view as Chair this should include a range of the highest quality available research evidence, using the most appropriate methods to answer the question at hand. Wherever possible this should include any available quantitative data, although it its acknowledged that qualitative data may be most appropriate in certain circumstances. Crucially, to ensure balance, evidence must also be provided on any identified drawbacks or pitfalls associated with the implementation of the change. The level of risk should determine the level of evidence gathering required. Furthermore, consideration should be given as to whether a pilot should be conducted prior to roll-out of certain technological deployments in policing.

If the business case is approved the project will progress straight to Stage 5 (implementation) if it does not impact people's working conditions, or to Stage 4 (consultation) if it does. All Organisational Change proposals are presented to the Joint National Consultative Committee (JNCC), which has representation from staff associations, prior to any consultation with staff. Technology projects will not routinely go through this step, but some major technology projects may have organisational implications and therefore go through Stage 4. Following JNCC, consultations may begin, often in groups (before individual meetings with who are impacted) and provide the opportunity for staff to make any redundancy mitigation counter proposals. Consideration should be given to the fact that diversity and representation in decision making structures may be particularly important in relation to technology in policing, given the impact on protected characteristics such as race.

Once a project moves to implementation phase (Stage 5) consideration should be given to introducing controlled pilots and an evaluation process for the impact of new technologies. A baseline measurement should be confirmed ahead of the introduction of technology and ultimately be used to assess the implementation and its impact. Evaluation is also particularly important given the requirement for ongoing review of DPIAs under data protection law, as the actual risks and harms that may arise as a result of the deployment of a new technology will only become evident after deployment.

Policing research partnerships:

Policing Research Partnerships can play an important role in facilitating research, dissemination, knowledge exchange and impact. This can assist with reviewing the existing evidence base, shaping decision making and supporting evidence-based policing. It can also help with embedding evaluation (process or outcome evaluation) of technological adoption, providing a reflection of lessons learned in implementation, and assessment of benefits realised and any unintended consequences or harms.

SIPR is one such Policing Research Partnership, along with many others in the UK including the N8 Policing Research Partnership and other police-academic collaborations (e.g. the Open University) and university centres (e.g. Northumbria, UCL). Established in 2007 and supported during its initial five year phase by investment from the Scottish Funding Council and the Association of Chief Police Officers in Scotland, the Scottish Institute for Policing Research is a collaboration between Police Scotland, the SPA, and 14 Scottish universities. SIPR's mission is to support independent, multi-disciplinary policing research to enable evidence informed policy and practice.

The work of SIPR is advisory in nature, not decision making, but the intelligence and evidence gathered and generated is in a strong position to support several aspects of decision making, for example for Change Board, Strategic Leadership Board, SPA Board. SIPR may also be asked by Police Scotland and the SPA to assist in commissioning academic researchers to assist with reviewing the evidence base or considering the implications of the adoption of new and emerging technologies before implementation (e.g. Body Worn Video, BWV) or to evaluate the implementation of various technologies or approaches (e.g. Benefits of implementation of mobile devices with frontline officers in Police Scotland 2019-2020). There is no one formal route for this work to be commissioned but research typically transpires from discussions at committees (e.g SIPR Executive Committee), boards or with senior leaders. Consideration should be given to determining when an evaluation is needed, how it can be implemented ahead of the change happening and seeking SIPR's support in commissioning the evaluation prior to the technology coming into effect.

Other insights:

Police Scotland use the Citizen Space website to host consultations and engagement in order to gain an understanding of public opinion on a variety of issues. For example, it was recently used to gain understanding of public opinion on the use of BWV. A variety of other engagement approaches e.g. focus groups are also used (see Chapter 6).

Comparison with other comparable police forces may also provide valuable insights and benchmarking for decision making and evaluation. However, dip sampled business cases showed little comparison to how comparable new and emerging technologies had been implemented in similar police forces. This could be valuable evidence to include as part of the proposal for change and may also provide insights of lessons learned to inform implementation.

In conclusion, at present, although Police Scotland and the SPA have a governance process which may be used to oversee the implementation of emerging technology, there is no formalised inclusion of an assessment of the research evidence base in this process. Whilst reviews of the research evidence base may be undertaken in certain cases to inform the decision making or implementation of new technology (for example, BWV), there is no agreed process or criteria in place to formally require an assessment of the evidence base in the business case. This should therefore be included as part of the change process to enable the appropriate governance mechanisms to make informed decisions. Furthermore, it is crucial to consider the implications of new technology on the communities it impacts and ensure evaluation is in place where necessary. Engaging with external organisations and individuals from an early stage provides an opportunity to mitigate barriers to successful implementation early in the process, e.g. ICO guidance is that the DPIA process should involve consultation. Furthermore, in my view as Chair the level of evidence gathering required to inform decision making and whether new research or evaluation is needed or not will vary depending on the extant knowledge base and level of benefit or potential risk posed by the proposed technological adoption in policing in Scotland.

Research evidence & innovation:

Other chapters of the workstream 2 report (Buchanan et al., 2023) on technological innovation also have some useful insights of relevance to research evidence. In chapter 3 for example there is an acknowledgement that forming lasting partnership entities assists with better translating research into practice and with research and development investment for technology innovation. In addition, there is an acknowledgement that short-termism and a stop-and-start approach necessitates extra effort and funding over time to continue research and development. TechUK's contribution states that organisations who adopt a collaborative, consortia-based model may achieve better results than traditional 'ecosystem' approaches which may stifle agility, innovation and genuine engagement from experts. Although there are benefits to a consortia-based model the IAG acknowledges that procurement implications and conflicts of interest would need to be considered and mitigations put in place.

In chapter 4 techUK asked their members about evidence-based decision making and how the tech industry can engage with academia when developing evidence-based pilots. Their contribution acknowledges that evidence-based decision making, and policing research partnerships can assist police forces with connections, with achieving their intended outcomes, and result in more well researched policies and practices. When making informed decisions about the adoption of emerging technology in police organisations this may be done using a combination of best practice research evidence, industry knowledge and experience (by both policing and technology partners) and the experiences of the victims of crime (and others impacted by the criminal justice system).

There are a number of areas where academia, and the research community as a whole, can bring fresh insight into the process:

• 'Police practices should be based on scientific evidence about what works best and hence it is important that for any evidence-based pilot developing, industry must engage academia. Academia are keen to support the development, testing and promotion of innovative practice to help build the evidence-based solution and understand what would work best.' (Buchanan et al., 2023: 30).
• Partnerships, mentoring schemes, apprenticeships and test panels/groups which can road test new technologies and share ideas/challenges to ensure that new technologies are approached from an outcomes perspective.
• Research and evaluation may be used to inform or assess the adoption of technologies. For example, in 2019 SIPR supported Police Scotland in the run up to their commissioning an evaluation of the Digitally Enabled Policing Programme (DEPP), the 'Police Scotland Mobile Working Project' (MWP), which reported in 2020. This project equipped operational officers with a digital mobile policing solution to replace the traditional paper notebook and to provide remote, live access to key policing information systems. This is cited as a good example of how academia can assist and support the review of policing projects, whilst remaining independent and transparent. However, as Chair I would note that in order to safeguard the quality and independence of evaluation research it is important to ensure that where feasible a Research Advisory Group is established and ideally publications are peer reviewed. There are benefits to commissioning evaluations through a third party like SIPR, rather than by Police Scotland directly.
• Another example provided involves the The Digital First and GDS Service standards which emphasise (1) using evidence to quickly demonstrate that there is a good understanding of the problem to be solved before making substantial commitments (2) using research with real users and real data to quickly establish whether a worthwhile solution can be delivered before undertaking significant development and (3) focussing on rapid, iterative prototyping against agreed KPIs to drive effective design.

Evidence-based consultation & public engagement:

As outlined in the workstream 3 report (Campbell et al., 2023) research evidence also has an important role to play in informing public engagement and consultation. As Campbell et al. (2023) states, supporting evidence and materials was one of the areas identified for further consideration based on a review of the approach to public engagement on BWV, undertaken in order understand key lessons for future engagement.

In the initial engagement on BWV, Police Scotland shared an evidence base of reports. Concerns were raised that this was not fully representative of a wider range of literature and views on the use of BWV. Indeed, I would like to note as Chair that in my role as SIPR Director I raised concerns with Police Scotland that the short summary leading into the engagement exercise stated that evidence showed that BWV achieves certain outcomes. The wording was changed to remove the reference to evidence when I pointed out that the statements did not align with the findings of a systematic review of the highest quality research evidence (Lum et al. 2019).

Campbell et al. (2023: 21-22) state that 'the design phase of future engagement approaches will consider evidence with key stakeholders and seek an independent view of the robustness of the materials being provided to the public, guided by senior responsible officers and Police Scotland's in-house Academic Research team, whilst engaging with research partners. This will ensure, as far as possible, an appropriate range of information and views are shared and that Police Scotland has sought independent consideration of the materials being published and presented.'

The workstream 3 report also concludes that consideration could be given to adding to the evidence base of materials as suggestions are received from the public and groups taking part in the engagement/consultation. It is important that the materials being shared as part of the evidence base are fully accessible for all. Also, a user-centred approach must be taken to consider how someone gets to the online survey or consultation page and through which 'journey' /starting point, in order to ensure that there are multiple opportunities to engage with the evidence available to inform their views in advance of taking part. Beyond online consultations, various other engagement approaches appropriate to meeting the diverse needs of communities are used to encourage meaningful participation, see Chapter 6.

Finally, evidence and materials that support the engagement should represent a range of views to enable an open and transparent dialogue. Evidence and materials must be accessible and inclusive for a range of needs; to ensure that everyone is able to meaningfully participate in understanding the evidence and materials. Good practice would suggest producing 'easy read' versions of materials, and working with key partners who are experts on the subject to understand any tensions with any evidence being provided to inform decision-making. It is worth noting that the evidence base should also be referred to in a DPIA and it is good practice to publish DPIAs. Furthermore, equality legislation is even more prescriptive – The Equality Act 2010 (Specific Duties) (Scotland) Regulations 2012 describe how relevant evidence relating to persons who share a relevant protected characteristic must be considered, and that results (of the impact assessment) must be published in a manner that is accessible.

Furthermore, in my view as Chair in addition to making a range of materials available it would be important to develop (with input from independent stakeholders) a clear and succinct public facing summary of the existing research evidence (acknowledging any limitations and gaps in knowledge) to be shared during public engagement and consultation exercises and in the summary of the ethics and human rights case. Note the Scottish Biometrics Commission's process for introducing a new biometric technology or a new application of an existing biometric technology for policing and criminal justice purposes in Scotland (SBC CoP Appendix D) will need to be complied with for biometrics.

Suggested areas for further research:

The commissioned research report (Connan et al. 2023) makes a number of suggestions of potential areas which may warrant further research. These are listed below verbatim as they are covered in the commissioned report, but the IAG notes that there is a need to assess the strategic importance, level of risk, and potential value of the research for each of the proposed areas for further research in order to guide decision making on which areas should be prioritised for funding by various partners (Scottish Government, Police Scotland, Scottish Police Authority, the Scottish Institute for Policing Research etc.). It is also noted that in addition to guiding which areas may warrant prioritisation for funding, these suggestions will be informative to SIPR and the academic community who are keen to leverage external funding e.g. from research councils.

1. Suggestions for research relating to Electronic Database Technologies (Electronic database technologies represent one form of continually evolving technology that are used in contemporary police practice. Electronic databases store, organise and process information in a way that makes it easy to perform searches and analyses.)

a) Further research should be undertaken concerning the use of national datasets to gain a better understanding of the risks involved in the use of such technologies.

b) There is a need for greater integration between academic researchers, police, the policy community and third parties to develop and implement specific solutions for the embedding of these forms of technology in policing practice that are sensitive to the needs of all parties.

c) For more trials and assessments to be undertaken to establish best practice and decision making within a range of policing contexts in Scotland and the UK.

2. Suggestions for research relating to Biometric Identification Systems and Artificial Intelligence Technologies

a. The need for further research to be conducted to explore the benefits and limitations of the use of facial recognition technologies (of various kinds e.g. live or not) in different policing activities (namely, public order policing, crowds, and public events).

b. Development of a set of shared concepts and terminology to develop an ethics of algorithms and the building of a more rigorous evidence base for the discussion of social and ethical issues surrounding the use of AI in policing.

c. Consideration to be given to the statistical and scientific validity of proposed AI technologies and for context-specific evaluation methodologies to be applied for statistical algorithms.

d. The need to interrogate biases and limitations as to the efficiency of AI systems prior to development and use.

e. For police professionals and third parties that they work closely with (i.e. local authorities) to be involved in the design and implementation of these technologies to help promote ethical awareness and practice.

3. Suggestions for research relating to Surveillance Systems and Tracking Devices

a. If technologies are being contemplated consideration should be given to conducting research e.g. trials to explore the benefits and limitations of the use of these different forms of technology in different policing activities and contexts, e.g., in Scottish rural vs.urban contexts.

4. Suggestions for research relating to data protection, the law of evidence, and equality and human rights: applicable to all forms of emerging technologies

a. At the outset of designing, adapting, or adopting an emerging technology, consideration should be given to how that technology is to be used to ensure compliance with the law of evidence.

b. The roles and relationships under data protection law and data flows between all controllers and processors should be mapped out and understood prior to processing.

c. Further research should be undertaken to consider the legal and ethical implications for the use of emerging technologies in policing activities involving children, with a view to ensuring compliance with the United Nations Convention on the Rights of the Child.

Chapter 3 summary and conclusion

Decision making: It is crucial to have an understanding of the existing evidence base regarding the impacts of emerging technologies in order to enable appropriate decisions to be made regarding their adoption in Scottish policing. The governance process has been maturing, including the introduction of a joint Memorandum of Understanding between Police Scotland and the SPA, which aims to ensure early visibility and oversight of any new strategy, police or practice under consideration. Business cases form a key part of decision making and are expected to include an assessment of benefits of the proposed change. At present the inclusion of an assessment of the research evidence base is not explicitly required.

The research community, including policing research partnerships (e.g. the Scottish Institute for Policing Research, SIPR) can play an important role in facilitating research, assisting with reviewing the evidence base, supporting evidence-based decision making, evaluation, translating research into practice and supporting innovation. Other insights are also provided by Police Scotland, including for example the use of Citizen Space to gain an understanding of public opinion. Research evidence has an important role to play in informing public engagement and consultation. It is noted that the evidence base should also be referred to in the Data Protection Impact Assessment (DPIA).

3.1 All business cases (and hence templates) completed by policing bodies should require the inclusion of a basic assessment of the evidence base, drawing on available research (and learning from other jurisdictions) and including both benefits and dis-benefits. The level of risk (see chapter 8) should determine the level of evidence gathering required. For medium-risk projects (or projects with high value investment) an evidence review should be required. For all high-risk projects a more thorough evidence review (ideally with external input or review) should be required, or indeed in some cases the generation of further independent research may be necessary (particularly if the evidence base is lacking) to inform decision making regarding adoption.

3.2 Policing bodies should also draw together the evidence base to support consultation and public engagement work. A balanced, clear and succinct public facing summary of the existing research and other evidence (acknowledging any limitations and gaps in knowledge) should be developed (with input from independent external stakeholders, particularly for high-risk projects) in order to be shared during public engagement and consultation exercises.

Innovation: Scientific evidence about what works best should be central to innovation and shaping police practices. Using a combination of best practice research evidence, industry knowledge and experience, and taking into account the experience of members of the public (including victims of crime and others involved in the criminal justice system) is key in order to make informed decisions on the adoption of emerging technology in policing. In order to support innovation, if pilots are being developed, consideration should be given to doing this in collaboration with academia, industry and other stakeholders. If industry are developing pilots they should engage with the research community in order to build evidence-based solutions.

Evaluation: It is important to consider whether a pilot (rather than full roll-out) is the most suitable first step. Furthermore, it is crucial to consider whether an evaluation is needed, and how it can be in place ahead of changes happening. Evaluations may be focused on a pilot or full roll-out, and on either the implementation (process evaluation) or the impact of new technologies (outcome evaluation). Support in commissioning evaluations should be sought prior to technological adoption and baseline measurements must be gathered.

3.3 The decision about whether an evaluation of the impact of new technologies in policing is needed should be informed by the level of risk and the existing evidence base. For existing systems with a history of safe operation an evaluation would only be necessary if they undergo significant changes in their design or intended purpose. For high-risk projects an evaluation should be carried out and commenced at the earliest point possible prior to development, acquisition or adoption of a new tool, means or method of policing. In relation to commissioning evaluations, third party support (e.g. through SIPR) should be sought very early on and the evaluation should include baseline impact measurements. Steps should be taken to safeguard the quality and independence of the evaluation e.g. establishing a Research Advisory Group and peer reviewing of reports.

3.4 In addition to accessing learning and research on best practice in the use of emerging technologies from other jurisdictions (including from other police forces) to inform decision making (e.g. business cases and sixth case assessment) and design processes, opportunities for knowledge exchange (where possible in open fora) should continue to be maximised by policing bodies throughout implementation and ongoing review.

3.5 The existing knowledge base and suggestions for further research identified by Connon et al. (2023) should be reviewed by policing bodies and key stakeholders in order toprioritise areas for further research. An assessment should be undertaken of the level strategic importance, level of risk, and potential value of the research in order to guide decision making on which areas should be prioritised for funding by various partners (Scottish Government, Police Scotland, Scottish Police Authority, the Scottish Institute for Policing Research etc.). In addition, suggestions should be shared with SIPR and wider the academic community who are keen to undertake independent research and leverage external funding e.g. from research councils.