TEC programme data review and evaluation: options study

4.0 Implementation research

A key finding from the data review (and other evaluations) is that, although many of the technologies have established an evidence base, there is more to be done on implementation and scale up. This has also been noted in the literature ( e.g. Glasgow et al. 2013; Ossebaard and van Gemert-Pijnen 2016). Indeed, the TEC programme was designed to support implementation and we would expect it to form a central part of the ongoing research priorities. Our recommendation, therefore, is to explore the use of Implementation Science ( IS) strategies to inform programme development and future evaluation. The TEC programme has had a key role to date in supporting implementation as set out in this report. In this section, we describe the findings from our exploratory implementation research. We cover the following:

• Introduction to implementation science, including key findings
• Report on the stakeholder engagement exercise with frontline and programme staff to explore the barriers and enablers to implementation, scaling up and sustainability
• The impact of the TEC programme on implementation so far, including strategies that have been used and any gaps
• The future research agenda for TEC in this area.

4.1 What is Implementation Science?

Implementation Science ( IS) has become increasingly important to funders, researchers, programme developers and practitioners as an approach to bridging the gap between science and practice (Meyers, Durlak & Wandersman, 2012). This approach can be usefully applied to the Scottish TEC programme as it seeks to promote increased mainstreaming of technologies. The IS literature describes a broad landscape of theories, models, and frameworks that are used to examine a range of questions posed by IS practitioners (Fogarty International Center), including:

• How do you achieve more widespread use of effective practices, policies, and programs?
• What infrastructures or systems are necessary to ensure that dissemination and implementation are carried out successfully?
• How do organisations and practitioners build the capacity necessary to bring effective practices to scale community wide? (Wandersman, 2008)

The theoretical base for IS has developed from a diverse range of fields including Normalization Process Theory, Social Psychology, Systems Theory, Behavioural Science and Management. In the context of IS, implementation needs to be understood as a process that is complex and dynamic rather than as a final outcome: "It is not a single 'thing' to be accomplished but operates within an interactive system made up of "many moving parts" (May, 2013 pp 2).

Implementation in TEC is complex, dynamic and influenced by multiple factors: professionals, patients/users, and the social, economic, organisational and political context. Complexity is exacerbated by the fact that the place of delivery increasingly includes informal care and the home environment (Ossebaard and van Gemert-Pijnen 2016). Furthermore, it is the interplay between technical and social factors that produce a particular outcome (Obstfelder et al. 2007). Successful implementation and the sustainability of innovations requires the barriers and facilitators that influence that context to be well-understood (van den Wijngaart et al. 2017). In addition, the nature of the interventions requires a strong focus on rapid evidence gathering and a fleet-footed approach to implementation, which is out of synch with the timescales of traditional evaluation methodologies (Glasgow et al. 2013, see also Section 5). As Ossebaard and van Gemert-Pijnen (2016, p.415) have written, "documented drawbacks such as low acceptance, low adoption or low adherence need our attention today to make the most of eHealth's potential". In a review of the implementation of the UK's Whole System Demonstrator Project, Hendy et al. (2012) concluded that the implementation of a complex innovation such as remote care requires it to organically evolve, be responsive and adaptable but that this is not always aligned with the imperative to gather robust benefits evidence. For scaling up to be realised, this tension would have to be resolved.

IS and TEC are relatively new fields of scholarship and there is a limited, but growing, literature that brings the two together. There are several key points to note in this literature, including:

• Development, implementation and evaluation are deeply intertwined
• Stakeholder participation and co-creation is essential for effective implementation in technology-enabled care
• Digital technologies have the potential to create new procedures and infrastructures for healthcare delivery, which are not yet fully understood

4.2 Findings on barriers and enablers

The first step to identifying barriers and enablers to the adoption of technology-enabled care was to include this as a research question for the review of existing evaluations in Stage 3. The review found a considerable amount of information on barriers and enablers to implementation of telecare and HMHM. There was less research on this for video conferencing and digital platforms, reflecting the earlier stage of implementation for these technologies.

The data review was followed by two workshops to further explore these barriers and enablers as well as IS strategies. One workshop was attended by frontline delivery staff; the other by workstream leads and TEC programme staff. At the workshop, attendees were asked to work through a list of common barriers in order to identify potential solutions. Tables 1 and 2 summarise the findings from the literature, which are consistent with those identified by workshop attendees.

Across both exercises, the most commonly identified barrier was staff resistance. There may be several factors influencing staff resistance and several potential methods for resolving it. Understanding the influencing factors is crucial to ensuring that the right implementation strategy is adopted. For example, if staff are struggling with competing workloads, specialist skills can be brought in to provide additional resource and to support them to ease the transition to the new system. On the other hand, if the main issue is a fear of change, a training programme addressing this directly might work best. Other noteworthy barriers include digital skills, infrastructure, lack of specialist skills, lack of interest from senior managers and a lack of interoperability. A further barrier that emerged from the workshop discussion was the lack of appropriate data. Attendees (mainly TEC Leads) described their roles as partly a 'sales job'. They require distinct kinds of data for different audiences and spoke of the value of data on effectiveness. However, concerns were also raised about the quality of monitoring data and the weakness in some existing data collection systems was raised as an additional barrier.

Table 1: Barriers to TEC implementation from the data review

Table 2: Enablers to TEC implementation from the data review

4.3 Implementation strategies in the TEC programme

The IS literature identifies four stages of implementation:

• Exploring and preparing
• Planning and resourcing
• Implementing and operationalising
• Business as usual

At each stage, different activities are required that are facilitated by distinct conditions. For the TEC programme, assessing progress against implementation is challenging. There are distinct, but interrelated workstreams, each at different stages of the implementation cycle. For example, video conferencing is still at the planning stage, whereas telecare is much closer to business as usual. Cutting across this, the programme is dealing with a range of populations with different sets of issues, creating another layer of implementation challenges. However, our research found that despite these complexities, the challenges also overlap ( i.e. staff and users experiencing the same barriers and enablers across workstreams). This suggests that implementation, and increasingly evaluation, can be addressed programmatically.

The logic model workshops in Stage 1 show that the TEC programme is already employing many strategies that are consistent with recommendations in the IS literature. This was confirmed at the implementation workshop with workstream leads. At this workshop, participants were taken through a list of strategies to identify corresponding activities. This list of strategies was adapted from the work by Powell et al. (2015) on the Expert Recommendations for Implementing Change ( ERIC) project ^[6] .

Following the workshop, we refined and adapted the original list to fit with the Scottish TEC context. There were several steps to this. First, ERIC was an American initiative and we removed those that were culturally inappropriate ( e.g. where they referred to health insurance). We then removed a small number that were repetitive and grouped the remainder where the strategies were similar or overlapped. This left us with 15 headline strategies with several sub-strategies within each. These are listed in Appendix 5.

4.4 Reflections on the impact of the TEC programme

The implementation research provided an opportunity to consider the impact of the TEC programme as a whole. During logic model development staff were keen to explore the programmatic impact, specifically whether there was any evidence that the programme was greater than the sum of its parts. Additional impacts, above and beyond the individual workstreams, were identified in developing the logic models but there were few evaluations that addressed this question directly with the bulk of the studies looking at technology impacts.

We used the implementation research to gather further information on the impact of the TEC programme. First, as discussed above, a substantial finding from the implementation workshops is that a large amount of formal and informal implementation work is being carried out applying strategies and methods identified as best practice in the international literature. Whilst the success of these has not been evaluated, it is clear that the approach taken is broadly consistent with the prevailing evidence of how best to develop such a programme.

In addition, workshop participants were asked to complete a survey, which explored TEC programme impacts from an implementation perspective. Respondents were asked whether they agreed with the statement that the TEC programme team support them in implementation. All attendees 'agreed' or 'strongly agreed' (with most strongly agreeing) with this statement. The impact of the programme was also discussed in the workshop and attendees were broadly of the opinion that their work would not be possible without the support of the programme.

The additional evidence collected during the implementation research allows us to conclude that the programme is contributing, and further evaluation can inform our understanding of the size of that contribution. We explore some of the options for undertaking such research in the next section.