Learning from 25 years of preventative interventions in Scotland

5. The Prevention Evidence Base

Evaluation is essential for evidence-based policy making, as it provides an assessment of whether policies and interventions are being delivered as intended, producing the intended results and helps identify ways to improve them.^[6],[7] The recently published Evaluation Action Plan sets out the Scottish Government’s vision for evaluation. In the forward the Minister for Parliamentary Business states:

‘By helping us learn what works, and what doesn’t, evaluation can give us insights, help us improve the delivery of our programmes, and help us ensure we are providing value for money. Better evaluation can mean better outcomes for the people of Scotland.’^[8]

However, it also needs to be acknowledged that evaluating preventative interventions is not always straightforward. Evaluation Support Scotland point out that evaluating prevention is difficult, in particular, measuring something that has not yet happened.^[9]

5.1 Strength of the evidence

While a critical appraisal of the quality of the evidence ^[10] was not conducted, a ‘traffic light’ classification of confidence in the evidence was used to inform the selection of case studies. The classification included whether a policy/ programme was evidence-informed, whether there was robust evidence on the preventative impact of the intervention, or else good process evaluation evidence and some evidence of impact. Consideration was also given to whether an evaluation had led to the improvement of programme delivery, or to additional funding for an intervention.

5.2 Types of evaluation evidence in the case studies

The case studies include interventions evaluated using a range of different methods and include qualitative, quantitative and mixed-method study designs. The case studies were selected primarily on the basis of the strength of the evidence, as their principal aim is to demonstrate examples of the actual preventative impacts (social and economic) of a policy, intervention or programme. Therefore the vast majority of case studies include some type of impact evaluation, including, for example: randomised control trials (e.g. financial incentives for smoking cessation in pregnancy, Childsmile); quasi-experimental designs with before/ after measures (e.g. NNP, 20 mph limits) and comparison sites (e.g. 20 mph limits, Safeguarding Vulnerable Road Users); theory-based evaluations (MUP, Childsmile) and natural experiments (MUP).

It was also important to include economic evaluation, in order to highlight the cost effectiveness of preventative interventions, and whether the preventative benefits of the policy or programme justify the costs. There are fewer of these evaluations (as highlighted in a recent review of evaluations of Covid-19 interventions),^[11] but they are instrumental in providing evidence on cost effectiveness, informing funding decisions and increasing accountability and transparency. Around half of the case studies included some form of economic evaluation, e.g. Childsmile, Covid-19 Vaccines, FSS, financial incentives for smoking cessation in pregnancy, MUP, the NNP and smokefree legislation.

While the focus was on the impact of preventative interventions, process evaluations were also important in demonstrating how a preventative intervention was implemented/ the extent to which it was delivered as intended, as well as exploring the range and diversity of experiences and the programme outputs. The vast majority of case studies included some form of process evaluation. These were used to inform and refine the programme (e.g. Caledonian System, Breastfeeding Friendly North Lanarkshire), to inform the impact evaluation (e.g. financial incentives for smoking cessation in pregnancy), and to help interpret the results from the impact evaluation (e.g. MUP).

Most of the case studies used multiple evaluation methods, with around half combining impact, economic and process evaluations. Case studies with all three types of evaluation are better able to provide a full understanding of whether an intervention worked, how, why and for whom, and at what cost.^[12]

Only a minority of case studies took a single evaluation approach (Safeguarding Vulnerable Road Users, BFNL and Access to Welfare Advice in Schools). The most common approach was a combination of impact and process evaluations, or of impact, economic or process evaluation.

5.3 Examples of high quality, innovative and robust evaluation

Several of the case studies include examples of innovative evaluation methods, such as pioneering data linkage approaches. The effectiveness of the Covid-19 vaccines was demonstrated by EAVE II (Early Pandemic Evaluation and Enhanced Surveillance of Covid-19), which was one of the first national scale healthcare surveillance platforms in the world. EAVE II has received international recognition for its work investigating the real-world effectiveness of the early COVID-19 vaccines.

The financial incentives for smoking cessation in pregnancy case study shows how multiple studies of incentive schemes that began in Scotland influenced policy in England, informed an international evidence-base and were included in Cochrane reviews. The studies also directly resulted in a change in NICE guidance from 2021.

MUP was the focus of an extensive multi-component evaluation coordinated by Public Health Scotland (PHS), which included 12 evaluations and over 40 publications.

Outcomes from the Childsmile intervention are also being investigated via a data linkage project, which involves linking multiple routine administrative national health and education datasets to evaluate the effectiveness of the programme. The evaluation of FNP also involved an innovative approach to data linkage using a natural experiment methodology.

The case study on Safeguarding Vulnerable Road Users was the first road trial of its kind and has become the largest known study of motorcyclist behaviour in the world.

5.4 International learning

Some of the case studies were based on rigorous high quality international evaluation evidence before being implemented in Scotland. Whilst programmes do not always transfer and adapt well from one geographic or cultural setting to another, there are several examples of preventative policies developed in other countries that have been successfully adopted in Scotland.

For example the FNP started in the US in the 1970s (known as Nurse-Family Partnership). It is underpinned by a body of academic literature and has received the highest possible evidence rating from the Early Intervention Foundation.^[13] FNP was brought to Scotland under license and rolled out in 2010.

The Housing First approach was also developed in the US and is underpinned by compelling international evidence on the effectiveness of the approach in ending homelessness for people with co-occurring mental health and/or substance misuse issues.^[14] Between 2010 and 2013, Turning Point Scotland delivered the first pilot Housing First approach in the UK.

The Caledonian System was developed from 2004, following a call from the Scottish Executive Effective Practice Unit to develop an accredited domestic violence intervention. It is informed by international evidence and best practice on what works to prevent domestic violence.

5.5 Other characteristics of the evaluations

Several of the evaluations were commissioned by the Scottish Government (SG) or an agency of the SG. There is not a complete record of the costs/ overall investment in evaluations, but where this information is available it ranges from £20,000 to several hundred thousand pounds. Putting an exact price on the cost of some of the larger programmes of evaluation work is difficult due to the mix of evaluative work underway.

Many of the evaluations include examples of collaborations often (but not exclusively) with University researchers. Examples include the EAVE II collaboration for the Covid-19 vaccination evaluation and the Scottish Collaboration for Public Health Research and Policy for the 20 mph limits evaluation. Partnership approaches between evaluators and organisations such as local authorities and health boards are also common across the case studies.

The scale of the evaluations are influenced in part by the size, impact and profile of the policy or intervention. Some of the evaluations are part of a larger suite of co-ordinated studies, covering a number of study designs (e.g. FSS, MUP, SCP) or else different aspects of the intervention have been evaluated separately a number of times (e.g. Smokefree Legislation, Covid-19 vaccines, NNP, Childsmile, 20 mph limits). However, many of the evaluations are standalone / one-off evaluations of an intervention (e.g. Safeguarding Vulnerable Road Users, the Caledonian System, Access to Welfare in Schools).

While many of the case studies highlight evaluations that have now concluded, for some the evaluation programme is ongoing (e.g. Childsmile), or it is likely that the intervention will be evaluated again in the future (e.g. Caledonian System).

One of the challenges is understanding the impact of a preventative intervention over time. While some evaluations were run over a longer time frame and better able to capture this (e.g. the evaluation of Smokefree Legislation between 2005-2011, the evaluation of 20 mph limits between 2017-2020, or through longitudinal research^[15] designs such as Childsmile and MUP), others took place over a much shorter timeframe and were not able to demonstrate longer term preventative impact (e.g. Caledonian System, Access to Welfare in Schools, BFNL, SCP).

There is evidence to suggest that with some more complex preventative interventions, it may take several years for the benefits to be felt. For example early evaluation evidence from Sure Start in England^[16] (a network of children's centres and other services to support local families with children under 5) showed increased service usage in the early years, however longer term evaluations have demonstrated significant service demand reduction across health care usage and additional education needs alongside increased educational attainment.