National Development Project Fund (NDPF) - evaluation: final report

1 Introduction

The National Development Project Fund (NDPF) is part of the Scottish Government's investment in services to reduce problem drug and alcohol use and reflects the priorities of the Rights, Respect and Recovery strategy published in November 2018. Ten projects were awarded NDPF funding in January 2019 to address gaps in advocacy services, family inclusive services, and start-up investment for new approaches to recovery. In October 2019, the Scottish Government commissioned Iconic Consulting to evaluate the Fund. This is the final report of the evaluation, which is intended to share learning from the projects and evidence their impact.

Background 

Rights, Respect and Recovery is the Scottish Government's strategy to improve health by preventing and reducing alcohol and drug use, harm and related deaths. The strategy focuses on prevention and early intervention, developing recovery orientated systems of care, promoting a human rights and person-centred approach to individuals and families, and a public health and evidence-based approach that puts lived experience at the core. The figure below – taken from the Rights, Respect and Recovery Action Plan 2019-21 – summarises the strategy's vision, priorities and outcomes.

The Scottish Government introduced three funds to deliver an investment of £20million per year over the duration of the current parliament (2018/19 to 2020/21) to support the delivery of services to reduce problem drug and alcohol use. NDPF was one of the three funds. £1million per annum was ringfenced for the NDPF for projects of national significance operating at either a national or a local level focused on addressing gaps in:

• Advocacy services.
• Family inclusive services and support. 
• Start-up investment for new approaches to treatment, support and recovery. 

Following an open application process the ten projects shown in Table 1 were awarded funding in January 2019, to the end of March 2021.

Evaluation

The overriding purpose of the NDPF evaluation was to provide the Scottish Government with timely, robust information on the performance of the Fund and the project management in order to support shared learning.

The evaluation had three specific aims, each one accompanied by research questions, namely:

1. To enhance understanding of the programme development process to support continuous improvement of the fund management and delivery.

• How accessible and relevant was the application process and how can it be improved?
• How well designed is the programme as a whole?

2. To support individual projects in an advisory capacity to improve the quality of self-monitoring and evaluation. 

• Are the proposed self-evaluation approaches appropriate?
• Will they provide adequate information to support a national evaluation?
• How can they be improved?

3. To provide an overview of the outcomes achieved by the individual projects and how the programme has performed overall, to support shared learning. 

• Has the programme met its overall objectives?
• What are the shared learning messages and how can these best be disseminated?

To address the first study aim, the evaluation team reported on the design, development and initial management of the Fund in December 2019. This short report was informed by feedback from the ten funded projects and the Scottish Government's alcohol and drug policy teams. 

To address the second study aim, the evaluation team initially reviewed the ten funded projects' monitoring and self-evaluation plans set out in their application forms. Individual meetings took place with the ten projects in November and December 2019 to discuss their plans and potential gaps or weaknesses identified by the evaluation team. Where necessary, recommendations were made to address gaps or strengthen the monitoring and self-evaluation approaches. Support was also offered to help implement the recommendations, where required.

This final evaluation report addresses the third study aim. At the outset of the evaluation, the Scottish Government emphasised the importance of establishing learning of national significance which could, potentially, inform the future development of services to reduce problem drug and alcohol use across Scotland. Evaluating the Fund was challenging given the diverse nature of the ten projects which varied in their approaches and target groups, not only across the three priorities – advocacy, family inclusive services, and start-up investment – but also within each of the three priorities. The evaluation therefore focused on gathering evidence from each project and drawing out learning where possible. Fieldwork consisted of: 

• A document review including monitoring reports submitted by the projects to Scottish Government, internal progress reports, promotional material, case studies and other outputs produced by the ten projects.
• Interviews with 18 individuals responsible for managing or delivering the ten projects.
• Interviews and feedback from 15 stakeholders connected to the ten projects including representatives of alcohol and drug services at an operational level and Alcohol and Drug Partnerships (ADP) at a strategic level. 
• Interviews with 11 beneficiaries from seven of the projects who provided valuable first-hand experience of the support they received and its impact.

The fieldwork was undertaken between October and December 2020. Interviews were conducted via telephone and video conferencing due to COVID-19 restrictions. 

Limitations of the evaluation 

It was not possible, at this stage, to fully assess the impact of NDPF funding. There were two limiting factors.

The first issue was one of timing. The projects were ongoing at the time of writing this report with funding continuing until the end of March 2021. As noted above, fieldwork for this evaluation was undertaken between October and December 2020, and the latest monitoring reports related to the period July to September 2020. This was six months before the projects' planned end meaning further delivery and impact was still to take place and not captured by the evaluation. In addition, the projects started at various points during Spring 2019 meaning the delivery time to the end of September 2020 was approximately 18 months. This was a relatively short delivery period that included, in some cases the need to recruit and/or train staff, and raise awareness of the project with referral partners and the target groups, before delivery commenced. Delivery itself was also affected by the COVID-19 pandemic as discussed in more detail in Section 3.

The second issue was one of evidence. The funded projects were required to submit quarterly monitoring reports to the Scottish Government using a template provided at the outset. The timely submission of these reports was mixed, limiting the availability of information on implementation and impact available for this evaluation. Five projects submitted all of the monitoring reports required between April 2019 and the end of September 2020. Another project submitted monitoring reports covering the whole period albeit in the form of one report covering April 2019 to March 2020, as well as reports covering the quarters before and after this period. The researchers did not have access to all of the quarterly monitoring reports for the remaining four projects.

Report structure 

This report is structured as follows:

• Section 2 discusses the learning from the ten projects with specific reference to the three priorities of advocacy, family inclusive services, and start-up investment.
• Section 3 covers implementation and impact of the ten projects, the impact of COVID-19, and fund management.
• Section 4 presents the evaluation conclusions and how they relate to the evolving policy context.