Techscaler Programme 2022-2024: early evaluation - main report

Appendix D: Lessons learned from undertaking the early evaluation

The main challenges encountered in the design and delivery of the early evaluation of the Techscaler Programme, and which will provide useful learning for the Scottish Government in commissioning future evaluations of the programme, can be clustered under the following themes.

Some other internal lessons have also been shared directly with the Scottish Government client team.

The Techscaler Programme does not operate in a vacuum — rather it is one part of the wider Scottish entrepreneurial ecosystem

The STER was clear that ‘the value of interventions should be assessed only with respect to their impact on the overall ecosystem’s output, rather than being limited to the scope of that part of the ecosystem in which they are applied.’ The Techscaler Programme does not operate in a vacuum — for example, Scotland’s enterprise agencies support businesses to innovate and scale, and Scotland’s universities play a key role in nurturing talent in the tech sector and creating spinouts.

It is also worth noting that while Cost Benefit Analysis (CBA) is a recognised good practice approach for economic appraisal, it will underplay/value some of the wider (tangible and intangible) benefits we would anticipate being generated from a programme of this nature in future years, such as positive spillovers, recycling entrepreneurial talent and agglomeration effects.

External economic, social and political factors, for example shifting policy priorities, availability of funding (both public sector and private sector investment), regulation and legislation might also positively and/or negatively affect the Techscaler Programme’s ability to deliver on its stated objectives.

Techscaler information and data

What success looks like for the Techscaler Programme was not explicitly defined within the Services Contract

Rather, by design the Scottish Government procured the delivery of the Techscaler Programme to CodeBase as an innovative contract and set out seven high level objectives to allow for flexibility in design and guide delivery. While this has helped to encourage a flexible and agile approach to delivery it does present some challenges such as varied stakeholder views on what Techscaler is (and is not).

Quarterly delivery plans/reports are agreed between CodeBase and Scottish Government that set out what has been delivered in the previous quarter and what the upcoming delivery phase looks like. Scottish Government also has monthly meetings where policy colleagues discuss delivery and progress, including what has been achieved and what is upcoming.

It has been challenging to robustly evaluate the Techscaler Programme as there is no clear definition of success (or what a successful end point looks like) and there are no intermediate measures/metrics (such as defined targets) to assess progress and performance against. That is, metrics beyond activity and output measures. This potentially under-represents the wider impact of the programme.

Key information and data should be available at the start of the commission where possible

Scottish Government analysts and policy should endeavour to have key information, including financial information, available at the start of the commission and for policy colleagues to be clear about parameters. This could help to minimise any delays in providing information and data. Briefings could set the evaluation in context and explain the difference between evaluation and audit.

The Scottish Government has recently established a Centre of Expertise in Appraisal and Evaluation to provide advice and training to analytical and policy colleagues on conducting appraisals and evaluations. The centre aims to ensure government policies and interventions offer value for money and to learn what does and does not work. It has responsibility for delivering the Scottish Government Evaluation Action Plan (August 2024) and is jointly led by the Scottish Government’s Chief Economist and Chief Social Researcher.