Innovation data baseline: final report

5. Mapping of Indicators

The sections above have presented the detail on the three agencies current approaches, process and systems used to measure the benefits and impacts generated through innovation spend/investment (across specific interventions that are broadly representatives of the wider innovation support landscape).

The section below provides a high level mapping across the three agencies' current approaches to performance monitoring/measurement against the conceptual model and indicator frameworks presented in Sections 2 and 3. This is not intended to be a comprehensive review and will provide a 'snapshot' of the relative strengths and gaps/weakness of the current approaches.

We have prepared a Green Amber Red (GAR) assessment based on the following:

• Green - the processes currently in place have strong alignment with the conceptual framework and are gathering and reporting a range of relevant data;
• Amber - the processes currently in place have alignment with the conceptual framework, however there are gaps either in terms of the indicator data (what is being gathered) or the data collection process (e.g. forecast data, timescales, attribution or inconsistencies); and
• Red - the processes currently in place are not gathering or reporting against the indicators outlined within the conceptual framework and/or there are challenges with data collection.

While only comprising a high level review, Tables 5.1 - 5.6 reinforce the consistent message in the preceding sections that data collection is variable across the agencies and inconsistent across individual projects/programmes.

Specifically, we would note that the collection and reporting processes for gathering data on the inputs to innovation (which are mainly financial), and to a lesser extent, the activities that are supported, are relatively robust across the cross-section of programmes/projects that were reviewed.

However, if we look at the output/outcome and impact data being captured and reported, current approaches are not able to accurately and consistently capture and report performance across the agencies. One of the key findings is that, across all the projects/programmes there is no data being gathered or reported with regards to increasing productivity. As noted, a key strategic objective for Government is for Scotland to be within the top quartile of OECD countries in terms of productivity (and equality, wellbeing and sustainability), and investing in innovation is one of the mechanisms to help achieve this. Given this focus, it would therefore be reasonable that some consideration to capturing and measuring the impact on productivity would be appropriate. How this is done/achieved accurately and consistently is a more difficult question to answer.

Based on all the review work and discussions with stakeholders, it is worth highlighting that at the individual project/programme level some of the current systems/processes would only require relatively minor adjustment to strengthen their practice to monitoring, whilst across others there are some notable gaps that would require more fundamental revision.

Looking at the current practice of the three agencies, it is fair to say that while we have a relatively good understanding and evidence base for what activities the investment and inputs into innovation are delivering, we have less detail and evidence for the outputs and longer term impacts. We are therefore unable to test and validate whether our theory of change for innovation holds true or assess the entirety and extent of the returns that might be delivered. This is not to say that the theory of change is not valid - just that the current methods and approaches do not gather and report the relevant data to robustly assess.

This last point is particularly salient. As noted in the upfront section, the Enterprise and Skills Analytical Unit identified that the purpose of innovation is to generate a positive change ("new ways of combining existing (and/or new) resources to better address existing (and/or new) needs").

In the context of providing investment to the three agencies this is with a clear focus on using innovation as a driver for economic growth and productivity. The available performance monitoring data evidence does not provide a sufficiently clear assessment of the extent to which the investments made are delivering against these objectives.