Environment, Natural Resources and Agriculture research strategy 2027-2032: consultation analysis

7. Question 5

Theme: Impact approach

Question 5: Do you think the proposed impact framework is an appropriate way of defining, monitoring, and evaluating the impact of research funded through this programme?

Introduction

The vast majority of all consultation respondents (62, 89%) answered question 5.

The vast majority of respondents considered the proposed impact framework an appropriate approach but suggested refinements

Theme 1: Having an impact framework is essential

The vast majority of respondents (all respondent types) who answered this question were supportive of the use of an impact framework as a mechanism for defining, monitoring, and evaluating the impact of research funded through the SRP.

Some respondents said that the proposed impact framework would provide a ‘strong foundation’ or a ‘sound basis’ for embedding impact within the programme. There was feedback, for example, that the proposed impact framework would:

• provide a ‘clear structure’ for defining, monitoring, and evaluating the impact of research funded through the SRP
• ensure a stronger focus on outcomes through the theory of change model – and that this should help provide a clear route or pathway to implementation and impact. The theory of change would provide structure, coherence, and a means to synthesise and evaluate impacts across Missions
• help to support a stronger culture of co-development and stakeholder/ community engagement in defining, monitoring, and evaluating the impact of research funded through this programme – and that the impact framework has the potential to connect research outputs to real-world outcomes and impact

“Yes, the proposed Impact Framework is appropriate. It provides a clear structure for defining, monitoring, and evaluating research impact, and supports a culture of co-development and stakeholder engagement.” Seafood Scotland

“If Theory of Change is embedded as an impact planning tool, it could provide useful indicators for monitoring and formative feedback throughout the pathway to impact, significantly increasing the likelihood of achieving impacts at scale.” Individual respondent

Theme 2: Support for the proposed impact framework was typically caveated in some way (or conditional)

The majority of respondents who supported the proposed impact framework said refinements were needed to ensure the framework was ‘fully effective’ and ‘captured the real-world value of funded research.’ The impact framework was considered to have potential, but only if strengthened.

Measuring success and timescales for obtaining visible impacts

As noted, respondents welcomed the focus on outcomes and impact as part of the research strategy – they considered it vitally important that impact was an ‘integral part of the process rather than as an end product.’ It was considered important that the framework clearly recognised ‘Impact is often only achieved after a prolonged period of scientific endeavour’ and that it would be important that the research funded through the SRP did not unintentionally prioritise short-term impacts.

“The strategy should make sure that there is not over-reporting and prioritising quicker, short-term impacts over longer lasting change.” Lantra Scotland

“Different workstreams will deliver impact at different stages of the new five-year programme and this will need to be factored in to assessing valuable projects that may not have important early outputs. Thus, over reliance on this monitoring and evaluating approach will need to be considered to avoid devaluing projects with longer timeframes and fewer up-front outcomes.” Scottish Animal Welfare Commission

“The timescale for obtaining visible impacts is likely much longer than envisaged here. Again, perhaps it would be more productive to implement strategies that bring about outcomes based on existing evidence and proof of benefit, for example, advancement of organic practices and encouraging small scale local production.” Scottish Organic Stakeholders Group

The ‘inherent complexity’ of monitoring and evaluating the impact of research funded through the programme was frequently acknowledged in many of the responses. To support a more effective monitoring and evaluation process, respondents considered it important, for example, that:

• the impact framework was firmly grounded in a ‘quadruple helix’ model – connecting science, policy, practice, and civil society
• monitoring delivery of impact should be clear at all levels of the structure – Missions, Challenges, ARIs and project levels
• SMART metrics were developed to monitor and evaluate the impact of research funded through the programme – KPIs (including proxy indicators or evidence pathways) which could be used to gauge assessment of impact at different points in a project
• evaluation processes recognised the value of qualitative outcomes (and metrics) and lived experience data alongside more traditional quantitative outcomes (and metrics)
• the range of metrics and indicators used to capture and report on impact should go beyond policy and economic outcomes to include ecological, social, cultural and community impacts as well as reflect different types of research undertaken, ‘including community-led or small-scale work’ – and that impact should be considered and defined through ‘the lens of practical outcomes’
• the strategy was clearer in outlining which ‘policies and strategies the various Missions and ARIs enable and how’ – this ‘would help to evaluate whether intended impacts have been achieved’
• the three impact routes (related to the three defined customers) could consider other customers – for example, ‘land managers are an important “customer” group that would benefit from decision support tools. Nature itself, and the public as consumers, users and influencers of food systems and biodiversity, are key customers not included in the current model.’ Others felt that the impact framework was missing the general public as a ‘customer’ of the research
• the Missions and Challenges to be addressed by the research were described at a high level – it would be beneficial to provide greater detail on these, to identify specific indicators and evaluation mechanisms for assessing impact
• it would be useful to show what ‘success looks like’ in a given time frame, perhaps by ‘working back’ from what success looks like to identify the activities needed to bring about the desired change

Ensuring the impact framework meaningfully supported a process of continuous learning and development and looked to strengthen knowledge transfer activities were also identified as important. For example, it was suggested that the impact framework should:

• look forward, identifying adoption pathways and potential future impacts, not just retrospective outcomes
• recognise the value and learning from failure as well as success
• ensure mechanisms and opportunities for peer-to-peer learning, knowledge exchange, and for promoting success stories

There needs to be strong co-production and user engagement from the outset of the project inception through to post-delivery learning

Respondents were in broad agreement that community and stakeholder engagement and input should be central and embedded into processes for defining, monitoring, and evaluating the impact of research funded through the SRP.

Respondents said this would be key in delivering ‘valued and impactful research’ and ‘outcomes that were legitimate, trusted, and adopted at scale,’ and in helping ‘connect up areas of synergy across the programme’. The connection between research outputs and ‘practical delivery on the ground’ was considered vitally important in this regard. Limitations such as time and other constraints associated with a genuine collaborative and co-production approach were also recognised within responses.

Respondents suggested that:

• there should be more structured mechanisms for co-design and community engagement, ensuring that partnerships go beyond consultation to genuine co-production
• co-production should extend to the identification of key impacts and approaches to measuring success
• the strategy could be clearer on how the research supported could strengthen and empower communities to play stronger and more consistent roles in similar research in future
• the use of project steering groups that contained a mix of researchers, collaborators and stakeholders could be a useful route to help maintain the involvement of the ‘likely customers of the research’ during the lifetime of a project (and could also help to maximise impact)
• there should be flexibility within the impact framework to allow for effective collaboration and provide opportunities for different research methods to be used to achieve the required outcomes

“We also felt the impact framework was missing the general public as a ‘customer’ of the research. To safeguard biodiversity and the environment it’s important we engage with the public to put forward the case for why the environment needs to be protected.” Rowett Institute

“We suggest the list of beneficiaries (government, industry, and land managers) should be broader with inclusion of communities, civil society organisations, charities, and public service users. Distinguishing between beneficiaries who experience impact and stakeholders who enable or deliver it will strengthen the understanding of how impact is created and shared.” James Hutton Institute

“It is noted a desire to move to a process that mimics the Research Excellence Framework (REF) evaluation of UK Higher Education Institutes that involves significant stakeholder engagement and input through the production of testimonials. We would caution against inadvertently generating significant stakeholder fatigue, irrespective of how impactful research outcomes are.” Society for the Environment

“The greatest barrier to impact is when research stops at publication. Long-term impact will only be achieved if post-research phases remain farmer-led, regionally coordinated and embedded within established delivery networks such as Monitor Farms, where peer-to-peer learning sustains adoption.” Quality Meat Scotland

Ensuring processes for monitoring and evaluation are not overly burdensome or bureaucratic

Where mentioned, the general view provided by respondents (in particular Research Institutes and Centres of Expertise, and third sector organisations) was that processes ‘must not become overly burdensome’, ‘detract from delivery’ or simply become ‘a tick-box’ exercise. Rather support was expressed for an approach to monitoring and evaluation that: builds on and/or adapts existing reporting arrangements; was ‘proportionate’; enabled ‘iterative learning and reflection’; and ‘reduced administrative load.’ Respondents said:

• there should be clarity on expectations between funders and research providers from the outset to ensure a shared and clear understanding of the impact framework
• the development of co-designed templates could help support a consistent approach to reporting on the impact of the research funded
• appropriate training to support impact reporting should be offered

“concerns about…whether it will genuinely drive improvement rather than simply add another layer of reporting. The ambition to focus on real-world outcomes is welcome, but the framework risks becoming a tick-box exercise if annual reporting and “impact narratives” are not kept proportionate and meaningful. There is a danger that researchers will spend more time justifying their work than actually delivering impact, especially if support and training are not robust.” National Sheep Association

A comment made was that a greater emphasis on demonstrating the impacts of research may result in additional reporting requirements to provide supporting evidence – a suggestion made was that impact reporting should be built into research plans as milestones or deliverables, particularly in years 3 to 5, to help with targeting materials as the projects progress.

The use of case studies were welcomed but present only a partial or limited assessment of impact

Support was expressed for the proposed inclusion of case studies to complement other research outputs. They were generally viewed in a positive light, considered to align with existing approaches to impact and knowledge exchange, and respondents noted that case studies could help: demonstrate progress; highlight wider and less tangible impacts; and bring research findings to life. However, concerns were also raised that ‘focusing on case studies’ may give a ‘partial or skewed’ view of impact, as seen in REF exercises ‘where reporting was biased towards the most significant examples.’

The role of Impact Officers and Champions

The role of Impact Officers and Champions were broadly welcomed – this was considered a ‘positive step’ and respondents noted that they could play a ‘key role in connecting and promoting research outcomes within and beyond the environment, nature, agriculture and rural policy sectors.’ Respondents said these roles should be ‘proactive’ and ‘independent’ to gather impact narratives and case studies from the Missions. Respondents, however, called for further clarity in terms of their ‘selection, role, time commitment and training’ and also emphasised the importance of ensuring good information flow and communication of impact. Specifically, it was noted that there should be strong levels of communication across the Impact Officers and Champions.

A small number of respondents suggested that utilising existing Impact Officers and Champions could be beneficial, such as those from SEFARI and individual projects.

“The introduction of Mission Impact Officers and Champions could be valuable, but only if they are empowered to challenge underperformance and share learning across projects, rather than just collate paperwork. The framework’s flexibility is positive, but it must not become an excuse for vague or inconsistent measurement.” National Sheep Association

“The types of impact are highly varied, and the Impact officers/ Champions/ Challenge leads will need a specific skill set to effectively deliver these, requiring training (time and budget) to be allocated and delivered prior to the start of the SRP to allow stakeholder engagement from day one as planned.” Moredun Research Institute

The EPIC Centre of Expertise on Animal Disease Outbreaks noted that the approach to Impact Officers could be counterproductive for some Centres of Expertise.

“The draft strategy mentions the possibility of Mission Impact Officers being appointed from the Centres of Expertise. This may be a counterproductive move for EPIC, in so far as effort has been made already to establish that impact is the responsibility of all scientists, with those leading areas of work having a particular responsibility in this respect. To reflect the diversity of delivery models across the Centre of Expertise, it might be more effective to specify a requirement for impact to be supported, delivered and reported on, but to leave it to the individual Centre of Expertise to propose mechanisms to achieve this.”

Respondents who were unsure about the effectiveness of the impact framework

A small number of respondents (primarily individuals and other stakeholders) were unsure whether the impact framework offered an appropriate way of defining, monitoring and evaluating the impact of research funded through the programme. Some of the points raised by these respondents have been reflected in the narrative above and in summary included that the proposed impact framework:

• was considered too broad and did not provide enough specificity regarding outcomes and impacts
• lacked clear metrics or indicators for measuring impact (for example, productivity and profitability were mentioned)
• placed too much emphasis on case studies and this risked providing a partial view of impact
• risked operating in a ‘linear, siloed mindset’ – it was suggested that a holistic, cross-sectoral approach may be more beneficial in delivering meaningful benefits and achieving outcomes