Scottish Rural Development Programme 2014-2020: Knowledge Transfer and Innovation Fund evaluation

Methodology

This chapter outlines the analytical approach taken to this evaluation. The evaluation was conducted in-house by Scottish Government Social Researchers.

Research questions

The current evaluation considered the following research questions:

• Did KTIF funded projects improve skills and knowledge?
• Did KTIF funded projects deliver innovation and improvement on the ground?
• Are there any particular positives that occurred as a result of the events/projects?
• Did KTIF provide funding to interested organisations?
  • How much funding was provided?
  • How many projects were funded?
  • What was the average amount awarded per project?
  • What was the acceptance rate of applications?
  • What were the main reasons for rejected applications?
  • Which type of organisations and how many of each type applied?
  • How was KTIF advertised/how did recipients hear about it?
  • How easy or difficult was the application process?
  • What award criteria were applications considered against?
• Did the organisations funded by KTIF deliver vocational training designed to develop skills and transfer knowledge?
  • What kind of events were delivered?
  • How many training days were delivered?
  • What kind of topics were covered?
  • Who attended the events?
  • How many participated in the events?
• Did the organisations funded by KTIF deliver projects aiming to introduce new and innovative approaches to agricultural practice?
  • What kind of projects were delivered?
  • What was the broad topic/problem the project sought to address?
  • How many partners were involved in projects?
  • Who was involved in the project?
• How was the topic area or need for the project identified?
• What steps were taken to ensure any findings or learnings from the funded projects were communicated or put to practice more widely?
• Was KTIF good value for money?
  • What % is KTIF of the overall SRDP 2014-2020 budget?
  • What were perceived as the main benefits of KTIF?
  • What % of project budgets was supplied by KTIF?
• How can we improve KTIF in the future?
  • What changes should be made to KTIF; is there anything to be done differently?

Research ethics

The research was approved following the internal Scottish Government ethics process. The interviews were voluntary, recorded with respondents’ consent, and respondents were provided with a participant information sheet providing information on the rationale and purpose of the research.

Methods and analysis

Three main data sources were used as an evidence base for the evaluation:

1) Interviews with respondents who were e.g. involved in the delivery or administration of KTIF; in receipt of KTIF funding; or working for key stakeholder organisations.

2) Final project reports from projects that were awarded funding.

3) Administrative and financial data.

Firstly, the eight interviews were conducted by a Scottish Government Social Researcher and were conducted as video calls over MS Teams. Each interview took around one hour. Six interviewees were from external organisations, while two interviewees were from the Scottish Government (though one had left the organisation by the time the interview took place). All interviewees were chosen because of their experience of, knowledge of, and/or involvement in KTIF (through applying for funding, sitting on the Project Assessment Committee (PAC)^[1], and/or having expertise in knowledge exchange and innovation in the agricultural context). Five interviewees had experience of sitting on the PAC, and four interviewees represented organisations that had received KTIF funding.

The interviews were semi-structured, giving the interviewer some flexibility in pursuing issues raised by the respondent, while covering the main topics across the interviews. A copy of the interview schedule can be seen in Annex C and an overview of interviewees in Annex D. Some detail on the interviewees (e.g. whether they sat on the PAC) has been added throughout the report to help situate their contributions in a wider context.

Each interview was transcribed and analysed thematically. The qualitative analysis involved an initial reading of transcripts in order for the researcher to familiarise themselves with the data and to identify potential recurring themes across the transcripts. The transcripts were then uploaded onto NVivo, a qualitative data analysis software, and coded into themes and sub-themes. For example, one of the main themes was ‘application process’, and some of the sub-themes sitting under this were ‘transparency’, ‘feedback’, and ‘criteria’.

Secondly, each project that was awarded funding submitted a final report to the Scottish Government. These reports vary in length and detail, but in general provide background to the organisation(s) involved and the topic area of the project; an overview of the aims and objectives of each funded project; key outputs and milestones; an overview of budgets and expenditure; and an overview of which/how outcomes were met.

Finally, the evaluation drew on administrative data such as details on budgets and expenditure; meeting notes; guidance documents; and terms of reference.

Limitations

The final project reports submitted by those in receipt of funding included evaluative data and information but these varied in depth and quality. Some reports did not include any evaluative data. The final report template included sections for project outcomes (‘how aims/objectives were achieved’ and ‘milestones’) as well as lessons learned (‘issues and challenges’ and ‘impacts’). The guidance notes reiterate that funded projects should report back on aims and objectives and if they have been achieved, and notes that information can be pulled from other reports produced (progress reports and evaluation reports).

Data were primarily gathered and analysed internally by the funding recipients, and only a limited number of projects were subject to external, and therefore independent, evaluation. There was no set methodology for conducting project level evaluations, so each funding recipient took a different approach to reporting any achievement of outcomes. Therefore, no direct comparisons can be drawn across the reports. Reports often gave limited or no information on e.g. respondent numbers, questions asked, and other key methodological information.

Interviews were conducted with eight respondents – although this provides a good basis for the current evaluation, further interviews could have been conducted with a broader set of stakeholders, time and resources permitting.

There are limitations to the extent to which any (especially longer term) outcomes can be known or evidenced. For example, while some final reports provide indications that participants intended to make farm changes (or reported they had implemented changes), it is not known if the intended changes were eventually implemented, or, if changes that were implemented, were still in operation in the longer term (and what the impact of these changes was).

Quality assurance

This report has been subject to multiple rounds of comments and feedback from Scottish Government analysts and policy officials.