Scotland House London Evaluation

The evaluation found that Scotland House London is widely considered to be a valuable strategic asset supporting Scotland’s interests in the UK and internationally, while suggesting practical improvements to increase use and refine the offering of SHL


9. What lessons arise for future evaluation approaches across the international network?

Key findings

  • While it is evident that the SHL team collect a significant amount of information to inform their reporting, it is currently challenging to systematically identify and evaluate the extent to which strategic objectives are being met. Challenges include frequent updates and changes to SHL’s long-term priorities and the strategic objectives associated with these, a lack of SMART objectives and a lack of consistency across years in what data is collected and how it is reported.
  • As is often the case when evaluating organisations focusing on strategic delivery and soft power/diplomacy, many interviewees found it challenging to present tangible examples of SHL’s impact. However, they also found it difficult to suggest how impact could be evidenced more effectively beyond collecting more data.
  • The mixed method approach used in this pilot evaluation has provided an initial assessment of SHL’s work. A similar approach would work well on an ongoing basis and for the wider Scotland House network, but key considerations include ensuring a range of participants is included to reflect both the strategic and practical role of international offices and having long-term, SMART objectives against which they can be assessed.

Learning from the review of existing SHL reporting

Based on the documents provided, the main way SHL captures data on its activities is through a Business Planning Excel spreadsheet. This contains multiple tabs in which long-term planning priorities, strategic objectives and deliverables (both ongoing and quarterly) are recorded. Data from this spreadsheet then informs an end-of-year report, which is provided either in the format of a PowerPoint presentation or a Word document.

It is important for SHL and its funders to be able to evaluate performance; indeed, a key element of this commission is to consider how the Scottish Government’s international network can be evaluated more effectively in the future. While it is evident that the SHL team collect a significant amount of information to inform monitoring and reporting, issues were identified when attempting to use the existing documents to consider how effectively SHL had met its objectives.

Other issues identified were:

  • A lack of clarity in which activities were to be reported against which objectives – a particular issue in situations where an output could be reported against two or more objectives.
  • Outcome information is not provided or, more commonly, confused with outputs.
  • Whether an output deemed ‘ongoing’ in the quarterly deliverables tabs was an ongoing objective, or an objective with a fixed endpoint, but was ongoing.
  • A lack of annual reflection to consider the extent to which outputs had contributed to the overall priority objective at a collective level, making it difficult to revise or build upon these for the subsequent year.
  • A lack of consistency across years regarding the way quantitative data was reported and what was reported.

To illustrate this with a specific example, while X, formerly known as Twitter, monitoring data was commonly provided in the monitoring and evaluation reports, it was unclear what the purpose of X activity served, and reporting was inconsistent. For instance, in two of the six reporting years, the % increase in X followers was given, and in two of the six years, the X engagement rate was provided.

Similarly, across a few of the six years, the number of First Minister visits was recorded, while in others, they were not. Diplomatic engagements were noted in three of the years, but diplomatic meetings were reported in two. It was not clear if diplomatic engagement was distinct from diplomatic meetings or if these overlapped.

The most consistently reported figures were for membership numbers, event numbers, particularly international trade events, hub visitor numbers and Scottish Government Minister/Ambassadors meetings.

Another feature in the data was a lack of overall trends across the years. This could have occurred for several reasons, such as staff being unclear about what objectives were in place and why they were working towards them, or it could relate to different measures being used to report the same activity.

As a result of these challenges, it is not possible for the researchers to systematically identify the extent to which strategic objectives are being met. Moving forward, it is recommended that additional resources be provided to undertake the following:

  • A logic modelling exercise is undertaken to clarify how inputs, outputs and outcomes/impact relate to each other and are to be achieved.
  • A core set of SHL SMART key performance indicators (KPIs) is set, linked to each outcome area and associated outputs.
  • A simplified recording tool is designed, ideally with one tab for each outcome area, with an emphasis on keeping this up to date and comprehensively completed.
  • Agreement over what outputs and outcomes will be reported on annually, and ensure consistency in reporting.
  • Training and written guidance for staff explaining the purpose of outputs and how to measure and record their outputs and outcomes.
  • An annual review exercise to consolidate data gathered across the year and determine the extent to which the KPIs have been met.

The benefits of actioning these recommendations include:

  • An ability to track over time the extent to which outcomes are being met and therefore determine the greatest impact areas.
  • An ability to determine over time what level of activity is feasible within existing resources, reducing pressure on staff.
  • Consistent tracking and reporting, allowing trends over time to be determined and comparisons to be made.
  • Improved staff ability to discern between outputs and outcomes.

Interviewees' views on improved monitoring and evaluation

As noted in Chapter 5, many interviewees found it challenging to present tangible examples of SHL’s impact. It was widely recognised that SHL undertakes a significant volume of activity in the form of a full event schedule and meetings across multiple policy areas. However, the lack of detail interviewees could provide about the subsequent impact of that activity aligns with the challenges outlined in the previous section.

“A lot of the business-to-business engagement is positive and well-hosted. How much does it propagate the ultimate objective, which is furthering Scotland's economy, for example, I'm not clear” – Non-SG Stakeholder

“From an inward investment perspective, we have a tried, tested, established mechanism for tracking and reporting the outcomes that we deliver, which is nice and simple: the number of companies we've supported, the jobs that are going to be created as a result of those projects, expenditure on R&D, expenditure on capital investment, etc. It's all captured… And obviously, that's exactly the sort of thing that we feed into the evaluation documents that are produced for all the Scotland Houses, the international reports, etc. It's more difficult, I appreciate other colleagues, including in the Scottish Government, where your main activities are around brokering introductions, driving activity, supporting visits, etc. You don't have quite the same tangible outcomes to report.” – Partner

A few interviewees commented that they felt the amount of information and data that is being collected by SHL partners has improved over time, while others noted the challenges of being able to accurately capture detailed data.

“The measure of success was, do you have 5 events running in a week? Whereas what I used to say to [Scottish Government colleagues] was, you're looking at this completely wrong because if all you're doing is delivering activity that doesn't generate anything, then it's just costing your money and you're not going to return on your investment. I think we have moved to a different place now in terms of assessing and measuring and considering the return on the investment that we get… Also, anecdotally, those conversations will take place between my folks in the office and the Scottish Government counterpart. ‘Remember that event we did last year, one of the companies that was at it has now made an announcement that we're going to invest in Scotland’. Both informally and formally, through evaluations, we try to make sure that we're feeding in.” – Partner

A few interviewees were asked what else could be done to better demonstrate SHL’s impact. However, these comments provided few concrete suggestions beyond the need to collect more accurate data.

“Being able to track what comes through Scotland House’s activity and where it leads to [would be helpful]. And some of that might be long, slow burn. It's not always immediate, but having the ability to understand what the impact actually is and the outcomes that are being achieved through the facility and being a good host would be important.” – Non-SG Stakeholder

“The team now does have some data, but I think it would be good to be even more assiduous about collecting regular data on this. As I mentioned before, it can be hard to attribute fully successes. And there will be others who claim a key role in those as well, including those based in Scotland, including civil servants. But as far as possible, the more data we can collect, the better. They already do that, but insofar as we have resources to do it. We should be doing even more.” – SG Stakeholder

One member with experience in this area outlined the importance of creating and tracking against a well-informed plan. They also expressed the view that SHL activity should not be assessed in isolation, but also against comparable entities.

“Getting outcome measures that are tangible is very hard. And it may not be possible. But if you can say you achieved your objectives… “Did you do that? Yes or no? And if you did, well, what was that worth?” Keep it simple. I think you can, and then you can tell a story… We're too modest. We need to get realistic, I think, both about what you can expect from a place like this, but also, the way to do that is to benchmark against what others are doing. And see what works well for them in London and see what doesn't.” – Member

One partner raised a specific point of potentially needing to have a greater understanding of the relationship between partners and their reasons for being part of SHL.

“I think there's something around really understanding the rationale for why Scotland Food and Drink came on, why Highlands and Islands Enterprise left, why can't we get other partners, agencies from Scotland? What is it? Have they never heard of us, or do they look at it and say, I'm not paying X%, whatever that equals? I don't know.” - Partner

Applying learning to evaluating Scotland’s International Network

As noted in the introduction, the Scottish Government is supporting ongoing efforts to improve monitoring and evaluation of Scotland’s international network. One aim of this evaluation was to act as a pilot, with learnings and recommendations informing future evaluations of both SHL and other parts of the network.

For this initial evaluation, the in-depth interviews focused on SHL partners and senior stakeholders from the Scottish Government and other agencies. Their input has been invaluable, especially their strategic perspectives on SHL’s purpose. However, in some cases, their engagement with SHL was more limited, with their colleagues more likely to work with or visit SHL on a regular basis. This meant they were less able to comment on operational issues, including the membership model, or have specific examples of impact.

While reflecting on SHL’s overall mission should always be a feature of future evaluations, particularly if this changes over time, future evaluations of SHL may benefit from including more regular users, including more members, in addition to senior stakeholders. This could help to capture further practical improvements and provide a more detailed qualitative understanding of impact.

Another consideration is that this evaluation has focused on the experiences of current users of SHL. There could be value in exploring the views of less frequent users, non-users or potential members, either in future evaluations or as additional research. This could be helpful in understanding the size of a potential user base and any additional barriers to use.

The online surveys provided both quantitative and qualitative data to inform the evaluation, reaching a wider audience than would have been achieved with solely in-depth interviews. Importantly, the surveys provided data which can be used as a benchmark, allowing future evaluations of SHL to track changes in understanding, usage and satisfaction over time. For this to be effective, however, the same questions will need to be used year-on-year. Data for SHL could also be used as a benchmark to compare with other parts of the network. Again, this would require the same questions to be used, which may not be possible given the different models and objectives of international offices across the network. Consideration could be given to including a few core survey questions which are applicable across the network to facilitate future comparisons.

A significant learning from this evaluation is the challenge of evidencing impact. It was difficult to systematically demonstrate outcomes due to a lack of consistent, long-term objectives, inconsistent reporting, and limited use of outcome-based measures. Much of the value of SHL noted by participants is indirect or intangible, such as relationship-building, influence, and reputation. These are hard to quantify, meaning that evaluation currently relies on informal feedback and avoided cost arguments, rather than clear evidence of a return on investment. As outlined in this section, future evaluations would benefit from a small set of SMART objectives and consistent data collection focused on outcomes rather than outputs.

This evaluation has not examined reporting across the wider international network, but it is possible that the same challenges around evidencing impact will apply to other locations. If that is the case, it will be important to ensure clear objectives are set for each location, which can be assessed against both quantitative metrics and qualitative evidence. Given the different political contexts and models being used in each location, a significant degree of flexibility is likely to be needed. However, this evaluation suggests that there could be value in having an overall logic model, linked to the International Strategy, with staff supported to create and use a model that meets their needs. It is hoped that this would lead to the creation of clear, longer-term priorities and SMART objectives, at least some of which are consistent across the network. This would also support a full impact evaluation with consistent data collection and provide a degree of comparison where it is possible to do so.

Contact

Email: socialresearch@gov.scot

Back to top