R100 Interim Evaluation

5 Methodology

5.1 Overview

5.1.1 This chapter sets out the approach to conducting the primary research with residents and businesses / organisations which aims to gather evidence of the potential outcomes and impacts identified within the logic maps. The chapter describes the relevant groups which are of interest to the research before detailing the approach taken to gather information from these groups. It includes details of the overall structure of the research, the methods employed, and the samples achieved.

5.2 Groups affected by R100

5.2.1 There are three groups of properties affected by the R100 programme and therefore of interest to the research:

• Group 1: those who have been provided with broadband via R100 and have taken up the connection
• Group 2: those who have been provided with broadband via R100 but have not taken up the connection
• Group 3: those who are scheduled to receive broadband via R100 but have not yet been connected

5.2.2 Group 1 clearly forms the focus of the research and Group 3 forms the basis for the counterfactual, a key role of which is to ‘validate’ Group 1’s estimate of their pre-R100 digital usage. There is also value in exploring why those in Group 2 have not taken any action as a result of R100. This could be due to a lack of interest, lack of necessary skills, lack of awareness, cost, or indeed a positive desire not to be connected / better connected to the internet.

5.2.3 Table 5‑1 shows the number of properties falling into each group as recorded in the Scottish Government R100 database.

5.3 Research Approach

5.3.1 The research involved two broad stages as follows:

• Stage 1: a survey of residential and organisations / businesses affected by the R100 Programme
• Stage 2: follow up focus groups / interviews with residents and businesses / organisations identified during Stage 1

5.3.1 In recognition of the potential for differential impacts across different groups in society, a specific Equality Assessment which aimed to evaluate the scale and nature of impacts across societal groups as defined within Public Sector Equality Duty (PSED), Fairer Scotland Duty (FSD) and the Islands (2014) Duty was also undertaken.

5.3.2 The approach to the delivery of each of the above stages and the equalities focused assessment is set out below.

5.4 Stage 1: Surveys

Survey Approach

5.4.1 In order to gather information from the three groups identified above, two online surveys were produced; the first aimed at residential properties and the second aimed at businesses / organisations.

5.4.2 To ensure only those in the above three groups and therefore those who were eligible to respond were able to access the survey, a postcode / address checker was used as a gateway to the survey. For this, each potential respondent was required to enter the postcode of their property. If their property was identified as falling within one the above three groups within the Scottish Government R100 database, the respondent was invited to select their property address from a subsequent drop-down menu. Should the property not be listed as in the Scottish Government dataset, the address did not appear when the postcode was entered.

5.4.3 Each survey incorporated separate sub sections accessible via appropriate survey routing for properties in each of the three groups identified above. Those who entered an eligible postcode and address were routed to the correct sub-section of the survey based on the group they fell into as recorded in the Scottish Government dataset. Where possible, this allocation was corroborated with the respondent in the initial survey questions. However, given that a high proportion of respondents are not aware that their current connection is or is due to be provided by R100, it was not possible to corroborate this information in all cases.

Survey Content

5.4.4 The surveys were structured around both the Investment Objectives and the logic maps. A broad overview of the topic areas covered in each survey is set out in the table below.

5.4.5 Given that the R100 programme is targeted at the household / property level and therefore the survey aimed to gather information at this level, it was requested that only one person from each household or business / organisation should complete the survey and they should do so on behalf of the households / business / organisation as a whole.

Survey Implementation and Promotion

5.4.6 The surveys were live for a 27-week period from October 22nd, 2024, to 1st May 2025.

5.4.7 The surveys were promoted to eligible respondents via:

• community awareness raising activities
• ‘To the occupier’ letters issued to eligible addresses

5.4.8 These are discussed in turn below.

Community Awareness Raising

5.4.9 The surveys were initially promoted via:

• the Scottish Government’s social media pages
• relevant email contacts who were asked to share the survey via both online media and their local networks:
  • the Scottish Government single point of contact at each local authority - the latter was asked to share the survey via the local authority social media pages and with any relevant external organisations / email groups (e.g. community councils). The Scottish Government engagement team provided assistance to local authorities with relevant material and appropriate text where required.
  • the organisations contacted to help inform the development of the Literature Review (see Appendix A)
  • all relevant community councils for whom an email address was available via online sources.

5.4.10 It is recognised that by definition online access for Groups 2 and 3 will be inferior to Group 1 and therefore promotional activities involving the internet such as social media may have been less likely to reach these groups. However, the results from Scottish Government’s evaluation of the Scottish 4G Infill Programme indicated that around 90% of respondents (in what are some of Scotland’s remotest areas) did have some broadband / 4G access.^[76] The use of online promotional activities was therefore felt to be reasonable, particularly given that it was delivered in combination with other activities such as promotion amongst local networks and occupier letters (see below).

Occupier Letters

5.4.11 Following an initial wave of survey responses, ‘to the occupier’ letters were issued to eligible addresses advertising the survey. The letters included information on R100 and the evaluation along with a link to the survey. In total, 4,500 letters were issued. The addresses to which the letters were issued were selected based upon consideration of:

• a target sample by group and the sample by group received to date via the initial wave of survey responses
• the distribution of properties across the Scottish Governments 8-fold urban-rural classification within the Scottish Government R100 database – this was used as a sample frame with the aim of maximising the representativeness of the sample achieved.

Achieved Sample

Number of Responses

5.4.12 In total, 941 unique^[77] household responses were received to the survey. The tables below show the breakdown of residential and business responses by group.

5.4.13 The overall number of household responses is satisfactory and provides a reasonable level of confidence in the results.^[78] As may be expected, the responses to the business survey are lower and therefore a note of caution is needed when interpreting the business survey results.

5.4.14 Figure 5‑1 shows the proportion of survey responses from each local authority area and Figure 5‑2 shows the survey responses as a percentage of the R100 properties in that local authority area. A breakdown of the residential and business responses by local authority area is included in Appendix B

5.4.15 The largest proportion of responses came from Aberdeenshire (21%), followed by Highland (20%). While the overall number of responses received accounted for a smaller proportion of the overall sample, a large proportion of responses were received from Orkney, East Renfrewshire and Shetland relative to the number of R100 properties in the area. There may be a variety of reasons for this, including potentially greater promotion of the survey by contacts in these areas.

Demographic and Business Profile

5.4.16 Information on the demographic profile of respondents to the resident survey and background information on the businesses who responded to the business survey is included in Appendix C Key points are as follows:

Residential properties:

• Couples with no children accounted for the largest proportion of respondents in the sample (48%), followed by couples with at least one child (29%) and single person households (13%)
• Overall, a third of households had children (at least one occupant under the age of 16) with the proportion with children slightly higher amongst Group 2 respondents (40%) compared to Group 1 (32%) and Group 3 (30%)
• Nearly a third of households had at least one member whose day-to-day activities were limited because of a health problem or disability

Business / organisations:

• The majority of respondents (89%, n=68) were commercial businesses with 11% (n=8) of responses coming from charitable / community organisations
• Businesses in the agriculture / forestry / fishing industry accounted for the largest proportion of respondents, making up just over 40% (n=25) of the sample. This was followed by businesses in the tourism sector (18%, n=11) and accommodation and food service activities (8%, n=5)
• The businesses were a mix of sizes - 6% had an annual turnover of less than 10k; 18% had an annual turnover between £10k and £50k; around 50% had an annual turnover between £50k and £500k; 19% had an annual turnover between £500k and £5 million and 3% had an annual turnover over £5 million
• On average, the businesses employed 6.5 full time equivalents. There were six sole traders in the sample; 31% (n=21) of the businesses employed temporary staff and 14% (n=9) used volunteers

Survey Analysis

5.4.17 The results of the Resident and Business survey are set out in Chapters 6 and 7.

5.5 Stage 2: Focus Groups / Interviews

5.5.1 In addition to the above surveys, a series of online focus groups / one-to-one interviews with residents and businesses affected by the R100 Programme were also undertaken.

5.5.2 As with the surveys, the focus groups / interviews were designed to gather information on the outcomes and impacts of R100 and the extent to which the project is achieving its Investment Objectives. However, while the surveys aimed to gather a representative sample of data, the focus groups / interviews aimed to explore the topic areas covered within the surveys in more depth and gather individual experiences to help supplement and augment the survey findings.

Approach

5.5.3 In order to gather potential focus group attendees / interviewees, a question asking respondents whether they would be interested in taking part in this aspect of the research was included within each of the above surveys.

5.5.4 In total, 233 respondents (207 residents and 26 businesses) stated that they would be interested in taking part in this stage of the research and provided their contact details for this purpose. Emails were issued to all 233 individuals asking for their availability across a range of dates. In order to encourage take-up, an incentive (a £30 Love2Shop e-voucher) was offered to those taking part in the focus groups. In total, 115 residents (out of the original 207) and seven businesses (out of the original 26) responded to the email regarding availability. Of those responding, three residents and one business stated that none of the dates were suitable.

5.5.5 Of the three categories of people affected by R100 (see section 5.2.1), there was a particular interest from respondents in Group 3. This is to be expected as this group is still awaiting connection and may therefore be more inclined to take part.

5.5.6 The resident respondents were allocated to six resident focus groups, with businesses being offered one-to-one interviews given their smaller number. Participants in the focus group were selected based on achieving:

• A roughly equal number of participants from each group
• A geographic representation which was broadly in keeping with the programme as a whole
• Availability of respondents to ensure meetings were well attended

Where possible, aiming for each focus group to contain attendees from just one of the three Groups affected by R100 – this was to assist with the discussion as the results of the survey suggested that the topics of interest would differ depending on which group respondents belonged.^[79]

5.5.7 In total, 49 participants eventually attended the focus groups / interviews (44 residents, and 5 businesses). The number of attendees at each focus group / interview is shown in Table 5‑5 along with the Group (1, 2 or 3) to which the attendees belonged. Table 5‑6 shows the total number of attendees by Group (1, 2 or 3) along with the breakdown between main contract and voucher recipients for Groups 1 and 2.

5.5.8 Geographically, the areas represented by the attendees were:

• The Highlands and Islands (postcodes IV, PH, KW, ZE, HS)
• West Coast and Argyll (postcode PA)
• Central Belt (postcodes EH, KY, DD, G6, G7)
• North East (postcode AB)
• Borders and South (postcode DG)

5.5.9 As noted above, only five businesses attended, all of which were in Group 3 and did not therefore have a broadband connection. Comments relating to business benefits / disbenefits should therefore be considered in this context. The more limited participation by businesses is perhaps to be expected given the monetary costs associated with attendance for this group.^[81]

5.5.10 In terms of an industry breakdown, two of the businesses were hydroelectric stations, one was a sheep and oyster farm, and two were accommodation providers (one Airbnb and one hostel).

5.5.11 The focus groups were delivered virtually via Microsoft Teams and lasted up to one hour each. There were two researchers in attendance, one steering the discussion and one scribe.

5.5.12 In advance of the focus groups, separate Resident and Business Topic Guides were prepared. Rather than a script, these were used as a broad framework to help guide the discussions. As such, all topic areas were not necessarily discussed in each focus group but rather the discussions were allowed to unfold as per the interest and knowledge of the participants in each case.

5.5.13 The focus groups were recorded to ensure all comments were captured and the recordings were deleted once the material was analysed.

5.5.14 The findings from the focus groups / interviews are summarised in Chapter 8. All comments have been anonymised, and identifiable details removed, including exact location, sex, and age.

5.6 Equalities Assessment

5.6.1 Given the scope of the R100 programme and the potential for differential impacts across different groups in society, an Equalities Assessment was also undertaken as part of the research. This aimed to evaluate the scale and nature of the impacts of R100 in relation to the societal groups and criteria as defined within PSED, FSD and the Island (Scotland) Act 2018.

5.6.2 The assessment used an integrated impact assessment framework which included Equality Impact Assessment (EqIA), FSD assessment, and Island Communities Impact Assessment (ICIA) and included a scoping and assessment framing exercise to align the above impact assessment approaches to the R100 programme, the development of bespoke framing questions, and a desk-based assessment in accordance with the framing questions and assessment criteria for EqIA, FSD and ICIA.

5.6.3 The findings from the Equalities Assessment were incorporated into the literature review and research findings as set out in Chapters 6, 7, and 8. Key recommendations to help mitigate some of the negative equality impacts identified are summarised in Chapter 9.