Techscaler Programme 2022-2024: early evaluation - main report

Appendix H: Technical impact assessment

This appendix provides additional detail on the data and methodology used to carry out the EIA and then includes additional results for the EIA including outliers and incorporating forecast impacts. We have provided a high-level guide to support readers’ understanding of technical terms and concepts referred to within this section.

Economic indicators and coefficients

A summary and description of the (quantitative) economic indicators used within the assessment is provided below:

• FTE jobs — used to measure the direct annual employment effects within the supported member companies that have or will be created. FTEs are based on average sector coefficients linking GVA to FTEs. We have developed the following coefficients and have matched them on a case-by-case basis to each supported beneficiary based on their reported SIC codes from Companies House. Where SIC data was not available an average coefficient based on all supported members was applied (see Table H.1).
• GVA — is a measure of economic output that considers the value of goods and services produced before allowing for depreciation or capital consumption. At a micro-level GVA is the contribution of each individual producer, industry or sector to the economy and measures the income generated by businesses after the subtraction of input costs, but before costs such as wages and capital investment. The evaluation has found that Techscaler has had a positive effect on productivity and is assessed as the forecast uplift in economic output per supported member. GVA has been calculated based on the reported turnover uplift and converted into GVA using sector average co-efficient ratios using data sourced from official government statistics^[36] (see Table H.1). The ratios are applied to the turnover generated by each supported member on a case-by-case basis.

Table H.1 summarises the economic coefficients used. Note that economic multipliers are also included as they are calculated based on the same SIC matching exercise.

Where SIC codes were not available, average coefficients based on the rest of the sample were used — the average coefficients are as follows:

• Turnover to GVA ratio: 0.59.
• GVA per FTE: £71,260.
• Turnover Multiplier: 1.52.
• GVA Multiplier: 1.51.
• Employment Multiplier: 1.44.

Gross and net impacts

• Gross impacts — the direct impacts that measure the overall change in economic activity that has occurred over the 10-year appraisal period.
• Net additional impacts — is the difference between what would have happened anyway in the absence of the Techscaler Programme (that is, the reference case) and the impacts/benefits generated by the support (that is, the intervention case), adjusted for displacement, leakage, deadweight, and multiplier effects:
• deadweight refers to the benefits and costs of an intervention that would still have occurred if support was not provided.
• the impact that growth within supported members is estimated to have on other businesses and the labour market (displacement).
• the proportion of impacts that will benefit those outside the defined spatial area (leakage outside Scotland).
• the positive spin-off benefits generated through income and supplier multiplier effects (that is, paying suppliers and salaries).

Other technical considerations

• Present Values (PV) — the total quantified value of the costs and net additional GVA over a defined timescale taking account of the time value of money (that is, £1 today is worth more than £1 next year). Impacts are discounted at the HM Treasury Social Time Preference Rate (3.5%).
• Constant Prices — the total quantified value of the costs and net additional GVA adjusted for inflation (that is, increases in costs and turnover are due to real growth and not inflation. Financial values are set at the base year 2024/2025. Impacts occurring in 2023/2024 and economic coefficients (where specified) are adjusted to 2024/2025 prices using data from the ONS GDP Deflator^[37].
• the Impact Profile is considered on an annual basis for a 10-year period from when a supported member either first accessed support or was expected to form a company to allow for the accrual of impacts. Members were first supported in 2023/2024 and the latest formation of a company is expected in 2026/2027. Therefore the time horizon for impacts runs from 2023/2024 to 2034/2035. This recognises that there may be considerable periods of time elapsed before impacts emerge. The EIA therefore reports on impacts that have been generated to date as represented in Chapter 7 (2023/2024 to 2024/2025) and that are forecast to occur in the future up to 2034/2035.
• Optimism Bias is the demonstrated, systematic tendency for ex-ante EIAs to be overly optimistic in forecasting outcomes (for example, time taken to implement interventions, costs of implementation, and impacts achieved). The method for applying optimism bias is described in Forecast Impacts section below.
• Grossing Up and Confidence Intervals — in order to calculate the overall impact of Techscaler it is necessary to ‘gross up’ the results to reflect the population of supported organisations. The impact data that is captured through the survey sample are ‘grossed up’ to the entire population based on the inverse of the proportion responding to the survey (for example, a response rate of 20% generates a grossing up factor of 100%/20% = 5.)

To date impacts including outliers

As noted in Chapter 7, the dynamics of an innovation ecosystem often produces a small number of companies who greatly outperform their peers and may themselves be considered outliers. Therefore, an additional analysis with the single “to date” outlier included in the “grossing up” process has been carried out with gross economic impacts and net additional economic impacts presented in Tables H.2 and H.3, respectively.

Source: Survey Data

Note: Jobs rounded to nearest 10. Turnover and GVA rounded to the nearest £1 million.

Source: Survey Data

Note: Jobs rounded to nearest 10 and GVA and turnover to nearest £1million.

Forecast economic impacts

The EIA presented in Chapter 7 considered economic impacts that have occurred to date (2023/2024 and 2024/2025). This analysis extends the EIA to cover forecast years up to 2034/2035. The methodology for quantifying impacts is identical to that described in the main report, however there a several further methodological steps taken when considering forecast impacts.

These are outlined below.

Attribution

The assessment of attribution considered reported attribution, level of engagement with Techscaler and other support programme accessed. For forecast impacts the temporal dimension is also considered with attribution declining in subsequent years following Techscaler engagement. This has been applied of the following basis.

Optimism bias

Optimism bias is applied to counteract the tendency for survey respondents to be overly optimistic about future impacts. The application of optimism bias represents a reduction in the reported forecast impacts.

It has been applied on a case-by-case basis and considers the following data points:

• stage of company — ideation/early-stage or growth/scaling.
• live product on the market — yes or no.
• level of investment/funding raised for growth/scaling companies. It is assumed that the level of investment must be over £100,000 to have an impact.

Table H.5 summarises the different rates of optimism bias applied based on the above criteria. The number of observations within the sample of supported members who meet each criteria for optimism bias are also specified.

Overall, the average level of optimism bias was 54%.

Business failure rates

A newly or recently formed business — ideation/early-stage members — are subject to new business failure rates. It is expected that within the startup and scaleup landscape many businesses will fail, and consequently fail to generate the forecasted impacts.

If there is a strong tech scale up ecosystem, this is not necessarily to the detriment of the technology sector as it allows for entrepreneurs to learn and re-engage with new business ideas that may be more successful in the long run. We apply Scotland’s Survival Of Newly Born Enterprises^[38] data to all ideation/early-stage members to simulate the proportion of business which are expected to fail. Note that after year 5, we assume a constant survival rate. Table H.6 presents these rates.

Outliers and grossing up

As the data has been ‘grossed up’ based on a sample population we have included a Confidence Interval (CI) to include additional sensitivity. Prior to grossing up, outliers are removed. Defined as being more than two standard deviations away from the mean forecast impacts, when considering the longer-term, 10-year impacts, 5 outliers have been identified.

Based on a sample size of 64 and population of 643, at a 95% confidence level, the confidence interval (or margin of error) is +/- 12%. The impact data has therefore been calculated and presented at a lower-point and upper-point estimate.

The ‘low-point’ and ‘high-point’ estimates reported in the tables below have been calculated by applying the (+/-12%) Confidence Interval to the ‘mid-point’.

Gross forecast impacts

The forecast gross impacts are presented in Table H.7. These represent sample impacts that have been “grossed up” to the population. They are also the “midpoint estimate” in that no margin of error has been applied.

Source: Survey Data

Note: Jobs rounded to nearest 10. Turnover and GVA rounded to the nearest £1 million.

Net additional forecast impacts

Applying the additionality factors of deadweight, displacement, leakage and supply chain and income effects as well as optimism bias and business failure rates to the gross impacts yields the Net Additional Forecast Impacts. These are presented in Table H.8 below.

Source: Survey Data

Note: Jobs rounded to nearest 10 and GVA and turnover to nearest £1million.

Economic efficiency and effectiveness

We have considered the value for money and return on investment (RoI) over the appraisal period considering both to date impacts (as presented in Chapter 7) and forecast impacts (as presented above). This is measured through:

• cost per job — the total net additional annual FTE jobs set against the total input costs (treated for Present Values).
• impact investment ratios — net additional GVA impacts set against the total input costs (treated for Present Values).

Cost per job

To date (up to 2024/2025) Techscaler has achieved a cost per job of £101,120 — while this is very high, it reflects the significant upfront investment from the Scottish Government and the time lag to generate impact.

By 2030, the cost per job is estimated at c. £8,750 — this is a very positive return, and we would note that it is in line with what we would expect for a startup and scaleup support intervention for high technology intensive companies.

Return on investment

Cost per job is simply a measure of inputs set against outputs, and when considering the wider value, particularly of a programme that is targeted at supporting high value/growth activity and reducing the productivity gap, it is important to consider the ‘economic value’ of these jobs. The RoI measures the wider economic value of these jobs (GVA) — the key Government measure for economic growth.

Table H.9 provides an assessment of the impact-ratios (return on investment) that Techscaler is forecast to deliver.

Over the 12-year appraisal period it is forecast that Techscaler will generate a RoI between 6.28 and 7.55. This means that for every £1 invested in the programme it will generate an additional £6.28 to £7.55 in net additional GVA within Scotland.