Publication - Research and analysis

International Development - national indicator development: research report

Published: 6 Nov 2020

The report outlines the research commissioned for the development of the indicator ‘Contribution of development support to other nations’ that forms part of the National Outcome ‘We are open, connected and make a positive contribution internationally’ in the refreshed National Performance Framework

87 page PDF

1.2 MB

87 page PDF

1.2 MB

Contents
International Development - national indicator development: research report
Annex D: Process for Selecting Indicator components

87 page PDF

1.2 MB

Annex D: Process for Selecting Indicator components

Assessing data availability for the indicator components

A primary constrain on the 25 suggested indicator components is whether there are robust and accessible data sources that are regularly updated so change over time can be tracked. We also needed to consider the design criteria set out for the development of the NPF indicator, in particular:

  • A measure for Scotland, not just the Scottish Government.
  • Largely focused on those policy areas where Scotland has devolved responsibility.
  • Based on transparent and international data sources that can be updated in a timely fashion and with minimal cost, so that Scotland's progress can be assessed on a regular basis.
  • Provide an indicator for Scotland that can be compared over time and is not dependent on comparison with other countries.
  • Be a cardinal indicator where changes in value relate as directly as possible to changes in policy coherence. There is limited value in selecting an indicator that is not dynamic or is already at a high level and has limited capacity to change.
  • Wherever possible not replicate an indicator already part of the NPF.

Discussions with the Scottish Government also confirmed that indicators that required a series of judgements on policy characteristics (such as the MIPEX indicators of migration policy) would also be excluded as these require expert judgement, be potentially open to challenge and require significant resources to update on an annual or bi-annual basis.

The following table sets out the reasons for inclusion or exclusion of the proposed indicator components in more detail.

Table D.1: Reasons for inclusion/exclusion of indicator components

Measure

Reasons for inclusion/ exclusion

Comments

Scotland's connectedness to ODA recipient countries

Included

Importance of partnership working stressed by most stakeholders but significant challenges in agreeing a simple indicator and securing data without undertaking a bespoke survey

Civic/ Citizenship engagement on International Development

Not available for Scotland and would require additional questions to be added to the Social Attitudes Survey and so cost rules out inclusion.

Important measure but obtaining a robust measure for Scotland would be expensive

Value contribution of individual expertise

Expert contribution vital to policy coherence but no single measure can capture range of inputs

A measure that has challenged existing indices such as CDI. Cost of collection a major barrier.

School involvement in international learning

Important measure of engaging Scottish young people in ID issues

All schools required to participate in activity so measure of engagement is meaningless. Cost of bespoke survey of young people's attitudes rules out indicator of change in perceptions.

CPD impact on domestic service

Number of services engaged in partnership work highlighted the impact on individuals who volunteer/ work as experts who then re-apply this knowledge to their work in Scotland

Challenge to represent such impacts in a single measure across a wide range of service areas. Would require a bespoke survey and so ruled out on cost grounds

CDI indicator - integration policies (MIPEX)

International set of criteria to assess migration policies' impact on integration of migrants/ asylum seekers

Excluded on grounds that it requires expert judgement of policies many of which are reserved.

Asylum seekers settled in Scotland per 100,000 population

Included: Broad measure of Scotland's openness to a diverse and inclusive society

Included as data is available from National Statistics.

HE Students from DAC Least Developed Countries / Total Non-EU Students

Included: Core measure of engagement in HE. HEIs stressed importance of diverse student body for future income streams from foreign students. Wider point that Scottish alumni help 'soft power' and partnership working. Also a common measure in other nation's assessment of policy coherence.

Included as data available from HESA.

Contribution of Scottish people to DEC appeals per capita

Measure of public engagement in humanitarian issues especially countries facing emergencies

Robust data for Scotland is not available and focus on DEC appeals mean that the measure would be partial. Exclude.

% of total waste exported

Included: Measure of Scotland's ability to address its own waste without relying on other countries to pick up the burden

SEPA measure for proportion of waste exported but a more appropriate indicator is % treated domestically. Included.

Metric tonnes of carbon per person

Measure of Scotland's carbon footprint but already included elsewhere in NPF

Excluded to avoid duplication with other NPF measure

Ecological footprint of imports (gha per person)

Measure of Scotland's carbon footprint but already included elsewhere in NPF

Excluded to avoid duplication with other NPF measure

Value of imports from ODA countries

Included: Direct measure of the degree to which Scotland provides access to ODA products

Included. Data on goods imports by country available from HMRC.

Value of Agricultural subsidies

Included: Key measure from both CDI and PCDI indices and devolved responsibility

Included/ Annual report by Scottish Government.

SDI R&D Funding for Innovation Projects in ODA countries

Unable to establish the range of activity included in this pathway

Excluded no data available

Variation rate of the Gini Index pre and post taxes and transfers

Unclear how this measure in Scotland would impact on policy coherence

Excluded. Measure of income disparities already included in NPF

% of international development funding targeted at vulnerable groups

Included: Measure of the concentration of aid expenditure on vulnerable groups

Include. Data from Scottish Government

Value of R&D contracts in HEIs with ODA-eligible nations

Included: Measure of the research collaboration with institutions based in ODA countries. Reflecting the range of projects and collaboration undertaken by Scottish HEIs

UKRI Gateway to Research data covers all research councils and GCRF and Newton Funds

PCDI - S2 Health life expectancy

Measure tbc

Value of the Low Carbon and Renewable Energy Economy (LCREE) in Scotland

Included: Captures the extent of Scottish expertise in renewable technologies.

Included. Large scale survey undertaken biennially provides robust estimate for Scotland.

Energy produced from renewable sources

Already included elsewhere in NPF

Excluded as duplicated elsewhere.

Water management

Limited ID activity so measure excluded

No of ODA countries supported by Police Scotland

Simple measure that captures the range of work undertaken by Police Scotland

Include, more complex measures do not have the source data to support their inclusion

No of prosecutions arising from bi-lateral investigations

Key part of PS activities is joint criminal investigations e.g. modern slavery targeting criminals in both Scotland and ODA countries

Exclude no source data

Creating a composite indicator for the NPF

The ultimate objective of this assignment has been to identify a number of relevant indicators that together provide a practical measure of Scotland's contribution to the development of other nations. For the purposes of the National Performance Framework (NPF), this measure needs to be a single composite indicator. So the final stage of this process is to determine how this should best achieved using international best practice.

This review has looked closely at the design and methodology underpinning two indices that measure the commitment of countries to international development. The Center for Global Development Contribution to Development Index (CDI) and the Policy Coherence for Development Index (PCDI) both contain indicators with relevance to Scotland.

The third major measure of international development is the United Nation's Human Development Index (HDI) used by the United Nations Development Programme. The HDI is an index of life expectancy, education and per capita indicators, which are used to rank countries into four tiers of human development (Very high, High, Medium and Low).These indicators are primarily focused on measurement of development rather than the various conditions that promote the beyond aid agenda and so are less relevant to Scotland's contribution to the development of other nations.

Although it has not been possible to adopt any of the available international methodologies - CDI, PCDI the HDI - in their entirety, we have drawn on their respective approaches to developing an index to ensure that the proposed indicator builds on existing international best practice.

There is no standard methodology for combining different performance measures into a single index. There are, however, a number of stages to the process common to these three main indices. These are:

  • Techniques for addressing any missing values and data outliers
  • Standardised performance measures to be able to combine values so that they reflect degree of change
  • Combine these using weights to reflect relative importance in contribution to international development
  • The methodologies adopted for each index are reviewed and assessed for their relevance to the design criteria for the NPF indicator in turn below.

Treatment of missing values and data outliers

The three indices have a similar two-stage approach to the treatment of missing values:

  • If the majority of data points have missing values, then the country in question is excluded from the analysis entirely. CDI for example focuses on the G20 countries on the grounds that most data is available for this group.
  • For single (or a small number of indicators) missing data points often use a proxy value - most often the average of values for that indicator from similar countries (e.g. HDI would take the average for low income countries to act as a measure for another low income country with a missing value; similarly PCDI uses the geographic groupings to calculate an average value).

Both CDI and PCDI use the range of data for countries to identify potential data outliers. An outlier is an observation that lies an abnormal distance from other values in a random sample from a population. The outliers often represent a measurement error or a highly atypical country and their inclusion in the statistical analysis may distort the analysis, particularly in the normalisation process of the variables. Each of the indices use statistical analysis techniques associated with average values and standard deviations to determine whether a variable value is outside the normal range and should therefore be excluded.

The availability of data for specific indicators for Scotland is already a consideration in their selection and so there is no need to adopt a missing value methodology. However, it is important to note that neither of these approaches could be adopted as they rely on comparative data being collected on other countries.

Normalisation methods

Different variables represent change using different scales (for example the proportion of all non-EU students from ODA countries and the value of imports from ODA countries). In order to combine these variables it is necessary to normalise (standardise) their scales. Each of the reviewed indeces uses slightly different techniques to achieve this:

  • CDI normalises using a standard statistical technique. Each country's raw score is standardised as a Z score, with a mean of 5 and standard deviation of 1 (so the vast majority of scores are between 3 and 7). Some variable scores are standardised negatively, which means a lower raw score translates into a higher standardised score. This is true of negative factors such as arms exports or greenhouse gas emissions. It is important to recognise that this process is reliant on having multiple observations of the same variable from a range of countries.
  • PCDI uses a Min-Max normalisation that transforms the variable values against a range between 0 and 1 (or between 0 and 100), which imply subtracting the minimum value to the observation and dividing by the range of the values of the variables. The majority of these are set by the observed values of the indicators.[43]
  • HDI uses a similar Min-Max approach with these lower and upper values acting as the "natural zeros" and "aspirational targets," respectively, from which indicator components are standardised. However, in a number of cases, the Min and Max values are set by external research and so do not vary according to the observed values. So, on the HDI income variable the lower value $100 and the upper $75,000 with the minimum being based on research and the upper set as a notional cut-off point. Each dimension (health, education, income) has an index calculated as actual value - minimum value/ maximum value - minimum value.

The issue raised by the min-max method for Scotland is what values determine the minimum and maximum values for each variable. CDI, PCDI and HDI use a mix of observed values (i.e. the minimum and maximum values for the variable among the countries for which data is being collected) and research-based or assumed values. The HDI concept of using an 'aspirational' upper value might be one way forward as this allows for some policy intent and would enable an equivalent process to function when there is only data for one country.

Again, comparative variable values are fundamental to these approaches and would, therefore, not conform to the Scottish Government design criteria for the NPF. Moreover, these approaches imply that a country's index is dependent on the values of comparative countries. Changes in these comparators will alter the value of the index even if the country's indicators values stay the same.

Weighting and Combining indicators

All three indices combine the normalised variables into a single value. Each index uses a slightly different approach:

  • CDI weights variables according to the evidence and their judgement on the contribution to development. These are not applied at the top component level - Aid, Finance, Technology, Environment, Trade, Security and Migration, that are combined as a simple average, but through the combination of sub-components and variables within sub-components.[44]
  • PCDI combines variables using a weighted average of the variables that firstly, contribute to development support or secondly, hinder development support. The weights are the coefficients of a regression model that reflect the contribution of the specific variable to development impact (e.g. GDP growth). These are then combined according to weights ascribed by PCDI that reflect the ability to design and implement policies consistent with development. These weights are determined by the PCDI authors and are set out below.
Economic Component Social Component Global Component Environmental Component Industry and infrastructure Component
Assigned Weight 3 1 2 3 1
  • HDI adopts a two-stage process. Firstly, the individual indicators are combined using either arithmetic or logarithmic averages. Secondly, the resultant domain indices are combined using a geometric average. The advantage of a geometric average is that the geometric mean does not allow significant changes in one indicator to dominate the index as a whole and so measures with very different scales can be combined.

In essence then, both CDI and PCDI contain an element of expert judgement in the application of the relative importance of different variables on the development process. In relation to the weights applied to reflect the ability to design and implement policies consistent with development, these are predominately based on expert assessment.

This suggests that the most straightforward approach is that the NPF indicator takes a year-on-year change in the combined indicator components as a measure of progress. This would work by the first year of the NPF indicator having a value of 100 and then subsequent measures would move in line with percentage change on this baseline value - for example, a 4% increase would provide a value of 104 and a 4% decline 96%.

Finally, this leaves the question of whether the individual pathways in the indicator should be given different weight in calculating the overall contribution to the development of other nations. Both CDI and PCDI have adopted expert judgement in providing differential weights to their elements. These have been criticised in the literature as being opaque and difficult to interpret. We recommend that no weighting be applied to the different pathways in the indicator at this stage, simply because we can find no robust evidence to suggest that any one element is more important than another. Should such evidence become available in the future then this recommendation should be revisited.

Table D.2: Proposed indicators and data sources for the NPF indicator Contribution of development support to other nations

Pathway - What this means for contribution to development of other nations

Suggested Indicator components

Pros

Cons

Data Availability

Inclusion/ exclusion

Reason

We work with partners to build capacity and engage in dialogue on development and human rights

Scotland's connectedness to ODA recipient countries

Simple quantitative measure that can aggregate Scotland's different types of connections to ODA recipience countries

Narrow input measure

Yes

Include

Central to Scottish contribution to ID process

Civic/ Citizenship engagement on International Development

Wider assessment of Scottish engagement

May not conform to expectations e.g. ScotCen report on attitudes to migrants

Need to add questions in Social Attitudes Survey.

Exclude

Would require additional questions in the Social Attitudes Survey and there is no resource for this

Value contribution of individual expertise

Capture scale of investment

Will not capture outputs/ outcomes which may be unrelated to scale of input

No current indicator but discussions with CGD

Exclude

No single measure identified but may be possible to include more specific measures for health, education, justice

School involvement in international learning

Take up of international development learning in schools

An input measure not capturing outputs on attitudes

Unclear whether all schools are required to deliver rendering take-up measure meaningless

Exclude

All schools required to deliver so measure is meaningless

CPD impact on domestic service

Real benefit from expertise/ volunteer programmes

Not measurable in a single indicator

Attributing change through a single indicator a challenge

Exclude

Not measurable in a single indicator

We support migrant and asylum seeker populations coming to Scotland

CDI indicator - integration policies (MIPEX)

Standard set of indicators

Need to rate answers to a number of qualitative questions. May not change

Available but need resource to undertake annual assessment

Exclude

Underlying criteria for MIPEX relate to UK policy choices not Scotland's.

Asylum seekers settled in Scotland per 100,000 population

Standard measure

Not clear what the additional impact of more positive migrant welcome has on UK determined numbers

Scotland may already be high and may have limited room for improvement

Include

Simple measure of Scotland's attractiveness to asylum seekers

We welcome students from developing countries to our educational institutions

HE Students from DAC Least Developed Countries / Total Non-EU Students

Definitive measure of engagement in ODA countries

HE providers only as SFC FE data not available by individual country.

Annual Data for HE providers in Scotland annually Academic year 2017/18 Scotland 2.7%

Include

Direct measure of Scotland's HE provision to ODA countries

We support other nations in humanitarian emergencies. Reducing humanitarian needs

Humanitarian emergency spend

Simple measure of contribution

Financial input to support not a reflection of PCD

Accessible from SG accounts but this would not include donations by Scottish people

Exclude

Financial input to support not a reflection of PCD and other funding explicitly excluded from measures ent

Contribution of Scottish people to DEC appeals per capita

Measure of public engagement in development

May not be accessible

Data held by DEC unpublished

Exclude

DEC data a measure of public engagement

We avoid contributing to climate change and environmental damage internationally

% of total waste exported (SEPA)

Direct measure of Scotland's recycling capacity

Time lag in availability

SEPA data 14.4% (2016)

Include

Direct measure of Scotland's ability

Metric tonnes of carbon per person

Simple measure of Scotland's carbon credentials

Does not capture carbon in imports

Annual update available

Exclude

Already captured in NPF

Ecological footprint of imports (gha per person)

Direct measure of Scotland's carbon impact overseas

None

Already reported in annual GHG reports

Exclude

Useful measure but unclear how this is amenable to Scottish policy action

We trade and invest fairly

Value of imports from ODA countries

Direct measure of the value of trade

Not currently available

No current measure of imports

Include

Direct measure of economic engagement with ODAs

Agricultural subsidies

Agricultural subsidies a key measure of fair access for ODA countries

Narrow and do not necessarily relate to export sectors of ODA countries

SG Budget accounts

Include

Devolved responsibility

Value of arms exports

Frequently used measure of do no harm

No Scotland-level disaggregated data currently available

Exclude

Not currently possible to measure Scotland's contribution to arms component manufacture

We support social enterprise, investment, innovation internationally

SDI R&D Funding for Innovation Projects in ODA countries

Measure of innovation support in ODA countries

Broad range of activity to capture in single indicator

SDI/ SE tbc

Exclude

No data collated on this activity

We work to reduce poverty

We promote equality and human rights

Variation rate of the Gini Index pre and post taxes and transfers

Already an NPF measure

Unclear what is the link to ID

Cross-reference to NPF

Exclude

Unclear what is the link to Beyond aid agenda

Direct measure of the targeting of support on the most vulnerable

% of international development funding targeted at vulnerable groups

Direct input measure

Only measures input

SG data on annual spent on IDF projects: Percentage of total IDF spending that the sum of projects that report or should report on protected characteristics represent.

Include

We promote knowledge exchange and share the experience and expertise of our public, private and community sectors

Value of R&D contracts in HEIs with ODA-eligible nations

A direct measure of engagement

Only partial capture of range of activity

UKRI Gateway to Research database

Include

UKRI Gateway to Research measure of research project spend involving partners in ODA countries

We work to improve health outcomes

PCDI - S2 Health life expectancy

No data currently available

Exclude

Data

NPF Health Risk Behaviours

No data currently available

Exclude

Data

We support climate change mitigation/adaptation and environmental protection/restoration

Value of the Low Carbon and Renewable Energy Economy (LCREE) in Scotland £11.1bn in 2017.

Reflects Scotland's renewable expertise

None

ONS primary measure of the importance of the low carbon economy based on a national survey of just under 24,000 businesses. Latest survey 2018 for 2017 with annual data back to 2015. Published Jan 2019.

Include

Reflects Scotland's renewable expertise

Energy produced from renewable sources

Reflects specific expertise in wind power

Duplicates above indicator

Cross-reference NPF

Exclude

Duplicates above indicator

Water management

Covers different aspects of Scottish expertise

Unclear of role in ID support

No measure identified

Exclude

Unclear of role in ID support

We support fairness under the law

No of ODA countries supported by Police Scotland

Simple direct measure

Simple measure of input

No data currently available

Exclude

Data

No of prosecutions arising from bi-lateral investigations

Demonstrates impact of ID work on justice in Scotland

Narrow measure

Not separately identified in crime statistics

Exclude

Not separately identified in crime statistics


Contact

Email: socialresearch@gov.scot