International Development - national indicator development: research report
The report outlines the research commissioned for the development of the indicator ‘Contribution of development support to other nations’ that forms part of the National Outcome ‘We are open, connected and make a positive contribution internationally’ in the refreshed National Performance Framework
Annex D: Process for Selecting Indicator components
Assessing data availability for the indicator components
A primary constrain on the 25 suggested indicator components is whether there are robust and accessible data sources that are regularly updated so change over time can be tracked. We also needed to consider the design criteria set out for the development of the NPF indicator, in particular:
- A measure for Scotland, not just the Scottish Government.
- Largely focused on those policy areas where Scotland has devolved responsibility.
- Based on transparent and international data sources that can be updated in a timely fashion and with minimal cost, so that Scotland's progress can be assessed on a regular basis.
- Provide an indicator for Scotland that can be compared over time and is not dependent on comparison with other countries.
- Be a cardinal indicator where changes in value relate as directly as possible to changes in policy coherence. There is limited value in selecting an indicator that is not dynamic or is already at a high level and has limited capacity to change.
- Wherever possible not replicate an indicator already part of the NPF.
Discussions with the Scottish Government also confirmed that indicators that required a series of judgements on policy characteristics (such as the MIPEX indicators of migration policy) would also be excluded as these require expert judgement, be potentially open to challenge and require significant resources to update on an annual or bi-annual basis.
The following table sets out the reasons for inclusion or exclusion of the proposed indicator components in more detail.
Measure |
Reasons for inclusion/ exclusion |
Comments |
|---|---|---|
Scotland's connectedness to ODA recipient countries |
Included |
Importance of partnership working stressed by most stakeholders but significant challenges in agreeing a simple indicator and securing data without undertaking a bespoke survey |
Civic/ Citizenship engagement on International Development |
Not available for Scotland and would require additional questions to be added to the Social Attitudes Survey and so cost rules out inclusion. |
Important measure but obtaining a robust measure for Scotland would be expensive |
Value contribution of individual expertise |
Expert contribution vital to policy coherence but no single measure can capture range of inputs |
A measure that has challenged existing indices such as CDI. Cost of collection a major barrier. |
School involvement in international learning |
Important measure of engaging Scottish young people in ID issues |
All schools required to participate in activity so measure of engagement is meaningless. Cost of bespoke survey of young people's attitudes rules out indicator of change in perceptions. |
CPD impact on domestic service |
Number of services engaged in partnership work highlighted the impact on individuals who volunteer/ work as experts who then re-apply this knowledge to their work in Scotland |
Challenge to represent such impacts in a single measure across a wide range of service areas. Would require a bespoke survey and so ruled out on cost grounds |
CDI indicator - integration policies (MIPEX) |
International set of criteria to assess migration policies' impact on integration of migrants/ asylum seekers |
Excluded on grounds that it requires expert judgement of policies many of which are reserved. |
Asylum seekers settled in Scotland per 100,000 population |
Included: Broad measure of Scotland's openness to a diverse and inclusive society |
Included as data is available from National Statistics. |
HE Students from DAC Least Developed Countries / Total Non-EU Students |
Included: Core measure of engagement in HE. HEIs stressed importance of diverse student body for future income streams from foreign students. Wider point that Scottish alumni help 'soft power' and partnership working. Also a common measure in other nation's assessment of policy coherence. |
Included as data available from HESA. |
Contribution of Scottish people to DEC appeals per capita |
Measure of public engagement in humanitarian issues especially countries facing emergencies |
Robust data for Scotland is not available and focus on DEC appeals mean that the measure would be partial. Exclude. |
% of total waste exported |
Included: Measure of Scotland's ability to address its own waste without relying on other countries to pick up the burden |
SEPA measure for proportion of waste exported but a more appropriate indicator is % treated domestically. Included. |
Metric tonnes of carbon per person |
Measure of Scotland's carbon footprint but already included elsewhere in NPF |
Excluded to avoid duplication with other NPF measure |
Ecological footprint of imports (gha per person) |
Measure of Scotland's carbon footprint but already included elsewhere in NPF |
Excluded to avoid duplication with other NPF measure |
Value of imports from ODA countries |
Included: Direct measure of the degree to which Scotland provides access to ODA products |
Included. Data on goods imports by country available from HMRC. |
Value of Agricultural subsidies |
Included: Key measure from both CDI and PCDI indices and devolved responsibility |
Included/ Annual report by Scottish Government. |
SDI R&D Funding for Innovation Projects in ODA countries |
Unable to establish the range of activity included in this pathway |
Excluded no data available |
Variation rate of the Gini Index pre and post taxes and transfers |
Unclear how this measure in Scotland would impact on policy coherence |
Excluded. Measure of income disparities already included in NPF |
% of international development funding targeted at vulnerable groups |
Included: Measure of the concentration of aid expenditure on vulnerable groups |
Include. Data from Scottish Government |
Value of R&D contracts in HEIs with ODA-eligible nations |
Included: Measure of the research collaboration with institutions based in ODA countries. Reflecting the range of projects and collaboration undertaken by Scottish HEIs |
UKRI Gateway to Research data covers all research councils and GCRF and Newton Funds |
PCDI - S2 Health life expectancy |
Measure tbc |
|
Value of the Low Carbon and Renewable Energy Economy (LCREE) in Scotland |
Included: Captures the extent of Scottish expertise in renewable technologies. |
Included. Large scale survey undertaken biennially provides robust estimate for Scotland. |
Energy produced from renewable sources |
Already included elsewhere in NPF |
Excluded as duplicated elsewhere. |
Water management |
Limited ID activity so measure excluded |
|
No of ODA countries supported by Police Scotland |
Simple measure that captures the range of work undertaken by Police Scotland |
Include, more complex measures do not have the source data to support their inclusion |
No of prosecutions arising from bi-lateral investigations |
Key part of PS activities is joint criminal investigations e.g. modern slavery targeting criminals in both Scotland and ODA countries |
Exclude no source data |
Creating a composite indicator for the NPF
The ultimate objective of this assignment has been to identify a number of relevant indicators that together provide a practical measure of Scotland's contribution to the development of other nations. For the purposes of the National Performance Framework (NPF), this measure needs to be a single composite indicator. So the final stage of this process is to determine how this should best achieved using international best practice.
This review has looked closely at the design and methodology underpinning two indices that measure the commitment of countries to international development. The Center for Global Development Contribution to Development Index (CDI) and the Policy Coherence for Development Index (PCDI) both contain indicators with relevance to Scotland.
The third major measure of international development is the United Nation's Human Development Index (HDI) used by the United Nations Development Programme. The HDI is an index of life expectancy, education and per capita indicators, which are used to rank countries into four tiers of human development (Very high, High, Medium and Low).These indicators are primarily focused on measurement of development rather than the various conditions that promote the beyond aid agenda and so are less relevant to Scotland's contribution to the development of other nations.
Although it has not been possible to adopt any of the available international methodologies - CDI, PCDI the HDI - in their entirety, we have drawn on their respective approaches to developing an index to ensure that the proposed indicator builds on existing international best practice.
There is no standard methodology for combining different performance measures into a single index. There are, however, a number of stages to the process common to these three main indices. These are:
- Techniques for addressing any missing values and data outliers
- Standardised performance measures to be able to combine values so that they reflect degree of change
- Combine these using weights to reflect relative importance in contribution to international development
- The methodologies adopted for each index are reviewed and assessed for their relevance to the design criteria for the NPF indicator in turn below.
Treatment of missing values and data outliers
The three indices have a similar two-stage approach to the treatment of missing values:
- If the majority of data points have missing values, then the country in question is excluded from the analysis entirely. CDI for example focuses on the G20 countries on the grounds that most data is available for this group.
- For single (or a small number of indicators) missing data points often use a proxy value - most often the average of values for that indicator from similar countries (e.g. HDI would take the average for low income countries to act as a measure for another low income country with a missing value; similarly PCDI uses the geographic groupings to calculate an average value).
Both CDI and PCDI use the range of data for countries to identify potential data outliers. An outlier is an observation that lies an abnormal distance from other values in a random sample from a population. The outliers often represent a measurement error or a highly atypical country and their inclusion in the statistical analysis may distort the analysis, particularly in the normalisation process of the variables. Each of the indices use statistical analysis techniques associated with average values and standard deviations to determine whether a variable value is outside the normal range and should therefore be excluded.
The availability of data for specific indicators for Scotland is already a consideration in their selection and so there is no need to adopt a missing value methodology. However, it is important to note that neither of these approaches could be adopted as they rely on comparative data being collected on other countries.
Normalisation methods
Different variables represent change using different scales (for example the proportion of all non-EU students from ODA countries and the value of imports from ODA countries). In order to combine these variables it is necessary to normalise (standardise) their scales. Each of the reviewed indeces uses slightly different techniques to achieve this:
- CDI normalises using a standard statistical technique. Each country's raw score is standardised as a Z score, with a mean of 5 and standard deviation of 1 (so the vast majority of scores are between 3 and 7). Some variable scores are standardised negatively, which means a lower raw score translates into a higher standardised score. This is true of negative factors such as arms exports or greenhouse gas emissions. It is important to recognise that this process is reliant on having multiple observations of the same variable from a range of countries.
- PCDI uses a Min-Max normalisation that transforms the variable values against a range between 0 and 1 (or between 0 and 100), which imply subtracting the minimum value to the observation and dividing by the range of the values of the variables. The majority of these are set by the observed values of the indicators.[43]
- HDI uses a similar Min-Max approach with these lower and upper values acting as the "natural zeros" and "aspirational targets," respectively, from which indicator components are standardised. However, in a number of cases, the Min and Max values are set by external research and so do not vary according to the observed values. So, on the HDI income variable the lower value $100 and the upper $75,000 with the minimum being based on research and the upper set as a notional cut-off point. Each dimension (health, education, income) has an index calculated as actual value - minimum value/ maximum value - minimum value.
The issue raised by the min-max method for Scotland is what values determine the minimum and maximum values for each variable. CDI, PCDI and HDI use a mix of observed values (i.e. the minimum and maximum values for the variable among the countries for which data is being collected) and research-based or assumed values. The HDI concept of using an 'aspirational' upper value might be one way forward as this allows for some policy intent and would enable an equivalent process to function when there is only data for one country.
Again, comparative variable values are fundamental to these approaches and would, therefore, not conform to the Scottish Government design criteria for the NPF. Moreover, these approaches imply that a country's index is dependent on the values of comparative countries. Changes in these comparators will alter the value of the index even if the country's indicators values stay the same.
Weighting and Combining indicators
All three indices combine the normalised variables into a single value. Each index uses a slightly different approach:
- CDI weights variables according to the evidence and their judgement on the contribution to development. These are not applied at the top component level - Aid, Finance, Technology, Environment, Trade, Security and Migration, that are combined as a simple average, but through the combination of sub-components and variables within sub-components.[44]
- PCDI combines variables using a weighted average of the variables that firstly, contribute to development support or secondly, hinder development support. The weights are the coefficients of a regression model that reflect the contribution of the specific variable to development impact (e.g. GDP growth). These are then combined according to weights ascribed by PCDI that reflect the ability to design and implement policies consistent with development. These weights are determined by the PCDI authors and are set out below.
| Economic Component | Social Component | Global Component | Environmental Component | Industry and infrastructure Component | |
|---|---|---|---|---|---|
| Assigned Weight | 3 | 1 | 2 | 3 | 1 |
- HDI adopts a two-stage process. Firstly, the individual indicators are combined using either arithmetic or logarithmic averages. Secondly, the resultant domain indices are combined using a geometric average. The advantage of a geometric average is that the geometric mean does not allow significant changes in one indicator to dominate the index as a whole and so measures with very different scales can be combined.
In essence then, both CDI and PCDI contain an element of expert judgement in the application of the relative importance of different variables on the development process. In relation to the weights applied to reflect the ability to design and implement policies consistent with development, these are predominately based on expert assessment.
This suggests that the most straightforward approach is that the NPF indicator takes a year-on-year change in the combined indicator components as a measure of progress. This would work by the first year of the NPF indicator having a value of 100 and then subsequent measures would move in line with percentage change on this baseline value - for example, a 4% increase would provide a value of 104 and a 4% decline 96%.
Finally, this leaves the question of whether the individual pathways in the indicator should be given different weight in calculating the overall contribution to the development of other nations. Both CDI and PCDI have adopted expert judgement in providing differential weights to their elements. These have been criticised in the literature as being opaque and difficult to interpret. We recommend that no weighting be applied to the different pathways in the indicator at this stage, simply because we can find no robust evidence to suggest that any one element is more important than another. Should such evidence become available in the future then this recommendation should be revisited.
Pathway - What this means for contribution to development of other nations |
Suggested Indicator components |
Pros |
Cons |
Data Availability |
Inclusion/ exclusion |
Reason |
|---|---|---|---|---|---|---|
We work with partners to build capacity and engage in dialogue on development and human rights |
Scotland's connectedness to ODA recipient countries |
Simple quantitative measure that can aggregate Scotland's different types of connections to ODA recipience countries |
Narrow input measure |
Yes |
Include |
Central to Scottish contribution to ID process |
Civic/ Citizenship engagement on International Development |
Wider assessment of Scottish engagement |
May not conform to expectations e.g. ScotCen report on attitudes to migrants |
Need to add questions in Social Attitudes Survey. |
Exclude |
Would require additional questions in the Social Attitudes Survey and there is no resource for this |
|
Value contribution of individual expertise |
Capture scale of investment |
Will not capture outputs/ outcomes which may be unrelated to scale of input |
No current indicator but discussions with CGD |
Exclude |
No single measure identified but may be possible to include more specific measures for health, education, justice |
|
School involvement in international learning |
Take up of international development learning in schools |
An input measure not capturing outputs on attitudes |
Unclear whether all schools are required to deliver rendering take-up measure meaningless |
Exclude |
All schools required to deliver so measure is meaningless |
|
CPD impact on domestic service |
Real benefit from expertise/ volunteer programmes |
Not measurable in a single indicator |
Attributing change through a single indicator a challenge |
Exclude |
Not measurable in a single indicator |
|
We support migrant and asylum seeker populations coming to Scotland |
CDI indicator - integration policies (MIPEX) |
Standard set of indicators |
Need to rate answers to a number of qualitative questions. May not change |
Available but need resource to undertake annual assessment |
Exclude |
Underlying criteria for MIPEX relate to UK policy choices not Scotland's. |
Asylum seekers settled in Scotland per 100,000 population |
Standard measure |
Not clear what the additional impact of more positive migrant welcome has on UK determined numbers |
Scotland may already be high and may have limited room for improvement |
Include |
Simple measure of Scotland's attractiveness to asylum seekers |
|
We welcome students from developing countries to our educational institutions |
HE Students from DAC Least Developed Countries / Total Non-EU Students |
Definitive measure of engagement in ODA countries |
HE providers only as SFC FE data not available by individual country. |
Annual Data for HE providers in Scotland annually Academic year 2017/18 Scotland 2.7% |
Include |
Direct measure of Scotland's HE provision to ODA countries |
We support other nations in humanitarian emergencies. Reducing humanitarian needs |
Humanitarian emergency spend |
Simple measure of contribution |
Financial input to support not a reflection of PCD |
Accessible from SG accounts but this would not include donations by Scottish people |
Exclude |
Financial input to support not a reflection of PCD and other funding explicitly excluded from measures ent |
Contribution of Scottish people to DEC appeals per capita |
Measure of public engagement in development |
May not be accessible |
Data held by DEC unpublished |
Exclude |
DEC data a measure of public engagement |
|
We avoid contributing to climate change and environmental damage internationally |
% of total waste exported (SEPA) |
Direct measure of Scotland's recycling capacity |
Time lag in availability |
SEPA data 14.4% (2016) |
Include |
Direct measure of Scotland's ability |
Metric tonnes of carbon per person |
Simple measure of Scotland's carbon credentials |
Does not capture carbon in imports |
Annual update available |
Exclude |
Already captured in NPF |
|
Ecological footprint of imports (gha per person) |
Direct measure of Scotland's carbon impact overseas |
None |
Already reported in annual GHG reports |
Exclude |
Useful measure but unclear how this is amenable to Scottish policy action |
|
We trade and invest fairly |
Value of imports from ODA countries |
Direct measure of the value of trade |
Not currently available |
No current measure of imports |
Include |
Direct measure of economic engagement with ODAs |
Agricultural subsidies |
Agricultural subsidies a key measure of fair access for ODA countries |
Narrow and do not necessarily relate to export sectors of ODA countries |
SG Budget accounts |
Include |
Devolved responsibility |
|
Value of arms exports |
Frequently used measure of do no harm |
No Scotland-level disaggregated data currently available |
Exclude |
Not currently possible to measure Scotland's contribution to arms component manufacture |
||
We support social enterprise, investment, innovation internationally |
SDI R&D Funding for Innovation Projects in ODA countries |
Measure of innovation support in ODA countries |
Broad range of activity to capture in single indicator |
SDI/ SE tbc |
Exclude |
No data collated on this activity |
We work to reduce poverty We promote equality and human rights |
Variation rate of the Gini Index pre and post taxes and transfers |
Already an NPF measure |
Unclear what is the link to ID |
Cross-reference to NPF |
Exclude |
Unclear what is the link to Beyond aid agenda Direct measure of the targeting of support on the most vulnerable |
% of international development funding targeted at vulnerable groups |
Direct input measure |
Only measures input |
SG data on annual spent on IDF projects: Percentage of total IDF spending that the sum of projects that report or should report on protected characteristics represent. |
Include |
||
We promote knowledge exchange and share the experience and expertise of our public, private and community sectors |
Value of R&D contracts in HEIs with ODA-eligible nations |
A direct measure of engagement |
Only partial capture of range of activity |
UKRI Gateway to Research database |
Include |
UKRI Gateway to Research measure of research project spend involving partners in ODA countries |
We work to improve health outcomes |
PCDI - S2 Health life expectancy |
No data currently available |
Exclude |
Data |
||
NPF Health Risk Behaviours |
No data currently available |
Exclude |
Data |
|||
We support climate change mitigation/adaptation and environmental protection/restoration |
Value of the Low Carbon and Renewable Energy Economy (LCREE) in Scotland £11.1bn in 2017. |
Reflects Scotland's renewable expertise |
None |
ONS primary measure of the importance of the low carbon economy based on a national survey of just under 24,000 businesses. Latest survey 2018 for 2017 with annual data back to 2015. Published Jan 2019. |
Include |
Reflects Scotland's renewable expertise |
Energy produced from renewable sources |
Reflects specific expertise in wind power |
Duplicates above indicator |
Cross-reference NPF |
Exclude |
Duplicates above indicator |
|
Water management |
Covers different aspects of Scottish expertise |
Unclear of role in ID support |
No measure identified |
Exclude |
Unclear of role in ID support |
|
We support fairness under the law |
No of ODA countries supported by Police Scotland |
Simple direct measure |
Simple measure of input |
No data currently available |
Exclude |
Data |
No of prosecutions arising from bi-lateral investigations |
Demonstrates impact of ID work on justice in Scotland |
Narrow measure |
Not separately identified in crime statistics |
Exclude |
Not separately identified in crime statistics |
Contact
Email: socialresearch@gov.scot
There is a problem
Thanks for your feedback