Publication - Research and analysis

International Development - national indicator development: research report

Published: 6 Nov 2020

The report outlines the research commissioned for the development of the indicator ‘Contribution of development support to other nations’ that forms part of the National Outcome ‘We are open, connected and make a positive contribution internationally’ in the refreshed National Performance Framework

87 page PDF

1.2 MB

87 page PDF

1.2 MB

Contents
International Development - national indicator development: research report
Annex C: Review of Policy Coherence for Development Indicators

87 page PDF

1.2 MB

Annex C: Review of Policy Coherence for Development Indicators

Background

PCD is an approach and policy tool for integrating the economic, social, environmental and governance dimensions of sustainable development at all stages of domestic and international policy making. It is the aim of Policy Coherence for Development to make foreign relations to be as ecologically, economically and socially coherent as possible and thereby to make international co-operation for international development more effective.[36] The literature on Policy Coherence for Development (PCD), and more recently Policy Coherence for Sustainable Development (PCSD), is extensive with the earliest references back to 2005 being developed by the OECD and European Commission among other institutions.

PCD has been discussed in relation to the concept of Global Public Goods (GPGs). GPGs are in principle available to everyone and each country has an interest in contributing to their promotion. Examples might include a fair, robust and market- orientated trading system for goods and services or climate stability.

PCSD stems from the 2030 Agenda and the Addis Ababa Action Agenda. In addition, in the case of PCSD, given that the SDGs are universal, policy makers have to secure broader policy coherence by pursuing multiple goals globally. The OECD has defined PCSD as an approach and policy tool relevant to all countries, to be used at the domestic and international levels of policy making.[37]

The European Commission in collaboration with the Member States, identifies five broad PCD priority areas: (i) trade and finance, (ii) climate change, (iii) food security, (iv) migration and (v) security.

Measuring the impact of PCD/ PCSD

There are a number of PCD/ PCSD frameworks in the literature but in general these focus on the policy procedures in place to ensure policy coherence for development but that the vast majority do not (yet) specify performance measures or indicators that will assess a country's progress and impact in implementing PCD/PCSD. An issue throughout is what has been set down in policy intent and what has been implemented to date.

In part this reflects the different mechanisms that countries have defined to drive their PCD process. These include:

  • Formal parliamentary committees or formal responsibility to consider PCD/PCSD in policy development
  • Annual reviews of PCD led by civil society organisations who report to government on progress and areas for improvement
  • Ex-post reviews of evaluations of international development initiatives to consider how other national policies have supported or hindered their effectiveness

A recent review of the use of PCD indicators by selected EU member states highlights that those countries who have a process do not necessarily then have official performance indicators that track PCD/PCSD progress.

PCD Mechanism 'Official' cross-government PCD indicators
Belgium Yes Not yet
Denmark Yes Yes
Finland Yes Not yet
Germany Yes Not yet
Ireland Yes Not yet
Luxembourg Yes Not yet
Netherlands Yes Yes
Sweden Yes Yes

Source: ECDPM Discussion paper 171, Use of PCD indicators by a selection of EU Member States, Jan 2015

A number of reviews have highlighted that there is a need to strengthen and make explicit the causal logic chain between PCD/PCSD.[38] An evaluation of the impact of PCD activity across member states similarly concluded that more work was required to establish a causal link between PCD/PCSD reviews of policies and the consequent impacts that might then be measured by indicators.[39] The overall conclusion of the evaluation was that while changes in policies could be evidenced, it was not yet possible to ascribe an impact of these changes in quantitate terms on the international development process.

Determining expectation for impact is clearly a crucial stage in the process for defining relevant and appropriate indicators that this may need to be revisited with Scottish Government colleagues.

Example of a logic chain linking PCD/ PCSD to indicators: Sweden
This image illustrates examples of chains of causality in the area of trade and finance in Sweden based on the ECDPM Discussion paper 171

Source: ECDPM Discussion paper 171, Use of PCD indicators by a selection of EU Member States, Jan 2015

In their seminal paper, King and Matthews (2012) also identified the challenges that can arise by a lack of precision in measure definition. This , for example, can be at the level of using concepts as indicators - in the Swedish example above, the indicators are not indicators as they have no defined scale and could not be quantified as they stand. These criticisms are repeated by ECDPM (2015) who found that across the member states reviewed "different logical frameworks mix up objectives, targets, actions and indicators."[40]

King and Matthews (2012) proposed that indicators be categorised to avoid this confusion:

  • Outcome indicators: these focus on outcomes such as socio-economic variables (e.g. income per capita, student enrolment rates, etc.).
  • Policy outputs: that capture the changes in policies designed to be more coherent with development. E.g. the level of tuition fees for students from developing countries, food tariffs for imports from developing countries, etc.
  • Policy inputs: that should be used where it may be difficult to summarise the output of a policy into a single indicator, e.g. the proportion of funding that supports the primary objectives of the developing country.
  • Policy stance indicators: relate to treaty or protocol agreements e.g. the signing of international agreements on financial transparency, etc.

They also proposed a set of criteria for the selection of PCD indicators:

  • Transparency: Can a layperson understand what is happening? Does the index hide or reveal facts?
  • Policy relevance: Does the indicator/index relate to important societal debates?
  • Analytical soundness: Does the indicator measure the problem, or rather something else?
  • Responsiveness: Does a politician have any chance to improve the indicator/index?
  • Time horizon: How quickly can results be expected? Non-ambiguity of "welfare message": Does everybody agree that "more is better", or vice versa?
  • Accountability: Does the indicator/index point at those who should be held responsible?
  • Robustness/ independence of assumptions: Could the value of the indicator change drastically by fumbling with some assumptions?
  • Measurability, data availability: Will we see comparable figures in the next ten years?

Implications for selecting indicators for Scotland's NPF

The criteria set out above set a relatively high bar for the current practice on measuring PCD/PCSD using indicators. While we have found a wide range of reports on PCD/PCSD frameworks and procedures, only two provide a coherent approach to setting out baskets of indicators that could be used to track progress. These are set out below.

King and Matthews indicators for PCD in Ireland

Using their own design criteria, King and Matthews set out a set of indicators that they recommended to the Irish Government as a method to track their PCD progress. These reflect the key areas of Irish Government's interests and in many ways are similar to the components used in CGD CDI.

Trade Policy Indicators

  • T.1.1 Average Tariffs on Manufacturing Imports, 2010.
  • T.1.2 Share of Duty-Free Imports, 2009.
  • T.1.3 Trade Restrictiveness Indicators for Manufactured Goods, 2009.
  • T.1.4 Trends in Import Growth Rates, 2007-2009.
  • T.2.1 EU and Irish Trade Preference Utilisation, 2009.
  • T.3.1 ODA Expenditure on Trade Policies & Regulations, % of 2008 GDP.

Agriculture policy Indicators

  • A.1.1 Average Tariff on Agricultural Imports, 2010.
  • A.1.2 National Levels of Market Price Support, 2009.
  • A.1.3 Trade Restrictiveness Indices for Agricultural Goods, 2009.
  • A.1.4 Growth in Agricultural Imports from Developing Countries, 2007-2009.
  • A.2.1 Trade-distorting Support, 2007.
  • A.3.1 Agricultural ODA Expenditure, 2008.

Fisheries Policy Indicators

  • F.1.1 Ireland's Participation in International Agreements on Fisheries Protection, 2010.
  • F.1.2 DAC Country Compliance Scores for FAO (UN) Code of Conduct for Responsible Fisheries, 2006.
  • F.2.1 Average MFN and Applied Tariffs on Fish and Fish Products, 2008.
  • F.3.1 Government Financial Transfers to Fisheries Sector, as a % of the Total Landed Value, 2007.
  • F.4.1 Ireland's Industrial Pelagic Fishing Possibilities in Morocco, 2007-2011.
  • F.4.2 FAO (UN) Code of Conduct for Responsible Fisheries, Compliance Scores for FPA Countries, 2006.
  • F.4.3 Marine Protected Areas, % of Country's Exclusive Economic Zone, 2010.
  • F.4.4 Ireland's Contribution towards Fisheries Capacity Building in Developing Countries, 2008.

Migration Indicators

  • M.1.1 Non-DAC Inflow as a Percent of Total Population, 2008.
  • M.1.2 Number of Residents in Ireland from Different Regions of the World, 2006.
  • M.1.3 Country of Origin of African Migrants into Ireland, 2006.
  • M.2.1 Support for Remittances to Developing Countries, 2010.
  • M.3.1 Total UNHCR Population of Concern + Applications/ Billion USD of GDP, 2010.
  • M.4.1 Ratio of Tuition Fees for non-DAC students to DAC students and Irish Students, 2004.
  • M.4.2 Proportion of non-DAC (to total) students in tertiary education, 2007.

Environment Indicators

  • E.1.1 Environmental Protection ODA (Commitment), 2008.
  • E.2.1 Average Annual Growth Rate of GHG Emissions/PPP GDP, 1997-2007.
  • E.2.2 Performance in Meeting Kyoto Protocol Targets, 2008.
  • E.2.3 ODA Expenditure on Climate Change, 2008 (Second Rio Marker).
  • E.2.4 ODA Expenditure on Desertification, 2008 (Third Rio Marker).
  • E.3.1 ODA Expenditure on Biodiversity (Disbursement), 2008 (First Rio Marker).
  • E.3.2 Adoption of Convention of Biological Diversity and Related Protocol, 2010.
  • E.4.1 MFN Tariffs on Bioethanol, 2010.
  • E.4.2 Subsidies for Liquid Biofuels (Ethanol and Biodiesel), Most Recent Year.

Finance and Enterprise Policy Indicators

  • FE.1.1 ODA Expenditure on Debt Relief, 2007- 2008.
  • FE.2.1 Existence of Double Taxation Agreements with Irish Aid Priority Countries, 2010.
  • FE.3.1 Level of foreign bribery enforcement in OECD Convention Countries, 2011.
  • FE.4.1 Restrictions on the Flow of Technology to Developing Countries, 2010.

Security Policy Indicators

  • S.1.1 Peacekeeping Contribution, UN-run Operations, Progressively Weighted to the Present, 1993-2009.
  • S.1.2 Peacekeeping Contribution, Non UN-run Operations, Progressively Weighted to the Present, 1993-2009.
  • S.1.3 Expenditure on Security System Management and Reform, 2008.
  • S.1.4 Participation in Four Essential Security International Treaty and Related Policies, 2010.
  • S.2.1 Exports of Major Conventional Weapons, 2008.

Development Aid Indicators

  • DA.1.1 Level of Overseas Aid (ODA), 2010.
  • DA.2.1 Irish Aid Partner Country GNI per capita, 2008.
  • DA.2.2 Governance Quality, Kaufman and Kraay Government Effectiveness Scores, 2009.
  • DA.2.3 Corruption Levels, Kaufman and Kraay Control of Corruption Scores, 2010.
  • DA.2.4 Economic Management Quality, 2010.
  • DA.2.5 Strength of Social Inclusion Policies, 2008.
  • DA.3.1 % of Aid Flows Disbursed for Government Sector, 2007.
  • DA.3.2 ODA Expenditure Lost to Tied Aid, 2009.

Many of the indicator categories cover policy areas that remain reserved matters for Scotland and as noted above, these categories closely reflect the CDI components and so do not directly address a number of the policy themes we are seeking to capture.

PCDI (Policy Coherence for Development Index)

The Policy Coherence for Development Index (PCDI)[41] is a tool designed to measure, evaluate and compare countries' commitment to sustainable, fair and equitable human development. The concept of Policy Coherence for Development (PCD) originally emerged in the early 1990s from the realisation that non-aid policies of donors affect developing countries and should not distract but rather be supportive of international development goals. The PCD concept initially emphasised the responsibility of developed countries to consider the effect on developing countries when formulating domestic policies across different sectors (trade, finance, migration, security, technology, science).

As the concept evolved, the PCDI has been developed to go beyond a 'do no harm' approach, also with a requirement to seek synergies between development co-operation and other policies as well as to correct existing incoherencies.

The PCDI analyses both the policies that make a positive contribution to a country's sustainable development and those that hinder it, not only within that country but also in third countries or on the planet as a whole. The PCDI is divided up into five components: economic, social, global, environmental and production.

Knoll (2014) reviewed 20 different policy domains through four different dimensions: environment, economic, social and political and grouped them together in five components based on their similarities and to provide a categorisation that was more accessible to decision-makers.

Economic Component

  • Fiscal
  • Financial

Social Component

  • Education
  • Health
  • Social Protection
  • Equality
  • Employment
  • Science & technology

Global Component

  • Peace & security
  • Co-operation
  • Justice & human rights
  • Human mobility & migration

Environmental Component

  • Energy
  • Biodiversity
  • Fisheries
  • Rural & agricultural development

Production Component

  • Industry
  • Infrastructure & transport
  • Tourism
  • Urban Planning

Source PCDI Report Chapter 4 p129.

Indicators were selected for each element of the matrix based on data from 234 countries and an initial set of 201 indicators. The removal of missing data reduced the dataset to 133 countries and 133 variables. These were then further reviewed using factor analysis to produce a list of 49 variables for 133 countries with six indicators in the economic component, nineteen in the social, ten in the global, eight in the environmental and six in the production component. These were organised into 31 indicators that promote policy coherence (such as inequality reduction, public spending on social protection and ratification of universal justice treaties) and 18 indicators that are contrary to sustainable development processes (such as school dropout rates, military spending and ecological footprint). The PCDI is an index and in a similar method to CDI, standardises the degree of change across variables and weights their influence before combining them for an overall score.

Of the 49 variables, 18 reflected indicators contrary to sustainable development processes, whereas the other 31 reflected indicators that favoured them.

The preparation of the variables involved the following actions:[42]

  • Grouping of countries: the countries were grouped into 6 groups:
  • Group 1: OECD countries, accession countries and countries with enhanced cooperation;
  • Group 2: South
  • Group 3: Latin America
  • Group 4: Europe and Central Asia
  • Group 5: Sub-Saharan Africa
  • Group 6: Middle -East and North Africa
  • Exclusion of variables with high missing values (>40% and some with >30%) following the priority of each variable and the number of rem each policy.
  • Grouping of categorical variables (1/0) into a scale variable
  • Elimination of variables with high correlations among them variables are related that two or more of them are quantifying the same information, therefore they may reduce the reliability of the index. This may induce a double count in the aggregation step, reducing the reliable the use of statistical methods necessary
  • An analysis of outliers was carried out for each variable with a Boxplot analysis. To perform this analysis, all the variables were reviewed and the outliers that appeared were replaced by another value based on statistical criteria (e.g. the highest non-outlier variable, the median value, etc.) and logical interpretation criteria.
  • A Min-Max normalisation was applied to normalise the variables to follow a range between 0 and 1 (or between 0 and 100)
  • The classification of variables into those which support a country's development and those that hinder a country's development.

The result of this process produced the set of variables outlined in table C.2:

Table C.2: Overview of PCDI variables

Dimension: Economic component

PCDI Variables that contribute:

  • FIS1 Tax revenue (%GDP)
  • FIS3 Variation rate of the Gini Index pre and post taxes and transfers (%)
  • FIS5 Environment protection expenditure (% GDP)

PCDI variables that hinder:

  • F2 Bank assets (%GDP)
  • F5 External service, total debt (TSD,US $ at current prices / Exports of goods and services (US $ at current prices)
  • FIS6 Financial Secrecy Index

Dimension: Social component

PCDI Variables that contribute:

  • EDU5 Survival rate to the last grade of education, both sexes (%)
  • EDU11 Net enrolment rate, primary, gender parity index (GPI)
  • PS1 Public social protection
  • PS5 Share of population above statutory pensionable age receiving an old age pension
  • PS8 Benefits incidence
  • IG5_6_7 Legislation against harassment and against marital rape
  • IG11 Mandatory minimum leave (in calendar days)
  • IG14 Position shown at the initiative of the UN in favour of the LGBT
  • S2 Health life expectancy
  • S3 Total density per 100,000 population: hospitals
  • S11 Improved sanitation facilities with access)
  • CIT6 Enrolment ratio of female with respect to male in tertiary education
  • CIT13 Percentage of graduates from tertiary education who are female (%)

PCDI variables that hinder:

  • EDU2 Rate of out-of school children of primary age, both sexes (%)
  • EDU8 Pupil-teacher ratio in pre-primary education
  • EDU9 Pupil-teacher ratio in primary education
  • EDU14 Repetition rate in primary education (all grades), both sexes (%) Benefits incidence in poorest quintile (%)
  • IG2 Unpaid family workers (% of female employment)
  • EM6 Difference of vulnerable Employment between women and men (%)

Dimension: Global component

PCDI Variables that contribute:

  • J4_5 Legality of homosexuality and of equal marriage
  • J6 Participation in the ratification of international treaties of the UN about human rights (%)
  • J8 Universal jurisdiction
  • J9 Ratification of UN treaties on International Justice
  • J13 Does a woman's testimony carry the same evidentiary weight in court as a man's?
  • J14 Can a married woman convey citizenship to her non-national spouse in the same way as a man?
  • J15 Are married women required by law to obey their husbands?
  • PYS6 International treaties about weapons
  • M4_5 Convention relating to the status of refugees and International Convention on the protection of the Rights of all members of their families
  • C3 Existence of a specific structure of cooperation an appreciation of its political rank

PCDI variables that hinder:

  • PYS1 Military Expenditure (% GDP)
  • PYS3 Military personnel (per 100.000 inhabitants)

Dimension: Environmental component

PCDI Variables that contribute:

  • P2 Artisanal fishing opportunities
  • P4 Clean waters
  • P6 Biodiversity
  • P9 Participation in treaties, conventions and agreements on fishing %

PCDI variables that hinder:

  • DR9 Use of fertilizers
  • B2 Ecological footprint by production (gha per person)
  • EN2 Ecological footprint of imports (gha per person)
  • EN4 Metric tons of carbon dioxide per person

Dimension: Industry and infrastructure component

PCDI Variables that contribute:

  • IT3 Improved water supply (% population with access)
  • IT4 Access to electricity (% of population)
  • IN1 R&D (%GDP)

PCDI variables that hinder:

  • T1 International tourist arrivals (% of the population in the host country)
  • IN5 Annual freshwater withdrawals, industry (% of total freshwater withdrawal)
  • IN8 Difference between male and female employment in the industrial sector (%)

These variables can be mapped to Scotland's five key policy areas. The full mapping is provided in Annex B but in summary the mapped variables comprise:

  • Climate change - variables comprise environmental protection expenditure (% GDP); ecological footprint by production (gha per person); ecological footprint of imports (gha per person); and metric tons of carbon dioxide per person.
  • Equality - a variety of variables are included. These comprise economic variables, the Gini Index, social variables, gender and education; LGBT policies; pension provision and benefit incidence; and global variables in relation to human rights.
  • Education - variables include enrolment rates for males and females and pupil teacher ratios.
  • Determinants of health - variables include healthy life expectancy, hospital density and various environmental components such as clean waters and use of fertilisers.
  • Justice - variables are split between social components; public social protection and legislation against harassment and global components including the ratification of UK treaties on international justice.

Contact

Email: socialresearch@gov.scot