Impact assessment in governments: literature review

4. How effective are impact assessments?

This section considers how impact assessments have changed policies, improved participation and transparency in policy-making, and improved learning and knowledge by policy makers. It also discusses whether impacts assessment are 'worth it' in terms of the costs/resources required.

4.1 Changes to policies

This sub-section is divided by type of impact assessment, since considerable literature exists on how different kinds of impact assessment have changed policies.

Environmental (SEA): Almost all of the literature on the effectiveness of SEA relates to plans and programmes, and is not covered here. The former Dutch 'E-test' of policies was

"used primarily for instrumental purposes and limited in its effectiveness. During the first five years, the E-test has led mainly to the inclusion and highlighting of environment-related information in explanatory memoranda to draft bills. However, this information played only a limited role in policy-making and contributed little to the environmental improvement of draft laws and regulations or to the transparency of this process" (van Dreumel, 2005).

Similarly, focusing on impact assessment to promote sustainable development, Owens et al. (2004) suggest that assessment procedures in general – at least as they were 16 years ago - had failed to live up to widespread expectations of becoming a vehicle for bridging knowledge from different sectors and integrating it into policy-making.

Equality: An Irish 'equality budgeting' pilot programme led to a wide range of changes, including the provision of grants to increase participation of women in sporting activities, a review of apprenticeship programmes with a view to increasing female participation, a gender initiative in awarding Science Foundation Ireland research grants, targeting smoking reduction amongst the less well-off, and support for refugees (Howard, 2019). However, more widely, the OECD (2019) was relatively critical of Ireland's lack of overarching equality strategy, lack of mandatory equality budgeting, lack of consideration of equality issues other than gender, and limited communication of its equality vision to departments and wider stakeholders.

In Northern Ireland, the Policy Appraisal and Fair Treatment initiative was used in the late 1990s as a basis for formal investigations into equality practices, and at least one judicial review relating to religious imbalance in college admissions (CCRU, 1999). A review of 50 cases of US 'environmental justice' analyses, primarily related to projects rather than policies, found that 33 of the 50 cases led to some kind of 'remediation': stopping construction of a proposed project, closing an existing site, achieving a substantial cleanup, substantial financial compensation for damages etc. (Hess and Satcher, 2013).

Health: Gray et al. (2011) analysed 135 HIAs internationally, and overall found little evidence that health issues were incorporated into plans or subsequently implemented. Harris-Roxas et al. (2012) also note that "HIAs are frequently rushed and often conducted after other impact assessment processes, with limited scope for the collection of new data upon which to base an assessment… HIA lacks evidence to demonstrate that it is effective in changing decisions and the implementation of policies".

In contrast, Haigh et al. (2013) and Harris et al. (2013) found that, of 47 New Zealand and Australian HIAs, all demonstrated some evidence of effectiveness, 65% directly led to changes in the policy, and 94% of 45 HIA authors felt that the HIAs had made a difference. Changes made as a result of HIAs included:

• Decisions being changed
• Elements of the HIA being integrated into the proposal
• HIA being used to enforce agreement on monitoring conditions
• HIA being used as a baseline assessment and framework against which progress of the policy is judged
• HIA findings being adopted in principle but needing amendment in order to be enforceable/ implementable (Harris et al., 2013).

The EOHSP (2007) found that all but one of 17 HIA case studies it analysed were in some way effective, either directly, generally or in an opportunistic manner. The report argues that the fact that none of the HIAs resulted in the cancellation of the proposed plan shows that HIA is not intended to hinder policy-making, but rather to help inform decisions with regard to their health effects.

Based on an in-depth analysis of four Dutch HIAs, Bekkers (2007) concluded that because health problems (e.g. obesity at the population level) have many causes and data are lacking about many problems, policy-makers are faced with inconclusive analyses, leading them to question the HIA problem definition and methodology. "HIA is only to a limited extent effective in reframing public policy to integrate health. HIA practices in all cases remain the primary responsibility of the public health sector with limited or hardly no involvement of the other policy sectors addressed" (Bekkers, 2007).

However, like Monteiro et al. (2018), she suggests that this is at least partly the fault of the HIAs themselves rather than of the decision-makers: "there is limited consideration [in HIAs] for 'usefulness' of knowledge to the policymakers addressed, nor for the feasibility of policy alternatives to make problems 'doable'" (Bekkers, 2007).

Interestingly, analysing four Irish HIAs, O'Mullane and Quinlivan (2012) suggest that HIAs are likely to be more effective at the local than the national level, in part because they claim that there is greater transparency and accountability and more consultation at the local level.

Regulatory: Of the RIA literature analysed for this study, that on Ireland seemed to be the most positive, though in a qualified manner:

"The RIA process has succeeded in moving Government Departments away from the traditional 'regulate first' approach and towards having evaluations of different options done before regulatory decisions are made. However, this may not always be the situation. If Departments are merely 'ticking boxes', after legislative solutions have been chosen, it amounts to a little more than a waste of resources – unless the RIA impacts on the evolving legislation" (Ferris, 2017).

Radaelli (2009a) takes a more critical stance, noting that there is limited evidence of RIA informing policy in countries such as the Netherlands, Sweden and the UK. He suggests that RIA is often used as a symbol rather than as a tool for improving policy.

The Netherlands is the only country for which we have found an up-to-date and detailed analysis of its RIA system, although the OECD (2020) suggests that its strengths and weaknesses are replicated in other countries. The Dutch government introduced an integrated assessment framework ('IAK') in 2011, which was a clear improvement on the previous panoply of uncoordinated assessments, but this still has problems:

"The OECD team could find little clear evidence that the IAK is having an impact on decision making within government… Parts of the [IAK] framework are well structured and strongly overseen, in particular the requirement to calculate regulatory costs on business. Furthermore, the political will continues to exist at Cabinet-level and throughout the ministries for an evidence based and transparent decision-making process… However, the current IAK has not become an integral part of governmental decision making process in the manner intended and tends to be produced late in the policy process. Nor is it driving sufficient transparency for stakeholders… it also lacks strong regulatory oversight to ensure regulatory quality and the integration of horizontal objectives" (OECD, 2020).

4.2 Participation and transparency

Even where an impact assessment does not lead to changes in a policy, it can have benefits in terms of improved transparency and accountability of decision-making, increased awareness of the public, and increased trust between stakeholders. These can "go beyond individual processes (and also strictly environmental issues) and can be related to... the broader process of 'democratisation of decision-making'" (Rega and Baldizzone, 2015). This transparency has both benefits and constraints: it helps to ensure that the policy is as 'good' as possible, but it can be used – or, in the ministry's views, abused – to oppose the policy (Turnpenny, 2009). Radaelli (2009b) uses a similar description for Dutch RIAs for which there is no clear output/report: "'Good' for consensus politics in an executive with a skinny majority, electoral volatility and coalition pacts… 'bad' for researchers wanting to score the Dutch RIAs".

Most of the literature on participation in impact assessment relates to plans and projects. The only references to participation in policy assessments we could find were for Ireland. The 2016 Programme for a Partnership Government (2016) promotes public consultations as an important opportunity for the public to have direct input into policy-making, although it does not clearly link this consultation to impact assessment. Early poverty proofing exercises in Ireland were criticised by community and voluntary groups because those who were potentially impacted by a policy were not involved in its proofing. This led to the current approach to poverty impact assessment having consultation as its first step (Johnston, 2017). Ferris (2017) suggests that "consultation is a crucial part of the RIA process" in Ireland and the website https://www.gov.ie/en/consultations/ gives details of consultations carried out by Irish government departments and local authorities.

At the policy level, the same short list of organisations may regularly be consulted, on the assumption that they represent the public interest. These organisations tend to be well resourced, with well-established channels of communication with policy-makers. Smaller groups and individuals may consequently be sidelined (Turnpenny et al., 2009). For US 'environmental justice' cases, Konisky and Reenock (2018) found that state regulatory agencies gave more attention to those communities in which environmental justice advocacy organisations operated, suggesting that such organisations play an important role in ensuring a positive outcome for impact assessments.

4.3 Knowledge and learning

Knowledge and learning is the dimension of impact assessment effectiveness that is most often cited in the literature as working well.Many of the benefits of impact assessment do not accrue to the policy being appraised, but rather through longer-term, indirect changes of beliefs and policy frames (Radaelli, 2009b). Hertin et al. (2009) distinguish between three types of knowledge that can emerge from impact assessment processes:

• Conceptual learning, where the knowledge 'enlightens' policy-makers by providing new information, ideas and perspectives, including challenging existing beliefs and opening opportunities for policy change
• Instrumental learning, where the knowledge is put to use in practice, for instance in the design of policies
• Political use, where the knowledge is put forward to attain political objectives, for instance to provide justification for decisions already taken or disarming an opponent's viewpoint.

In this case, 'instrumental rationality' represents single loop learning (e.g. how do we reduce a policy's health impacts?), whereas 'challenging existing beliefs' represents double loop learning (is this the right approach to policy formulation?) – see Figure 4.1.

Haigh et al. (2013) and Harris et al. (2013) refer to three types of single-loop learning resulting from HIA in New Zealand and Australia: increased knowledge about health and the social determinants of health, better technical skills and knowledge (e.g. use of data, assessment of evidence), and social learning (e.g. new relationships, engagement with other sectors and stakeholders, negotiation skills).

Much of the effectiveness of impact assessment, if carried out well, also involves double-loop learning:

"in order to truly integrate sustainability concerns in policy-making, the underlying challenge is in fact not the quest for instrumental rationality but rather one of policy learning and reframing. For... any new values to become prioritised, a reframing of values and priorities as such is needed. Such a change requires a conceptual learning process modifying existing beliefs and priorities" (Nykvist and Nilsson, 2009).

Harris-Roxas et al. (2012) give an intriguing example of how HIA can lead to consideration of completely new approaches (double-loop learning):

"The opportunity exists to improve HIA through paying greater attention to other forms of alternatives… Knowledge alternatives involve looking at different ways of understanding the issue or problem. For example, where malaria is endemic, the problem can be viewed as being environmental (standing water), social (barrier to use of preventive schemes like bed-nets), economic (lack of access to treatment), or cultural (agricultural practices leading to standing water, or cultural practice leading to proximity to standing water), among many others. Institutional alternatives involve new partnerships or different ways of working at an organisational level, while goal alternatives involve consideration of what is trying to be achieved and whether alternative approaches could be used to achieve those overall goals."

Comparable issues in the countries investigated as part of this research could include responses to childhood obesity or pandemics. In other words, simply looking at a problem from several points of view – a legal requirement of some forms of impact assessment – involves learning.

In the Netherlands, "knowledge on how to calculate regulatory burden is now quite firmly embedded across ministries" (OECD, 2020). In Ireland, poverty proofing has sensitised policy-makers across government about social inclusion and the need to consider the impact of their policies on people living in poverty (Office of Social Inclusion, 2006; Johnston, 2017). Interviewees in England, the Netherlands and Northern Ireland stated that HIA had created stronger health consciousness in decision-makers (EOHSP, 2007), and this is confirmed by Gray et al. (2011), Haigh et al. (2013) and Ward (2006). The EOHSP (2007) also notes that participation in HIA will increase a community's knowledge and ability to control things that influence their health.

Hertin et al. (2009) found few examples of conceptual learning where the assessment fundamentally challenged the problem definition or policy approaches. In contrast in Wales, the Future Generations Commissioner (2017) is explicitly aiming to do this, by challenging the Public Service Bodies (PSBs) that carry out wellbeing assessments to think outside the box: "Despite the evidence in the assessments showing a range of alarming trends, PSBs are only engaging in safe and non-contentious territory. PSBs need to evidence how they are identifying and exploring tensions between different policy issues and priorities to enable an honest discussion about new approaches that need to be taken". In Ireland, the Environmental Protection Agency is taking a similarly proactive stance towards SEA practice.

The literature also highlights the long-term nature of learning. The EOHSP (2007) found that the learning brought about by HIA would have a "lasting effect which will contribute to healthy decision-making in the future". Dunlop and Radaelli (2015) found that RIA's merit over time "is most likely to lay out arguments and evidence that can be used after the decision is taken… and go back to the same issues years later in the context of another [impact assessment]".

Knowledge and learning, in turn, can lead to better impact assessments. A relatively damning WHO (2018) survey of 64 HIA practitioners found that more of them considered HIA to be ineffective than effective (with no clear definition of 'effective'). However, "respondents from countries with a higher experience in HIA, for example, Lithuania and the United Kingdom, rated more in favour of HIA effectiveness than respondents from countries with less experience in HIA".

Some of the learning involved in impact assessment is formal: policy-makers may be required or encouraged to attend courses on impact assessment or related issues. Meetings with stakeholders can increase policy-makers' and the public's knowledge of the policy, impact assessment, and the likely impacts of the policy; and being involved in impact assessment will increase the public's understanding of policy-making and the impacts of policies (OECD, 2007). More informally, discussions between stakeholders can help participants to broaden their view and gain expertise. However, longer-term learning can be limited by staff changes and the consequent loss of institutional memory (Roggeband and Verloo, 2006; Therivel and Gonzalez, 2019).

4.4 Costs v. Benefits of Impact Assessments

Information on the costs v. benefits of policy impact assessments was so patchy and often so dated that we did not feel that it was possible to write cogently about the issue here. That said, some high-profile discussions about 'streamlining' or 'simplifying' assessments (e.g. White House, 2020; MHCLG, 2020) do make this issue pertinent.

EOHSP (2007) lists the costs of some HIAs dated 1994-2004; and Radaelli (2009a) noted that "The UK has been the only country in Europe to insist that benefits of regulatory proposals justify the costs [of RIA]". Key limitations are that policy-level impact assessments are often carried out in-house, and it is difficult to cost this; and policy-makers do not regularly revisit their policies and impact assessments. Furthermore, the costs of impact assessment fall on the policy-maker whereas the benefits are felt more widely. This means surveys of decision-makers probably understate the long-term benefits of policy changes resulting from impact assessments (Therivel and Gonzalez, 2020).

4.5 Conclusion

Impact assessments can lead to improved policies, greater participation and transparency, and particularly improved long-term knowledge and learning. The latter can go well beyond the policy in question and have longer-term benefits for other policies.

However, simply requiring impact assessments to be carried out does not mean that they will be carried out well or, indeed, at all. 'Tactics' are also needed. Berensson and Tillgren (2017) suggest that the use of HIA in Sweden to influence public policy "requires practitioners to be both tactical and technical… HIA must be tactical by taking into account procedural policy-making constraints and focusing on policy actors' values, interests and learning, and ultimately, institutional rules, procedures and mandates". In the context of US environmental justice, Hess and Satcher (2013) similarly suggest that effective assessment requires the judicious use of strategies and coalitions, and that impact assessment research needs to focus on what facilitates successful outcomes on the ground. Section 6 further discusses preconditions for effective impact assessment.