Future Directions and Opportunities - Monitoring and Evaluation
Monitoring and evaluation ( M&E) processes
4.1 In order to assist in the development of future M&E requirements and make connections between IDF projects and the programme as a whole, a review of current M&E processes was undertaken. This helped highlight possible opportunities to build on and test new ways of evaluating impact of Scottish Government policy in this area.
4.2 Theme 4 on Project Design has already emphasised the importance of establishing at the outset an expectation amongst grantees that they will need to monitor and report on their projects. Within that, it is also necessary to plan ahead, even at application stage, for an evaluation of project progress and impact.
4.3 Again, as highlighted elsewhere in the review, ongoing dialogue with grantees and beneficiaries is essential and a structured monitoring process (as part of an annual monitoring cycle) can help encourage that constructive, learning approach, which can lead to other benefits for grantees.
4.4 Looking ahead, the increase in the IDF means that there will be a greater expectation to report on and evaluate impacts, both at project and programme level. Steps have already been taken to address this and, as part of this ongoing process, any M&E framework will need to take account of the new policy environment.
4.5 The review brought forward thinking on how an M&E system might be aligned to the Scottish Government's international development outcomes. Such a structure could offer a framework to illustrate how future projects are contributing to the wider programme through: project application; reporting and monitoring formats; indicators; and information feedback systems. Such tools would need to be user-friendly and joined-up, so that information on, for example, application forms and reporting forms could be simply and easily cross-referenced.
The review tested a range of indicators in order to illustrate how M&E systems might be used to align to the Scottish Government's focus on outcomes. Using three policy-level outcomes in the 2008 International Development Policy, some illustrative indicators were generated. The three policy-level outcomes were
- The further development of Scotland's special relationship with Malawi
- The achievement of sustainable outcomes in support of the MDGs and Malawi's development priorities
- Strengthening Scotland's identity as a responsible nation.
4.7 These illustrative indicators (Box 23) show how the International Development Fund could aim to demonstrate its contribution to the International Development Policy and to Malawi's MGDS.Box 23 Illustrative policy-level outcome indicators for monitoring the International Development Fund
A. Further development of Scotland's special relationship with Malawi
A1 The Scotland-Malawi Co-operation Agreement is renewed by the end of the 3 year MDP cycle and the scope of the Scotland-Malawi Joint Commission expanded.
A2 The establishment of new Scotland-Malawi collaboration at all levels (private, public and NGO sectors)
B. The achievement of sustainable outcomes in support of the MDGs and Malawi's development priorities.
B1 The Scottish Government's Malawi Development Programme (MDP) contribution to the achievement of Millennium Development Goals in Malawi.
B2 The MDP contribution to Malawi's Key Development Themes as outlined in the Malawi Growth and Development Strategy 2006-2011.
C. Strengthening Scotland's identity as a responsible nation.
C1 The funds disbursed by the MDP equals the fund commitments made in the International Development Policy, 2008.
C2 The recognition by the Malawi Government of the role and function of the Scottish Government's MDP in supporting Malawi's development priorities.
C3 The contribution of Scottish Government monitoring information to the MGDS monitoring by the Malawi Government.
4.7 At the programme level, the review raises the possibility of taking a balanced approach to the measurement of programme performance using 4 dimensions:
- Fund administration (systems and processes)
- Financial management (accountability and value for money)
- Social development (developing partnerships and relations with partners)
- Technical, or sector, performance by strand (Education, Sustainable economic development, Health, Civil society development and Governance) and cross-cutting themes (Enterprise development; Gender issues and equality; Vocational training and education; and Strengthening the context for enhanced human rights and civil society development).
- Additional indicators were identified that reflected the excellence in research, science and technology that Scottish organisations are able to bring to development interventions in Malawi.
4.9 An additional option that might be considered is the use of narratives to draw out experiential learning. These provide rich texts that illustrate, in a more human way, the impact a particular intervention has had on the lives of beneficiaries. Such a process may not prove a burden, as these narratives have already been effectively used by some Malawi-focused IDF projects as part of their monitoring and reporting process.
4.10 A series of indicators were generated and tested on a sample of projects on the Education strand. Reassuringly, this process found that much of the information needed to populate the education relevant indicators was already being collected by the grantees and was available in reports that had been submitted as part of the initial performance assessment. The additional reporting burden on grantees in this case would therefore have been low. The associated administrative burden of such an approach could be further minimised by ensuring that any individual project only reports on a small number of the standard indicators. Moreover, many projects will already use equivalent indicators in their own monitoring processes in any case.
4.11 The illustrative indicators were also tested for alignment against all areas of the MGDS (2006-11) themes and sub-themes: all are covered except the Disaster management sub-theme, but this is not a strand that the Scottish Government has been supporting in Malawi through the IDF. This demonstrates the continuing relevance to the Malawian agenda, despite the wider remit of the strands. Alignment to the MDGs was also tested and they provide good coverage of the MDGs relevant to the Scottish Government's International Development Policy.
4.12 The Scottish Government will need to carefully consider the resources they have for managing any additional M&E systems in order to make evidence informed connections between project and programme level.