Needs Assessment: A Practical Guide to Assessing Local Needs for Services for Drug Users
Chapter 8: Monitoring and Evaluating
The aim of needs assessment is to better meet the needs of your target population. The process of needs assessment is about gathering information to find out what those needs are, and what are the best ways of meeting them. In most cases, a needs assessment exercise will result in change - either in the way existing services are provided, or in the introduction of new services or interventions. It is important to check if these changes are making a difference in relation to the identified needs. For that reason monitoring and evaluation should be an integral component of the process of needs assessment.
The purpose of monitoring and evaluation is to determine whether the changes you have made are having the impact you expected. The evidence gathered through monitoring and evaluation may also be used as the basis for further needs assessment. As outlined in the EIU Evaluation Guide 1, monitoring and evaluation are closely linked but involve two distinct processes. Monitoring is an ongoing process involving the continuous or regular collection of key information to allow regular checks on progress. It aims to check whether an intervention is going to plan but does not provide information about the changes that could be made to improve outcomes. An evaluation involves looking back to find out what difference an intervention has made. As such, it can be used to show how and why something is working or not working.
Monitoring and evaluation understood as a journey by car: Monitoring involves a flow of information on matters such as average speed, distance travelled, fuel consumption and whether the journey is following the pre-planed route and is on time. Evaluation addresses questions such as whether the route followed was the best one and, indeed whether the journey was worth undertaking at all (EIU Evaluation Guide 1).
Planning an evaluation
It is important to be clear from the outset why the evaluation in being conducted, who it is for and whether it is feasible. An evaluation will be most feasible if it is included as an integral part of developing the intervention itself, and if a 'baseline' has been established before the intervention is introduced. Evaluations vary in their subject, purpose, timescale, design, and methods. Involving service providers, clients, funders and other stakeholders in the planning can help clarify some of these variables. You will need to decide whether to undertake the evaluation internally or to employ external consultants. Consider what it is you want to know, the scope of the exercise and whether the evaluation requires particular expertise.
The sources of data for your evaluation will be many of those you used for the initial needs assessment exercise. In particular, two important sources of information are:
basic work-related data including information collected through the assessment of individual clients' needs; notes of meetings which describe what decisions were made and why; diaries and appointment books; budgets; correspondence
information from those involved, both practitioners and those receiving care, gathered from interviews, discussions and questionnaires.
An evaluation can take a number of forms. Two main types of evaluation are:
Process evaluation: Process evaluation focuses on how an intervention is working and why. It looks at processes and procedures. This type of evaluation can support plans to repeat an intervention somewhere else because it helps to identify how and why something does or does not work.
Outcome evaluation: The aim here is to find out whether the desired change has been achieved. A typical question addressed by an outcome evaluation would be: has the intervention made significant improvements in clients' lives?
In the context of evaluating changes following a needs assessment, it will be helpful to use both forms of evaluation. The box below lists some key questions to ask when undertaking an evaluation.
Key Questions to Ask when Undertaking an Evaluation
Are the original aims and objectives being followed, or still relevant?
What is happening? Is everything proceeding as expected? If not, why not?
What do service providers and service users think about the changes? Are things working for them? Why or why not?
What resources are being used? Are they adequate?
Have the aims and objectives of the changes been achieved?
How many people have benefited from the changes, and what are their characteristics?
Are the people who are benefiting from the changes the same people you intended to benefit from it?
EIU has published a number of 'Evaluation Guides' that will provide further information about how to evaluate an intervention or a change to a service. These are available from the EIU website at: www.drugmisuse.isdscotland.org/eiu/eiu.htm
When planning to monitor and evaluate think about:
How to monitor and evaluate so that you know whether the changes introduced are having the desired effect
Why you are doing the evaluation, who it is for and how it will be used
Involving service providers, service users and carers, funders and other stakeholders in planning the evaluation
Ensuring all the relevant information for the evaluation will be available to you when you need it
The most appropriate methodology for the evaluation
Whether the evaluation can be done internally or by an external consultant