Step 2: Review the evidence
What is the 'evidence base'?
For the purpose of evaluation and planning, “the evidence base” refers to all available information which might guide what you do in pursuit of your particular aims.
Evidence can come in many different forms, including anecdotes or personal experience. However, when we talk about evidence in this context, we are usually talking about empirical evidence – that derived from purposively designed research studies. However, be aware that because the evidence base is derived from multiple studies, is not always obvious what will work. Studies can have contradictory findings or may ask different kinds of questions.
The following short guide, produced by the Centre for Research on Families and Relationships, Inspiring Scotland and Evaluation Support Scotland, explains what it means to say a programme is 'evidence-based:'
Why review the evidence base?
Crucial for planning
A well-designed project will be based on the available evidence about ‘what works’ and what doesn’t, in relation to your aims. Reviewing the evidence base as part of the planning process will give you the best chance of achieving behaviour change.
Crucial for evaluation
However, following the 5 step process, reviewing the evidence is also a crucial phase in the evaluation process. Assuming that an experimental design (i.e. RCT) has not been possible, the 5 step process allows you to evaluate the project by assessing the quality of evidence behind a project’s theory of change. i.e. what reason do you have to believe that the project’s activities should lead to the outcomes envisaged? In addition, it is important that you have a clear idea of the causal processes which underlie the logic of your project so you can plan how you will gather evidence about whether or not they actually took place (see Step 4).
Sources of evidence
Including results of randomised control trials (RCTs), surveys and qualitative studies (e.g. interviews or focus groups). Systematic, literature or evidence reviews synthesise research evidence on a particular topic.
Evidence from prior evaluation
If your service (or a similar one) has already been running for a period of time, your own previous evaluations may provide evidence as to whether the approach works or not, how and for whom.
Over years of working in a particular field, your own experiences and those you hear about from others can be a further source of evidence. However, whilst valuable, it is important to remember that such evidence may be particularly subject to bias since it will not have been collected systematically.
Research and/or evaluation evidence should be used where available. However, there is no a simple answer to what counts as 'good evidence.' It depends on the question you are trying to answer. For more detail see these short videos from the Alliance for Useful Evidence: http://www.alliance4usefulevidence.org/
For best results use a range of evidence
To draw the most robust conclusions about ‘what works,’ and why, you should take account of evidence produced through a range of methods. For example, quantitative studies (including the results of RCTs) might help you to establish what usually works and for whom. Qualitative work (e.g. interviews with users who 'succeed' and ‘fail’ and/or with practitioners) might help you to understand the processes through which interventions work or don’t work and consider why barriers may exist to achieving your aims.
TIP If you are short on time and resources, systematic and/or literature reviews are an excellent source of evidence. They often analyse both quantitative and qualitative studies on a particular topic and should do the work of summarising all this evidence for you.
When time and resources are limited, evidence reviews (also called systematic reviews or literature reviews) are a realistic solution – enabling an overview of the evidence in a relatively short time.
Online databases and archives are the most convenient means through which to locate evidence reviews. The following slides provide links to topic-specific databases and some examples of individual evidence reviews in health, education, environment and sport behaviour change aims. However, the following databases can be of general help in locating relevant evidence:
|Search academic databases:||Search government archives:|
TIP Try searching for “evidence/literature/systematic review” + your behaviour change aim (i.e. “smoking cessation” or “increase recycling”).
* A = Archive of relevant publications, P = specific publication
There might be a wealth of evidence about ‘what works’ in some areas (e.g. smoking cessation). However, you may find a lack of research in relation to your aims. What should you do if this is the case?
Look at similar or related contexts
If you can identify related areas where a larger evidence base is available, you may be able to make logical inferences about what might work based on what has worked in these areas.
For example, there may only be limited evidence about how to best support persons who become addicted to online gambling. However, the evidence relating to gambling more generally, or addictions in general, may be useful.
Use a rationale
The above approach may not be appropriate or possible in all cases. However, it is always important that your ideas about what might work are based on some kind of rationale. You should be able to explain why, logically, your suggested intervention should achieve your intended aims.
A fictitious example
How the evidence base supports an intervention to promote young women’s physical activity
(what are we doing?)
(why are we doing this?)
| || |
| || |
| || |
| || |
Email: Catherine Bisset