The 5 Step Approach to Evaluation: Designing and Evaluating Behaviour Change Interventions

Step 2: Review the evidence

What is "the evidence base"?

For the purpose of evaluation and planning, "the evidence base" refers to all available information which might guide what you do in pursuit of your particular aims.

Evidence can come in many different forms, including anecdotes or personal experience. However, when we talk about evidence in this context, we are usually talking about empirical evidence - that derived from purposively designed research studies. However, be aware that because the evidence base is derived from multiple studies, is not always obvious what will work. Studies can have contradictory findings or may ask different kinds of questions.

The following short guide, produced by the Centre for Research on Families and Relationships, Inspiring Scotland and Evaluation Support Scotland, explains what it means to say a programme is "evidence-based:"
http://www.crfr.ac.uk/assets/CRFR_ESS_IS_Evidence_base_briefing.pdf

Why review the evidence base?

Crucial for Planning
A well-designed project will be based on the available evidence about 'what works,' and what doesn't, in relation to your aims. Reviewing the evidence base as part of the planning process will give you the best chance of achieving behaviour change.

Crucial for Evaluation
However, following the 5-step process, reviewing the evidence is also a crucial phase in the evaluation process. Assuming that an experimental design (i.e. RCT) has not been possible, the 5-step process allows you to evaluate the project by assessing the quality of evidence behind a project's theory of change. I.e. what reason do you have to believe that the project's activities should lead to the outcomes envisaged? In addition, it is important that you have a clear idea of the causal processes which underlie the logic of your project so you can plan how you will gather evidence about whether or not they actually took place (see step 4).

Sources of evidence

Research Evidence
Including results of randomised control trials (RCTs), surveys and qualitative studies (e.g. interviews or focus groups). Systematic, literature or evidence reviews synthesise research evidence on a particular topic.

Evidence from Prior Evaluation
If your service (or a similar one) has already been running for a period of time, your own previous evaluations may provide evidence as to whether the approach works or not, how and for whom.

Anecdotal Evidence
Over years of working in a particular field, your own experiences and those you hear about from others can be a further source of evidence. However, whilst valuable, it is important to remember that such evidence may be particularly subject to bias since it will not have been collected systematically.

Research and/or evaluation evidence should be used where available. However, there is no a simple answer to what counts as "good evidence." It depends on the question you are trying to answer. For more detail see these short videos from the Alliance for Useful Evidence: http://www.alliance4usefulevidence.org/

For best results, use a range of evidence

To draw the most robust conclusions about 'what works,' and why, you should take account of evidence produced through a range of methods. For example, quantitative studies (including the results of RCTs) might help you to establish what usually works and for whom. Qualitative work (e.g. interviews with users who 'succeed' and 'fail' and/or with practitioners) might help you to understand the processes through which interventions work or don't work and consider why barriers may exist to achieving your aims.

TIP!: If you are short on time and resources, systematic and/or literature reviews are an excellent source of evidence. They often analyse both quantitative and qualitative studies on a particular topic and should do the work of summarising all this evidence for you.

Finding evidence

When time and resources, are limited, evidence reviews (also called systematic reviews or literature reviews) are a realistic solution - enabling an overview of the evidence in a relatively short time.

Online databases and archives are the most convenient means through which to locate evidence reviews. The following slides provide links to topic-specific databases and some examples of individual evidence reviews in health, education, environment and sport behaviour change aims. However, the following databases can be of general help in locating relevant evidence:

Search academic databases:

http://www.mendeley.com/dashboard/

http://scholar.google.co.uk

Search government archives:

http://www.gov.scot/Publications/Recent

https://www.gov.uk/government/publications

TIP! Try searching for "evidence/literature/systematic review" + your behaviour change aim (i.e. "smoking cessation" or "increase recycling").

Area	A or P*	Topic	Link
Health and Social Care	A	Scottish Government Research	http://www.gov.scot/Topics/Research/by-topic/health-community-care
		Cochrane Collaboration	http://summaries.cochrane.org/
		NICE (guidance and evidence helpful)	http://www.nice.org.uk/
		Health Scotland	http://www.healthscotland.com/resources/publications/search-result.aspx?page=1
		Institute for Research and Innovation in Social Sciences (IRISS)	http://www.iriss.org.uk/resources
	P	Review of 6 health interventions	http://www.storre.stir.ac.uk/handle/1893/3171#.VEYd1o10zmI
		Preventing harmful drinking	http://www.nice.org.uk/guidance/ph24
		Smoking cessation services	http://www.nice.org.uk/guidance/ph10
		Drug treatment and recovery	http://www.gov.scot/Publications/2010/08/18112230/0
		Using cycling helmets	http://summaries.cochrane.org/CD003985/INJ_campaigns-to-encourage-children-to-wear-cycle-helmets
Education	A	Scottish Government Research	http://www.gov.scot/Topics/Research/by-topic/education-and-training
		Education Endowment Foundation	http://educationendowmentfoundation.org.uk/toolkit/
		Joseph Rowntree Foundation	http://www.jrf.org.uk/publications
	P	Attainment in writing	https://www.gov.uk/government/publications/the-research-evidence-on-writing
		Raising attainment/changing attitudes	http://www.jrf.org.uk/sites/files/jrf/education-attainment-interventions-full.pdf
Crime and Justice	A	Scottish Government Research	http://www.gov.scot/Topics/Research/by-topic/crime-and-justice
		Centre for Youth and Criminal Justice	http://www.cycj.org.uk/resources/
	P	Reducing reoffending	http://www.gov.scot/Resource/0038/00385880.pdf
		Reducing reoffending	https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/243718/evidence-reduce-reoffending.pdf
Sport	A	Scottish Government Research	http://www.gov.scot/Topics/Research/by-topic/sport
	P	Examining legacy of major sporting events	http://www.gov.scot/Resource/0044/00449028.pdf
		Barriers/enablers to regular exercise	http://www.gov.scot/Publications/2006/09/29134901/0
Environment	A	Scottish Government Research	http://www.gov.scot/Topics/Research/by-topic/environment
	P	Reducing climate change	http://www.gov.scot/Resource/Doc/340440/0112767.pdf
All areas	A	Evidence for Policy and Practice Information Coordinating Centre (EPPI)	http://eppi.ioe.ac.uk/cms/Default.aspx?tabid=60

* A = Archive of relevant publications, P = specific publication

Under-researched areas

There might be a wealth of evidence about 'what works' in some areas (e.g. smoking cessation). However, you may find a lack of research in relation to your aims. What should you do if this is the case?

Look at similar or related contexts
If you can identify related areas where a larger evidence base is available, you may be able to make logical inferences about what might work based on what has worked in these areas.

E.g. There may only be limited evidence about how to best support persons who become addicted to online gambling. However, the evidence relating to gambling more generally, or addictions in general, may be useful.

Use a rationale
The above approach may not be appropriate or possible in all cases. However, it is always important that your ideas about what might work are based on some kind of rationale. You should be able to explain why, logically, your suggested intervention should achieve your intended aims.

A fictitious example:

How the evidence base supports an intervention to promote young women's physical activity

Intervention (what are we doing?)	Evidence (why are we doing this?)
This project aims to increase physical activity from childhood into adulthood.	Multiple international systematic reviews, drawing on cross-sectional and longitudinal studies have demonstrated the positive impact of physical activity on physical and mental health (see Scottish Government Literature Review, 2004). Physical activity habits have been shown to become established within childhood.
The project is targeted at girls in the final year of primary school and first two stages of secondary school.	Statistical evidence shows that women are more likely to do little or no physical activity than men and that this divergence from their male counterparts begins around the age of 11 (Scottish Health Survey, 1998. 2003)
A choice of team and individual activities will be offered each week, e.g. dance or dodgeball. An emphasis will be made on enjoyment over competition or skill development. There will be no performances or leagues.	A systematic review of the international literature on promoting physical activity, highlighted a need for greater choice for young people, including non-traditional options. Reviews of quantitative and qualitative research by NICE (2007) demonstrate that competition and fear of having to perform may be barriers to taking part in physical activity, particularly for adolescent girls. However, enjoyment has been shown to be a key factor in overcoming these barriers (NICE 2007, Rees et al. 2006)
Social media will be used to promote activities and encourage network-building between participants.	The same reviews by NICE and case-study analysis by the British Heart Foundation (2011) have shown that peer approval and peer participation in physical activity encourages others to join in.