The 5 Step Approach to Evaluation: Designing and Evaluating Behaviour Change Interventions

Were there sufficient resources to run the intervention and how were they deployed?

• The evaluation should show what resources were required to run the intervention and whether they were sufficient to deliver the intervention as intended.

• The total cost of the intervention.

• Average £ spent on each user.

• What were funds spent on? How many staff were required, staff, staff case loads, costs of running sessions, cost of materials, venues etc.

• Gather views on whether resources were sufficient.

• To what extent was the evidence base embedded into the intervention?

Manager and staff

Annual accounts

Intervention level database

Costs, values and views

Costs can be reviewed periodically (e.g. annually)

PARTICIPANTS

Collect information on your users to check that you reach your intended target group.

• Set up the database so you can collect data on each user.

• Data can then be aggregated to provide important quantitative data on users e.g. percentages, averages etc.

• You can also see whether the intervention worked for some users but not others by breaking down outcome data into different types of users (e.g. different ages, offence types). Numbers have to be large for this to be meaningful though.

User ID number

N/A

User level database

Entered as 01, 02, 03 etc

Name

User survey

User level database

Name

Date of birth

User survey

User level database

Date of birth

Age at start of programme

User survey

User level database

Age

Gender

User survey

User level database

Column - Gender

Male= 1, Female= 2

Current levels of physical activity

User survey

User level database

Coded e.g.

1 = none, 2 = minimal… 6 = regular cardio exercise

Previous experience of physical activity

User survey

User level database

Coded e.g.

1= none, 2 = school-based, 3 = sports club…

Attitude towards exercise

User survey

User level database

Coded on scale of 1-5 (very positive to very negative)

ACTIVITIES

What did users experience?

• Information on activities is important because if activities didn't happen or were poorly delivered, then it is unlikely that outcomes will occur, if they did, something external to the intervention might be responsible.

• Work out the number and % of users who complete and did not complete the project as a whole and which activities they took part in.

Number of potential participants informed by school visits

School records

Intervention level database

Number of participants recruited

Weekly register

User level database

This can be calculated via the total number of participants entered into your database

Number of participants at each session choice of activity

Staff observation and weekly register

User level database and intervention level database

For each user record which sessions they attended and activities undertaken. Can code as:

Week 1: 1= attend 2=did not attend

Week 2: 1= attend, 2= did not attend etc.

Also record total numbers for each session and activity in a separate record of activities. This will enable you to identify patterns in individual behaviour as well as analyse overall attendance and participation each week.

Number of sessions run by outside clubs and classes

Staff observation and weekly register

In your record of sessions, record the sessions which were run by, or had visits from, outside organisers. This will enable you to analyse the impact these visits had on take up of outside clubs and classes.

How did users experience the project?

• Gather user accounts of what they actually did.

• The extent to which users valued the content of the project and their views on the way the it was delivered are important.

• User perspectives on what happened in the sessions, the length of sessions, the format, quality of relationships with organisers and peers, what they learned and skills they developed.

• What did they enjoy most and least?

• Did they come every week? If not, why not?

• Did they engage with social media? Why/why not?

User survey

User level database

Assign codes to closed responses

For example, user views on relationships with organisers:

Very poor = 1, poor = 2…….. Very good = 5

and enter into database.

E.g. 2 - enjoyment:

Very enjoyable = 5, enjoyable = 4…very unenjoyable = 1

Analyse qualitative open questions by theme (not entered into data base)

SHORT and MEDIUM-TERM OUTCOMES

Did change happen?

Quantitative measures of change

• Obtain a pre-intervention base line and post-project assessment end-line. (see info. collected on participants).

• Short term outcomes tend to be changes to attitudes, knowledge, learning, motivation or skills.

• Medium term outcomes show evidence of individual behaviour change.

• Measure the same outcomes at the start and exit point to see if change occurred.

• As well as scales, ask the users, supervisors and family as to whether they think users have changed and in what way.

• If there is no control group but you want to explore contribution you could elicit views on the relative impact of the intervention by asking users and family about perceived impact other interventions or support has had.

• Could also observe sessions at the exit to see if progress has been achieved.

• The difference between the baseline situation and the end situation is the measure of whether change happened.

• Record the results of tests and surveys with users at the start and end of the intervention:

Nb. Depending on your aims and approach, it might be appropriate to use psychometrics tests, physical tests (e.g. fitness or BMI), as well as surveys to measure attitudes, behaviour, competencies etc.

• Did users friends and family think progress was made?

• Professional judgement of progress (i.e. from teachers, doctors)

• User and family views on the contribution of external factors to outcomes, relative to intervention.

User survey

Family and friends survey

Survey of relevant professionals

User and family survey

User level database

Create two columns-one for the outcome variable before and one column for after the intervention . For example:

Attitude towards physical activity (before):

Very positive = 5, positive = 4…..Very negative = 1

and attitude towards physical activity (after):

Very positive = 5, positive = 4…..Very negative = 1

Could code answers

e.g. Teacher's views of participant's progress:

Very good = 5,
good =4..
none/minimal = 1

And/or transcribe interviews for more depth information

LONG TERM OUTCOMES

(Difficult to assess without RCT)

Did the intervention increase physical activity into adulthood?

Did the intervention improve long-term health and fitness?

• Did users think they had made sustained progress?

• Do stakeholders (eg. teachers and parents) perceive long-term changes in the group who engaged with the activity?

Longitudinal surveys of user and stakeholder views

Separate analysis conducted