Evaluating evolving and complex programmes: learning and reflections from the child poverty pathfinders' evaluation
This paper presents learning from the evaluation of the child poverty pathfinders in Dundee and Glasgow about evaluating evolving and complex programmes. It is intended to support policy makers, programme delivery teams and evaluators to get the best out of future evaluations of similar initiatives.
Appendix A: Possible measures for common child poverty project outcomes
The breakeven analysis for the Dundee pathfinder included in the phase 2 evaluation report focuses on three key outcomes: job starts, improved financial stability, and improved subjective wellbeing. These are not the only possible intermediate outcomes child poverty projects are likely to wish to influence. However, to the extent that they are common goals across its funded child poverty projects, it may be worth the Scottish Government considering how data on these could be collected more consistently across its funded programmes, to facilitate both individual evaluations and comparisons of outcomes across programmes. Potential questions to measure these outcomes, which could be included in guidance on a ‘common core’ of data items to be collected, are outlined below.
Outcome: Job Starts
Possible question: While the Scottish Government may wish to recommend a standard format and set of job outcome codes (e.g. defining what counts as full-time and part-time work and which categories should be used to record those not in work) for recording work status to ensure consistency, the precise question asked of clients to establish job status is less important than accurate recording of work status when someone first engages with an intervention, and then accurately tracking job starts (and length of employment). For those in, or moving into, employment, it is also important to capture hours worked (or at least whether they are full or part-time) and salary level (or at least whether they are paid at or above the living wage) in order to capture the nature and quality of jobs people are moving into. This data could also be used to track job progress, in addition to job starts – that is, are programmes also helping people in work to move into better paid and/or more full-time work?
Notes: There are known challenges collecting accurate and complete employment data when a programme is voluntary and open-ended. Given this, it is worth considering whether additional administrative resource is required to support delivery staff in collecting this data.
Outcome: Improved financial stability
Possible question: How well would you say you yourself are managing financially these days? Would you say you are...
Living comfortably, Doing alright, Just about getting by, Finding it quite difficult, Finding it very difficult
Source: Understanding Society
Outcome: Improved subjective wellbeing
Possible question: Overall, how satisfied are you with your life nowadays, on a scale of 0 to 10, where 0 is ‘not at all’ and 10 is ‘completely’.
Source: Office for National Statistics
Notes: This is the question that underpins the WELLBY method.[8] Given this, it could, in principle, be used to estimate the ‘social value’ (in monetary terms) of the impacts of an intervention independently of data on other contributors to this. This would require developing a robust quasi experimental design to isolate the impact of the scheme on life satisfaction, compared to a control group. (If this cannot be achieved, it would be recommended to continue to collect data on other outcomes, to allow for triangulation.)