There is substantial scope for improved practice in the monitoring and evaluation of homelessness prevention activities in Scotland. Traditional monitoring and evaluation practice does not serve the needs of complex, multi-faceted and multi-agency interventions of this nature and there are many challenges for monitoring and evaluation. Recent important changes in the national context and the shift in emphasis towards focusing public services on outcomes through Single Outcome Agreements ( SOAs) also underline the need for evidence of the impact and value of homelessness prevention and other activities.
This guidance is based on the learning from the eight Homelessness Prevention Innovation Projects ( HPIF) 1. Many of the lessons here are not confined to homelessness prevention and are of wider relevance across Community Planning Partnerships.
- Funders and commissioners need to send clear messages about expectations in relation to monitoring and evaluation of both pilot and established projects and programmes at the earliest stages.
- Resistance to evaluation is pervasive, but evaluation need not be viewed as a threat if it is approached as a built-in way of getting feedback, improving practice and ultimately outcomes for service users and communities.
- There is a need for a focus on outcomes, not outputs and to value both hard and soft outcomes.
- Many homelessness prevention interventions are precautionary, where there is a high risk of future homelessness, rather than a response to an impending crisis. Some deal with known individuals; others work at a more general community level. It may be impossible to prove that a specific intervention has been responsible for preventing homelessness. Definitive attribution of outcomes to specific interventions is probably an unattainable goal. Process improvement rather than proof should be the goal.
- Assessment of the counter factual or what would have happened without the intervention is conceptually, practically and ethically difficult. It is also not always possible to establish a baseline measure or to ensure data can be gathered at the 'exit' point to assess change over time.
- Monitoring and evaluation frameworks should be developed in partnership once outcomes have been agreed, not over-prescribed in advance or based on indicators which have little direct relevance to the specific intervention. The challenge will be to develop meaningful local indicators and targets that ultimately can be mapped to the national outcomes through SOAs.
- Informal review or reflective practice can be a valid and valuable part of a more formative evaluation process. It is often seen as being outwith the formal monitoring and evaluation process. It should become more systematic and appreciative rather than largely reactive to difficulties. The simple and incidental things or stories of how things are working are important and can reveal as much as any structured framework - both are valuable.
- Even if a project is on a small scale it is still important to evaluate. Effective small scale interventions should be maintained and others are likely to wish to adopt a similar approach.
Embed the use and generation of evidence into programme design and delivery
Good practice in embedding the use of and generation of evidence into practice starts at the commissioning stage. The following questions may help service planners in thinking about homelessness prevention as a process and in devising a range of appropriate service responses based on a range of evidence and linked to better monitoring and evaluation practice. These might form the basis of a more iterative, formative approach to project design, implementation and evaluation as shown in Figure 1.
1. How do we understand the nature of the threat to people without a secure home or at risk of losing their home?
2. What sources of support already exist?
- What is the nature of the threat or risk?
- How does that threat make people vulnerable to homelessness?
- What are the perceptions and views of those who face this threat?
- How imminent is that threat?
- Is a crisis response needed immediately or should a more precautionary process be adopted? Or both?
- Should the response be targeted to particular individuals already or easily identifiable &/or at a more generic community level?
- Are there wider benefits from adopting a community level response? Is it acceptable to provide benefits to people not at the highest risk?
- What are the particular local housing and community issues that impact on the nature of homelessness?
- Is there sufficient information for people at risk of losing their home about their options and rights?
- Do we have sufficient information about existing/other agencies and services to signpost people to?
- What are the barriers that prevent or inhibit access to those services or support?
- Are we the best or most appropriate agency to provide a response?
- What resources or support does the individual person at risk potentially have access to?
- Can we do anything to enhance these sources of support?
- Who else should we be working with to enhance our responses to homelessness?
3. What kind of response would prevent the loss of the home or assist in obtaining accommodation?
4. How can we change professional attitudes and behaviours?
- What options are available?
- Does the view of an apparent solution change if a longer term view is taken?
- What are the needs and wishes of those most at risk?
- Are there more immediate or higher order needs than housing?
- Do we expect this response to be appropriate for all groups of people? For example: for men and women; young people; people from ethnic minorities?
- Is additional support or mentoring needed to help sustain people in their accommodation?
- Is the service to be provided for all people at risk or more targeted? How targeted does it need to be?
- What are the barriers that reduce the effectiveness of existing services and ways of working?
- How will devising new tools and protocols change practices and behaviours?
5. How will we know what's working?
- What would 'success' look like to us?
- Might others have different views of 'success'? For example, other stakeholders? Service users?
- What's the link between that success and the prevention of homelessness? Might we also achieve other outcomes and if so, are these valuable?
- When might we expect to see these results?
- If 'success' may be difficult to define or take some time, what sort of evidence will convince us that we are at least on the right track?
- For our evaluation to be useful should it be a process or an event?
- How can we balance 'lightness of touch' with generating learning for ourselves and others?
- Looking beyond the immediate intervention we are planning, what else do we need to keep an eye on to help us make sense of our specific impact?
Figure 1 below shows how the use of formal evaluation evidence can be blended with other sources of evidence and tested in practice through formative evaluation as a project or programme is implemented.
Figure 1: Evidence-informed practice and formative evaluation
Encourage a formative approach to project design, implementation and evaluation
Commissioners should encourage proposals that are based on a blend of existing evidence about previous or similar initiatives or work with a particular client group; service user views and local knowledge about needs and what is likely to work in a local context; and professional judgement based on established practice and emerging ideas or promising practices. Established programmes and pilot projects should both benefit from this approach.
- Allow prospective bidders sufficient time and give encouragement to them to consult and work up good ideas, broker new partnerships and generate deliverable proposals.
- Be clear about expectations regarding monitoring and evaluation. Give very clear signals about the value of evaluation and provide advice or support from the very start of a programme.
- Don't assume that good practice from one context will automatically transfer to another.
- Encourage a more formative approach to evaluation to test evidence and strengthen local implementation, enhance learning and support continuous improvement.
- Expect an outcome focus and clarify the high level national or local outcomes that project specific or local interventions should link to. Logic models or other tools such as the Weavers or Planning Triangle can help to make these connections.
- Ask proposers to spell out their assumptions at the beginning about typical pathways or links between certain activities, behaviours and the risk of homelessness (or whatever other outcome is desired), including a timeline. A clear signal should be given to encourage greater realism in claims for intended outcomes.
- Make any monitoring and evaluation framework meaningful and useful. Don't over-prescribe detailed indicators in advance or over-burden projects by asking for information that it is not clear how it will be used.
- Ensure that evaluation is proportionate to the scale of the project and identify any funding for evaluation activities.
Evaluate in partnership
Evaluating an initiative that aims to prevent homelessness brings real challenges. Some projects are providing a crisis response, where the threat of homelessness is imminent whilst others are more precautionary, aiming to lessen the risk of future homelessness. It may be impossible to prove that your intervention has been responsible for preventing homelessness presentations. Indeed in some cases, there may be more presentations as a result. Definitive attribution of outcomes to specific interventions is probably an unattainable goal; indeed 'proof' of this nature may not be the point. A multi-agency approach to evaluation will help people to recognise the complexity of the context in which projects are working and enable better understanding of the shared contributions of a variety of partners to the ultimate outcomes.
- Identify the relevant partners in the evaluation of the initiative or project, include front line staff in this process and meet early in the life of the project - ideally at the planning stage.
- Together, consider your perspectives about what would be happening if the project is successful.
- Consider whether some of the project clients should be involved in this process; they may have different views of 'success'.
Clarify and measure your outcomes
Many homelessness prevention projects are actually working to build resilience to crisis, prevent crises or reduce the chaotic nature of homelessness. Often projects are working with clients to equip them with the skills, knowledge, confidence and so on that will help them to deal better with future crisis in their lives. These varied objectives mean that notionally 'objective' direct measures of success in terms of a reduction in homelessness presentations may not be appropriate. An outcome map may show that there are primary and secondary outcomes; many projects are actually aiming to develop more effective agency systems and processes to ensure greater efficiency and enhance partnership working. In such cases, outcomes for service users or potential homelessness people are secondary and such projects need to think about how to measure primary organisational outcomes.
A theory of change approach is valuable in encouraging a greater focus on outcomes, rather than outputs. It provides a roadmap to show how you plan to get from your starting point to your ultimate goal or outcome. The greatest value of this exercise is likely to be the project design stage. It is also a useful practical first step to thinking about what indicators should be in the monitoring and evaluation framework. There are a number of tools available to assist this process.
- Develop a 'theory of change' for your project, with all relevant stakeholders. This means that together you need to spell out your assumptions about typical pathways or links between certain activities, behaviours and the risk of homelessness.
- Produce a timeline; spell out in advance what you expect to see happening, at what stage in the life of the project, so that you will know whether the project is going in the right direction towards intended outcomes and how far it has travelled.
- One approach is to produce an 'outcome map' which shows the links between your inputs, activities (outputs) and intended outcomes.
- Don't be over ambitious in terms of the intended outcomes that you claim you can achieve, especially in a short time.
- Decide what measures of success you will use and develop appropriate indirect or interim success factors or outcome measures.
- Distinguish between primary and secondary outcomes.
- Make sure your measures actually reflect your intended outcomes; it may be appropriate to use 'proxy' measures of the success of homelessness prevention.
- Make sure these measures are meaningful for clients, staff, managers and commissioners by checking them out with them.
- Don't try to measure everything: instead, pick a basket of indicators that covers the range and depth of the type of work you are doing and allows for any required comparisons across projects.
- It may make more sense to evaluate more formatively with all the agencies involved and to ask 'how is our intervention working?' rather than set a future date at which an overall summative assessment is made as to whether the intervention has 'worked'.
Value both soft and hard outcomes
Soft outcomes are those more intangible aspects of the work such as the development of social skills, confidence, motivation, health awareness, growing networks and personal resilience to crisis and so on. Soft outcomes are no less valid than their hard more tangible counterparts; indeed a failure to achieve both hard and soft outcomes may undermine any achievement of hard outcomes such as maintaining the person in their existing home.
Whilst many precautionary interventions are interested in developing more intangible or softer attributes and skills amongst their target group as valid outcomes of their work, there is a need to develop better understanding of the validity of qualitative evidence and develop skills in this area.
- Identify project relevant key hard and soft outcomes that work together.
- Make the capture of evidence about soft outcomes an integral part of your evaluation.
- A good principle for the capture of soft outcomes is to ask service users or other target group to assess their own starting point and progress towards outcomes.
- Sometimes you don't have a baseline. This might be because you are not working with a fixed and known client group on a 'casework' basis (for example, if you're issuing an information resource) so that the target group is a continuing changing 'flow' rather than a stock of people. In these circumstances, think carefully about when it is appropriate to measure success.
- Think about what would be feasible data collection arrangements given the context and client group you are working with. The formalities of form-filling may be a strong deterrent and in practice not usually a priority. This can be a question of attitude and the extent to which data collection is valued and used.
- If you do have a known client group, but haven't collected baseline data, it's still possible to ask people to assess their own changes retrospectively.
- Remember that 'not everything that counts can be counted'. Any assessment of the soft outcomes will need qualitative feedback from agency staff and service users themselves. This need not be onerous or expensive; encouraging more reflective practice is a good way of evaluating intangibles.
- Be open to a variety of qualitative approaches to gather feedback. Don't dismiss stories or accounts from staff; used appropriately they can provide valuable information about particular experience which would be overlooked in quantitative approaches.
Be formative, flexible and appreciative
Some of the most useful learning may arise from a more formative approach to evaluation; an on-going, light touch process that only collects data that is meaningful and which is useful for action. As one of the HPIF projects said 'I think in another sphere that would just be referred to as a continuous improvement and learning approach and I think we ought to be doing that.'
This guidance implies that the evaluation process is likely to be more positive and useful if it is well planned in advance. This is true, but you will also need to stay alert for incidents, stories or other accounts from clients, staff and partners which illustrate how things are working. This will help to develop a fuller understanding of other data and also pick up on unanticipated outcomes or spin offs.
- Encourage all stakeholders to be more evaluation minded; encourage people to share accounts or stories of how things are working - or not. Make informal review processes more systematic by encouraging a discipline of recording, sharing and analysing experience throughout.
- Don't overlook the positive accounts so that you will know what you are doing right (and keep doing it).
- Use evaluation as a chance to provide feedback to staff, give credit for success and motivate staff.
- Stay flexible - stick to the plan if it makes sense to do so but not so rigidly that you miss opportunities or unanticipated outcomes or spin-offs.
- Work backwards not forwards; work out critical milestones when it would be useful to have some feedback - both as the project is unfolding and towards the end.
Involve clients in a real not tokenistic way
Involvement of service users or clients should not be tokenistic. Asking questions is not participation. Asking service users to help you to consider what the questions should be is much more inclusive evaluation practice. There is no template to guide the approach to this issue and it needs to be considered in the specific context and client group with which you are concerned. A peer-led evaluation may be appropriate.
- Include a service user perspective in the planning of the evaluation; this might include their views about your theory of change and what success would look like from their perspective.
- Recognise the diversity amongst service users and ensure that the design and execution of the evaluation minimises barriers to participation.
- Ethical conduct is a particular concern amongst more vulnerable service users. Particular reassurance will need to be given to some service users about their participation in evaluation activities and the approach may need to enable people to be accompanied by a friend, support worker or advocate.
- Don't use methods to collect data that you know from experience are unlikely to get a good response rate. Think more creatively about what approaches will engage people, rather than always issuing questionnaires or having focus groups.
Think about how to use external input most effectively
Formative evaluation is likely to involve a greater degree of self evaluation. Greater self-evaluation can be an important motivator and source of feedback for staff. External evaluation can often outsource much of the learning, although an evaluation does not automatically have to be conducted by an external evaluator. The is scope for greater use of hybrid approaches that involve external input in more limited and strategic ways to better effect depending on the skills and capacities that exist within the project or wider organisation. External evaluator or mentors may act as a 'critical friend', providing bespoke support, capacity building and robust analysis of internal perspectives.
- Review the skills and capacities within the organisation or project to support stronger self-evaluation.
- Identify and consider how to address any concerns that objectivity will be compromised.
- Consider how existing contacts between staff and service users and regular feedback from staff can be used to minimise the reporting burden.
- Think about the appropriate blend of external and internal or self-evaluation and the timing of any external commission.
Accountability and learning: developing monitoring and evaluation in the third sector
Jean Ellis with Tracey Gregory, CES September 2008 www.ces-vol.org.uk
Homelessness Prevention 2007, Shelter Scotland
This reports current thinking about homelessness prevention. It is aimed at national and local policy makers and at practitioners who want to develop their own range of services. www.shelter.org.uk
Evaluation of Homelessness Prevention Activities in Scotland, Hal Pawson et al Scottish Government, March 2007 An evaluation of homelessness prevention activities of local authorities and partner agencies in Scotland. www.scotland.gov.uk
Evaluation of Homelessness Prevention Innovation Fund Projects, Cathy Sharp and Lucy Robertson, Scottish Government, November 2008. www.scotland.gov.uk
Guidance and Resources
Evaluation Support Scotland
Practical support and access to on-line evaluation resources and tools.
Charities Evaluation Service
Information and publications on all aspects of monitoring and evaluation, including an on-line resource guide. www.ces-vol.org.uk
Homeless Outcomes (London Housing Federation)
A resource for homelessness agencies taking an outcomes approach to their work, including guidance on using the Outcomes Star to measure change when working with individual vulnerable service users. www.homelessoutcomes.org.uk
Managing Outcomes A Guide for Homelessness Organisations
Sara Burns and Sally Cupitt. www.ces-vol.org.uk
Explaining the difference your project makes - A BIG guide to using an outcomes approach
Sara Burns and Joy MacKeith, Triangle Consulting, October 2006. This includes the Weavers or Planning Triangle.
Theory of Change
Assistance and materials to support the use of the Theory of Change approach.
How to gather views on service quality - guidance for social landlords
Cathy Sharp and Sheena Murdoch, 2006. This provides guidance on methods which can be used to obtain service user feedback, of relevance to a wider range of public and voluntary services. www.scottishhousingregulator.gov.uk