We are testing a new beta website for gov.scot go to new site

Evaluation of Homelessness Prevention Innovation Fund Projects - Research Findings

Listen

The Homelessness Prevention Innovation Fund ( HPIF) was announced as part of the Statement on the Abolition of Priority Need to the Scottish Parliament on December 21, 2005. The fund supported eight new projects to stimulate innovation and expand the range of activities usually undertaken under the umbrella of homelessness prevention. This research draws together lessons from across the HPIF projects to provide local evidence of effectiveness, national learning about homelessness prevention, and lessons about effective evaluation practice. It has also produced two practical tools to help practitioners to embed the use and generation of evidence into programme design and delivery.

Main findings

  • There are valuable lessons to be drawn about homelessness prevention and a strong message that the use of existing evaluation evidence to design appropriate interventions and evaluation of effectiveness needs to be given greater priority by senior managers and staff responsible for homelessness strategy and implementation.
  • The prevention of homelessness does not happen in isolation from other issues. An exclusive focus on homelessness may not yield the most efficient results; rather a holistic or systemic approach is needed.
  • Prevention is both a process and an event; working over time with individuals and communities to enhance their resilience or protective factors and responding to the impending crisis of the loss or lack of a home. The most effective interventions may not be directly primarily at the housing circumstances and homelessness presentations may increase as a result of some interventions.
  • There is a need to design prevention interventions clearly from the outset so that they are outcome focused and address the needs and preferences of communities, rather than what professionals want to provide or think will attract funding.
  • More efficient ways of working and better communication and relationships between statutory and voluntary sector agencies are important contributions to prevention efforts, as well as interventions targeted directly at those at risk of homelessness.
  • Knowledge of rights and options amongst those at risk of homelessness is a protective factor and may contribute to less chaotic, crisis-driven responses to housing crises. The provision of a clear option to remain within the existing home in situations where there is domestic violence can break the link between domestic violence and homelessness.
  • Some of the most effective interventions have been those that have attempted to provide as 'normal' an experience as possible for the at risk group. There are a number of ways of working demonstrated here that normalise and de-stigmatise services and provide a flexible and responsive client-orientated approach.

Key lessons for programme design and evaluation practice

There is substantial scope for improved practice in the monitoring and evaluation of homelessness prevention activities in Scotland. Traditional monitoring and evaluation practice does not serve the needs of complex, multi-faceted and multi-agency interventions of this nature and there are many challenges for monitoring and evaluation. Recent important changes in the national context and the shift in emphasis towards focusing public services on outcomes through Single Outcome Agreements ( SOAs) also underline the need for evidence of the impact and value of homelessness prevention and other activities.

  • Funders and commissioners need to send clear messages about expectations in relation to monitoring and evaluation of both pilot and established projects and programmes at the earliest stages.
  • Resistance to evaluation is pervasive, but evaluation need not be viewed as a threat if it is approached as a built-in way of getting feedback, improving practice and ultimately outcomes for service users and communities.
  • There is a need for a focus on outcomes, not outputs and to value both hard and soft outcomes. A theory of change approach is valuable in encouraging a greater focus on outcomes, rather than outputs.
  • Many homelessness prevention interventions are precautionary, where there is a high risk of future homelessness, rather than a response to an impending crisis. Some deal with known individuals; others work at a more general community level. It may be impossible to prove that a specific intervention has been responsible for preventing homelessness. Definitive attribution of outcomes to specific interventions is probably an unattainable goal. Process improvement rather than proof should be the goal.
  • Assessment of the counter factual or what would have happened without the intervention is conceptually, practically and ethically difficult. It is also not always possible to establish a baseline measure or to ensure data can be gathered at the 'exit' point to assess change over time.

Monitoring and evaluation frameworks should be developed in partnership once outcomes have been agreed, not over-prescribed in advance or based on indicators which have little direct relevance to the specific intervention. The challenge will be to develop meaningful local indicators and targets that ultimately can be mapped to the national outcomes through SOAs.

  • Informal review or reflective practice can be a valid and valuable part of a more formative evaluation process. It is often seen as being outwith the formal monitoring and evaluation process. It should become more systematic and appreciative rather than largely reactive to difficulties. The simple and incidental things or stories of how things are working are important and can reveal as much as any structured framework - both are valuable.
  • Even if a project is on a small scale it is still important to evaluate. Effective small scale interventions should be maintained and others are likely to wish to adopt a similar approach.

The figure below shows how the use of formal evaluation evidence can be blended with other sources of evidence and tested in practice through formative evaluation as a project or programme is implemented.

Figure 1

Background

An announcement in September 2006 allocated a total of £230,000 to eight specific projects. Some projects were concerned with delivering new services to particular groups while others focused on new ways of working within the delivering organisations.

The projects were:

East Dunbartonshire multi-agency training project: a Web-based diagnostic tools for statutory and voluntary sector front line staff.

East Lothian: Domestic Abuse resource pack. Production of an information pack for women experiencing domestic abuse.

East Lothian: CAB rent arrears project. Direct access to the information held within East Lothian Council's revenue systems for staff within two CABs.

Edinburgh Safe as Houses: A sanctuary scheme for women experiencing domestic violence.

Falkirk Anger Management: Conflict resolution courses for homeless people or those at risk of losing their accommodation.

Glasgow Housing and Employment Service: Work opportunities and housing for formerly looked after young people.

Tayside Accommodation and Skills Project: Training in skills to access and maintain accommodation for people with a history of offending and who are homeless or at risk of homelessness.

WISH Project - Women in Sport and Health

Provision of a range of sports and other activities for women who are at risk of homelessness to build their confidence, self-esteem, health and wellbeing and social networks.

Research Aims and Objectives

The aim of this research was to draw together lessons from across the eight HPIF projects to maximise opportunities for learning and to inform the development of homelessness prevention activities across local authoritiesin Scotland. This was a separate commission from the HPIF itself and was intended to be flexible enough to offer assistance to individual projects with their own project evaluation strategies where this was appropriate. In this way it would provide local evidence of effectiveness, national level learning about homelessness prevention and lessons about effective evaluation practice.

The objectives of the research were to:

  • Provide support to enhance individual project evaluations.
  • Assist the HPIF projects to learn from each other, both in terms of actual prevention practice and evaluation strategy, and to share their experiences.
  • Maximise the learning gained from the projects and the likelihood of such learning being put into practice within local authorities and their partners.
  • Disseminate learning about effectiveness to local authorities and the Scottish Government.

Methodology

The research commenced in April 2007. Tailored support was offered to projects to suit the particular scale and stage of each individual project, so that projects with small scale funding, those nearing the end of their funding period (with good quality data in place) and those with proven robust evaluation arrangements had less support.

Support provided was varied but included:

  • Development of a theory of change for the intervention.
  • Help to identify, map and measure hard and soft outcomes.
  • Graphic facilitation of outcome maps.
  • Devising suitable monitoring frameworks and outcome indicators to support outcome focused evaluation.
  • Support to assist projects to analyse and interpret their monitoring data.
  • Advice and facilitation of methods to promote dialogue and engagement in evaluation of project staff, volunteers and service users and external partners.
  • Facilitation of on-going, formative analysis and review of data for learning.
  • Drafting of evaluation reports.

An Exchange Event which brought all eight projects together was held in October 2007. Since then support was focused on encouraging projects to prioritise and complete their evaluations.

Conclusions

The abolition of the priority need distinction from 2012 will broaden local authority responsibilities towards homeless households and preventative work is becoming increasingly important as local authorities work towards this milestone.

This research has highlighted some examples of promising practice and provided useful insights about homelessness prevention and evaluation practice. The research identifies two practical tools to assist practitioners:

  • A series of questions which focus on the threat or crisis which may help service planners in thinking about homelessness prevention as a process and in devising a range of appropriate service responses.
  • Good Practice Guidance to support the development of an outcome focused approach to project and programme monitoring and evaluation linked to local and national outcomes as part of Single Outcome Agreements.

However, the findings of this research are limited by both the scope of the original programme and they way that some of the projects have been implemented and evaluated.

Evaluation was often perceived more as an audit process, than an opportunity for learning and change. Whilst the original research brief encouraged action research approaches, this had not been integral to the original HPIF project design and expectations. As a result, this undermined the potential for such an approach, alongside existing ideas about valid or legitimate evaluation practice and the organisational and cultural constraints under which projects were operating. This is itself an illustration of an important lesson about the need to build in monitoring and evaluation practice from the very start.

Many of the apparent lessons about evaluation practice are actually lessons of programme design which should be a high-level concern amongst managers responsible for homelessness strategy and service delivery.

A move towards a prevention-oriented service requires a shift in organisational culture and ways of working. The lessons identified here endorse that need to change wider practices and behaviours across a range of services.

Changes in the national context

There have been important recent changes in the wider national context in which preventative activities are being undertaken. In particular, there has been a growing national emphasis on developing and delivering public services that focus clearly on achieving clear outcomes. At a local authority level, Single Outcome Agreements ( SOAs) have set out the outcomes, indicators and targets that are being worked towards in each council area. As with other services and activities, homelessness prevention work will increasingly need to build an evidence base which can demonstrate impact, to a broad range of Community Planning Partners. This will be particularly important given the removal of the ring-fencing around both the Homelessness Task Force funding for tackling and preventing homelessness and Supporting People funding.

Many of the lessons here are not confined to homelessness prevention and are of wider relevance across Community Planning Partnerships.

This document, along with "Evaluation of Homelessness Prevention Innovation Fund Projects" (the full research report of the project] and further information about social and policy research commissioned and published on behalf of the Scottish Government, can be viewed on the Social Research website at: www.scotland.gov.uk/socialresearch. If you have any further queries about social research, or would like further copies of this Research Findings summary document, please contact us at socialresearch@scotland.gsi.gov.uk or telephone 0131-244 7560