We are testing a new beta website for gov.scot go to new site

Evaluation of Homelessness Prevention Innovation Fund Projects

Listen

CHAPTER 6: SUMMARY AND CONCLUSIONS

6.1 The experience of the HPIF projects illustrates that the prevention of homelessness does not happen in isolation from other approaches; an exclusive focus on a housing response to homelessness may not yield the most efficient results, rather a holistic or systemic approach is needed. Homelessness prevention is a corporate responsibility for local authorities, with implications for a wide range of services including housing, social work, community services, community safety, revenues and benefits as well as the wider voluntary and community sector. This has implications for programme and project design and for a need for evidence of effectiveness.

6.2 There are valuable lessons about homelessness prevention from the experience of the eight HPIF projects. There is also a strong message that the use of evaluation evidence to design appropriate interventions and the generation of evaluation evidence from the implementation of interventions needs to become a greater priority amongst senior managers and staff concerned with homelessness strategy and implementation.

6.3 Since the HPIF projects and this research were originally commissioned there have been important changes in the national context and the emphasis has shifted towards focusing public services on outcomes. Single Outcome Agreements ( SOAs) cover all the local government services in each local authority area and are linked to the National Performance Framework. This Framework is fully integrated into the Spending Review and underpins delivery of the Government's agenda. SOAs will set out local outcomes, indicators and targets and increase the need to build an evidence base which can convince Community Planning Partners of the impact and value of homelessness prevention activities. The removal of ring-fencing of funding for the implementation of Homelessness Task Force recommendations for tackling and preventing homelessness and Supporting People funding will also lead to an increased need to provide evidence of outcomes achieved through homelessness prevention and housing support.

What is effective homelessness prevention?

Develop a holistic or systemic approach

6.4 The prevention of homelessness does not happen in isolation from other issues. An exclusive focus on homelessness may not yield the most efficient results; rather a holistic or systemic approach is needed. A holistic or systemic approach to homelessness prevention does present a challenge to much existing practice. A more systematic analysis of homelessness in the local context may suggest more holistic and genuinely innovative interventions that are not directly or primarily about housing.

Define homelessness prevention

6.5 Many lessons about effective practice flow from a need to think more clearly about what it is that is to be prevented and to think about homelessness prevention more broadly than homelessness presentations. The clear definition of homelessness prevention has important practical value.

Think about homelessness prevention as a process and an event

6.6 Prevention should be both a process and an event; working over time with individuals and communities to enhance their resilience or protective factors and responding to the crisis of the loss or lack of a home. It may be impossible to prove that a specific project intervention has been responsible for preventing homelessness. Definitive attribution of outcomes to specific interventions is probably an unattainable goal; indeed process improvement, rather than proof should be the goal.

Ensure interventions draw on evidence of the needs and preferences of communities

6.7 There is a need to scope prevention interventions clearly from the outset so that they address the needs and preferences of communities, rather than what professionals want to provide or think will attract funding. Professional judgements do have role to play, but as a part of a blend of evidence of need and analysis of existing evaluation evidence, user views and local knowledge.

Map the links between proposed activities and intended outcomes and clarify the focus

6.8 Better programme planning and project design is crucial to devise the most appropriate outcome focused interventions. Mapping the links between planned activities and intended outcomes in advance may support a clearer understanding of the real focus of the activity and allow projects to identify what about the work they are doing may be expected to contribute to the prevention of homelessness. Equally, it may suggest that the necessary intervention needs to have a different focus, for example, on a differently defined client group. Attention to the barriers to access and sustainability and the most effective timing of an intervention are also crucial.

Think about context: not everything transfers

6.9 Assumptions are often made about what 'good practice' will transfer; these may overlook the distinctive context, needs or preferences of a particular group. This research has shown that gender distinctions have been overlooked, although there are other issues that have not been explored at all such as the needs of particular minority ethnic communities and rural homelessness.

6.10 In any locality, there is likely to be a range of interventions encompassing immediate crisis responses for targeted individuals to more long term community level interventions designed to enhance individual and community resilience to crisis. If programmes based on such an analysis are coupled with a commitment to formative evaluation, then there is a better prospect that evidence about 'what is working' will be generated. This is likely to be a more fruitful approach than one which seeks to base interventions on generalised evidence, without reference to local context, user views or professional experience.

Address the barriers to joint working and better working relationships

6.11 More efficient ways of working and better communication and relationships between statutory and voluntary sector agencies are important contributions to prevention efforts. Effective interventions are based on and can contribute to strong partnership arrangements. Prevention efforts need to also address the barriers to joint working and better working relationships.

Provide information about rights and options

6.12 Knowledge of rights and options by those most at risk of homelessness is a protective factor and may contribute to less chaotic, crisis-driven situations, if not necessarily to fewer presentations. The provision of a clear option to remain within the existing home in situations where there is domestic violence can break the link between domestic violence and homelessness and could be more widely valuable as a potential option for people seeking advice, if it can become embedded into an advice-led housing options approach.

Consider eligibility, targeting and participant motivation

6.13 A crisis-focused intervention is likely to be easier to target on those most at risk, but on its own, more limited in reach and potential. More precautionary interventions raise a number of complex issues about eligibility, targeting and participant motivation. Competency or motivationally-based referral criteria may not be appropriate in a crisis-response intervention but may be necessary in more precautionary interventions. Mentoring support may also need to be provided. Some interventions may need to be based on more loosely structured eligibility criteria that overlap with broader community development outcomes.

Make prevention activities as normal as possible and be flexible and responsive

6.14 There are also lessons about ways of working directly with at risk groups. These include addressing stigma by 'normalising' interventions as much as possible, even opening up access to others beyond those deemed most obviously 'at risk'. Other lessons include the importance of paying attention to language, humanising services and providing a flexible and responsive approach within a broad agreed framework, rather than a pre-defined and rigid service which cannot readily adapt to the varied and evolving needs of clients. Provision of some kind of mentoring may be appropriate.

6.15 Part of designing a flexible and responsive intervention is to consider in advance what the main barriers to access and sustainability are likely to be and address them. There are lessons here also for the need for continued senior level involvement to prioritise and authorise changes to project implementation and funding.

Lessons for programme development and evaluation practice

6.16 The recent greater outcome focus of public services adds force and momentum to the lessons identified here about the need to address monitoring and evaluation practice. It underlines the lessons about the need for funders to make their expectations very clear and to build an outcome focus into programme design and planning, rather than seeing it as primarily an evaluation task.

Embed evaluative thinking into practice at all levels

6.17 An important message from this work is that the use and generation of evidence (or evaluative thinking) has to become embedded into the way of working at all levels. This should start with programme design. Funders and commissioners need to send clear messages about expectations in relation to monitoring and evaluation of both pilot and established programmes at the earliest stages. Commissioning processes should give prospective bidders sufficient time and encouragement to develop robust, deliverable or genuinely innovative proposals. They should expect an outcome focus, but one that is based on the articulation of the links between the desired activities and the prevention of homelessness and which addresses the varied outcomes desired by multi-agency interventions.

6.18 Whilst research-based evaluation evidence is only one form of valid knowledge, it should be drawn on alongside local sources of intelligence and professional judgements to support evidence-informed practice.

Encourage formative evaluation and self evaluation

6.19 Traditional monitoring and evaluation practice does not serve the needs of such complex, multi-faceted and multi-agency interventions. It is often based on narrow perceptions of evaluation as summative, external and more akin to audit than learning. As a result, resistance to evaluation is pervasive and practitioners tend to view evaluation as not useful 22. This is a Catch-22 situation; evaluation need not be viewed as a threat if it is approached as a built-in way of getting feedback, improving practice and ultimately outcomes for service users. Funders need to endorse this perspective, prioritise evaluation and, if necessary, provide appropriate support early on.

6.20 Informal review, self evaluation or reflective practice has sometimes taken place but tends to be seen as being outside the formal monitoring and evaluation process. This is not usually systematic or fully recorded and is often reactive to problems rather than a more balanced appraisal of positives and negatives. Such approaches could be a valid and valuable part of a more formative evaluation process; this should be a legitimate on-going, yet light touch process that only collects data that is meaningful and which is useful for action.

Evaluate in partnership - agree what success would look like

6.21 Many homelessness prevention projects are based on multi-agency working in recognition that prevention of homelessness does not happen in isolation from other interventions. It may be impossible to prove that a specific project intervention has been responsible for preventing homelessness. Definitive attribution of outcomes to specific interventions is probably an unattainable goal; indeed 'proof' of this nature may not be the point. Often all partners are not included in early discussions about how to evaluate the work and in agreement about relevant monitoring indicators. A multi-agency approach to evaluation will help people to recognise the complexity of the context in which projects are working and enable better understanding of the shared contributions of a variety of partners to the ultimate outcomes.

Make the monitoring and evaluation framework meaningful

6.22 Monitoring and evaluation frameworks should be developed in partnership once outcomes have been agreed, not over-prescribed in advance or based on indicators which have little direct relevance to the specific intervention. The challenge will be to develop meaningful local indicators and targets that ultimately can be mapped to the national outcomes through SOAs.

Be proportionate in expectations and identify funding for evaluation

6.23 Expectations of evaluation should be proportionate; lessons about what is effective and of low cost are valuable but the problems of resistance to evaluation reported here will be magnified for modestly funded projects unless expectations are clear and funding is provided.

Promote good practice in the generation of evidence

6.24 There are a number of related lessons for evaluation practice in the generation of evidence. These include the need to define and agree, prospectively, with partners, what success would look like; the need to think about how to measure valued hard and soft outcomes in advance, be clear about what indicators will be most meaningful and realistic and the methods likely to be viable with vulnerable and sometime chaotic client groups; to be formative, flexible and appreciative in approach and to review existing evidence collected through administrative systems to make best use of existing evidence and systems and not over-burden projects with duplicate reporting requirements.

6.25 The views of service users will be important in assessing the achievement of client-focused outcomes. This may be collectable on an on-going basis if systems are established at the outset to capture incidents, stories or other accounts from clients which illustrate how things are working for them. Where this is not possible, provision may need to be made for a specific exercise to gather their views. There is scope for greater creativity about the use of qualitative methods and use of approaches that engage people, including peer-led evaluation, rather than always issuing questionnaires or having focus groups.

6.26 The use of external evaluation input may be valuable but it is not the only or most useful approach. There is scope to use external input in more limited and strategic ways to better effect depending on the skills and capacities that exist within the project or wider organisation.

Practical steps for more effective practice

6.27 The following questions which focus on the threat or crisis rather than the person may help service planners in thinking about homelessness prevention as a process and in devising a range of appropriate service responses linked to better monitoring and evaluation practice. These might form the basis of a more iterative, formative approach to project design, implementation and evaluation.

How do we understand the nature of the threat to people without a secure home or at risk of losing their home?

  • What is the nature of the threat or risk?
  • How does that threat make people vulnerable to homelessness?
  • What are the perceptions and views of those who face this threat?
  • How imminent is that threat?
  • Is a crisis response needed immediately or should a more precautionary process be adopted? Or both?
  • Should the response be targeted to particular individuals already or easily identifiable and/or at a more generic community level?
  • Are there wider benefits from adopting a community level response? Is it acceptable to provide benefits to people not at the highest risk?
  • What are the particular local housing and community issues that impact on the nature of homelessness?

What sources of support already exist?

  • Is there sufficient information for people at risk of losing their home about their options and rights?
  • Do we have sufficient information about existing/other agencies and services to signpost people to?
  • What are the barriers that prevent or inhibit access to those services or support?
  • Are we the best or most appropriate agency to provide a response?
  • What resources or support does the individual person at risk potentially have access to?
  • Can we do anything to enhance these sources of support?
  • Who else should we be working with to enhance our responses to homelessness?

What kind of response would prevent the loss of the home or assist in obtaining accommodation?

  • What options are available?
  • Does the view of an apparent solution change if a longer term view is taken?
  • What are the needs and wishes of those most at risk?
  • Are there more immediate or higher order needs than housing?
  • Do we expect this response to be appropriate for all groups of people - for example: for men and women; young people; people from ethnic minorities?
  • Is additional support or mentoring needed to help sustain people in their accommodation?
  • Is the service to be provided for all people at risk or more targeted? How targeted does it need to be?

How can we change professional attitudes and behaviours?

  • What are the barriers that reduce the effectiveness of existing services and ways of working?
  • How will devising new tools and protocols change practices and behaviours?

How will we know what's working?

  • What would 'success' look like to us?
  • Might others have different views of 'success'? For example, other stakeholders? Service users?
  • What's the link between that success and the prevention of homelessness? Might we also achieve other outcomes and if so, are these valuable?
  • When might we expect to see these results?
  • If 'success' may be difficult to define or take some time, what sort of evidence will convince us that we are at least on the right track?
  • For our evaluation to be useful should it be a process or an event?
  • How can we balance 'lightness of touch' with generating learning for ourselves and others?
  • Looking beyond the immediate intervention we are planning, what else do we need to keep an eye on to help us make sense of our specific impact?

Concluding comments

6.28 This research has highlighted some examples of promising practice and provided useful insights about homelessness prevention and evaluation practice. However, the findings of this research are limited by both the scope of the original programme and they way that the projects have been implemented and evaluated. Some of the HPIF projects have not delivered their original commitments to implement and evaluate an innovative approach to homelessness prevention.

6.29 The original research brief encouraged action research approaches, although this was not integral to the HPIF projects design and expectations. As a result, this undermined the potential for such an approach, alongside existing ideas about valid or legitimate evaluation practice and the organisational and cultural constraints under which projects were operating. However, the research proved to be crucial in delivering much of the evaluation-based evidence about homelessness prevention.

6.30 In practice, for many, evaluation was perceived more as an audit process than an opportunity for learning and change. This is itself an illustration of an important lesson about the need to build in monitoring and evaluation practice from the very start. Many of the apparent lessons about evaluation practice are actually lessons of programme design which should be a high-level concern amongst managers concerned with homelessness strategy and service delivery. A move towards a prevention-oriented service requires a shift in organisational culture and ways of working. The lessons identified here are an important endorsement of the need to change practices and behaviours.