We are testing a new beta website for gov.scot go to new site

Evaluation of Homelessness Prevention Innovation Fund Projects

Listen

CHAPTER 2: HOMELESSNESS PREVENTION - DEFINITION AND INNOVATION

2.1 The recent evaluation of homelessness prevention activities in Scotland found that after three years of the new focus, most homelessness prevention services in Scotland remained 'small and often fairly experimental'12. A typical response by local authorities had been to develop services or procedures to target particular groups seen to be 'at risk'. The research found that it was difficult to generalise about what works best for specific groups. This is in part because interventions that work well in one context may not be as useful in another. For example, Rent Deposit Guarantee Schemes have been judged to be successful by many local authorities, but they do depend on there being a sufficient local supply of accommodation at affordable rents.

2.2 Judgements about what is effective homelessness prevention practice are also hampered by the typical lack of monitoring data and need for improved evaluation practice. The research found that there was substantial scope for improved practice in the monitoring and evaluation of homelessness prevention activities 13. Other research similarly concluded that too little attention is paid to the monitoring and evaluation of the work that is done to prevent homelessness and there is a strong need for the evaluation of effectiveness:

"With so much potentially being described as 'prevention', it is difficult for organisations to decide which is the best approach. At worst, there is a danger that all sorts of things, which are worthy of support in general, will be labelled as prevention work because of its currency. This amplifies the need for evaluation of effectiveness"14.

2.3 In devising effective homelessness prevention strategies, local authorities need evidence to enable them to decide what is likely to be the most effective in their area. This research is in part a response to these issues and challenges.

Defining homelessness prevention

2.4 There is no standard accepted definition of homelessness prevention in use in Scotland. The Code of Guidance provides descriptive information on the kind of activities that local authorities should provide; however, this does not discuss emerging areas such as mediation, housing options work or the use of the private sector 15. Many local authorities have difficulties precisely defining their homelessness prevention activities. This has implications for monitoring and evaluation in terms of defining what constitutes success and how this might be measured.

2.5 Prevention activities tend to fall into two broad categories. Firstly, crisis response, where the threat of homelessness is imminent and secondly, precautionary, where there is a high risk of future homelessness 16. The crisis response category includes interventions that aim to reduce the need for someone to make a homeless application to the local authority and to avoid the need for rehousing in social housing. The precautionary category includes projects which are designed to help people to retain their existing accommodation, manage life transitions and build personal resilience, for example, by building social networks and supporting employment.

The Homelessness Prevention Innovation Fund: Project Profiles

2.6 The eight HPIF projects all go beyond what might usually be considered to be homelessness prevention work. All of them are precautionary projects although there is considerable variation in terms of the imminence of the risk of homelessness; some are clearly much more concerned with resilience building for the long term, whilst others are working with people more immediately vulnerable to an incident of homelessness. A number of the projects adopted an approach which was designed to encourage their clients to make a change from being a passive or dependent receiver of services to a more active participant or agent in their own lives and in their engagement with services. Examples include developing skills to manage anger, raising self-esteem through success in sports or other activities or encouraging women to take the initiative in responding to domestic violence. There are also different 'target groups' for the intervention not all of whom are the end service user or potential homelessness applicant. For example, a number of projects were geared towards ensuring more effective agency systems and processes to ensure greater efficiency and enhance partnership working.

2.7 Table 2.1 shows the location and variation in scale of the different projects. Sections 2.8-2.15 provide a short descriptive profile of the eight projects, based on the original funding proposals.

Table 2.1: The Homelessness Prevention Innovation Fund Projects

Project name

Location

Funds

Original start and end dates

Edinburgh Safe as Houses

Edinburgh

£55220

March 07

March 08

East Dunbartonshire Multi-Agency Training Tool

East Dunbartonshire

£25000

Aug 06

March 07

East Lothian - Domestic Abuse Resource Pack

East Lothian

£5404

Aug 06

April 07

East Lothian - CAB Rent Arrears Project

East Lothian

£1700

Sept 06

September 07

Falkirk Anger Management

Falkirk

£3000

Sept 06

March 07

Glasgow Housing and Employment Service

Glasgow

£30444

Sept 06

March 08

Tayside Accommodation and Skills Project

Tayside

£59568

Jan 07

January 08

WISH Project - Women in Sport and Health

Forth Valley

£49973

Jan 07

January 08

Safe as Houses ( SAH) - City of Edinburgh

2.8 The focus of this project was the installation of practical safety measures in the homes of women at risk of domestic abuse, in order to allow them to stay in their current accommodation. This was modelled on Sanctuary Schemes that have been implemented in a number of local authorities in England. The project was piloted with women living in council tenancies in the north Edinburgh area. Referrals were made from a range of agencies including those that were part of the multi-agency domestic violence forum. It was expected that SAH would facilitate the installation of safety measures in about 20 homes in a year. The project also provided specialist practical advice and emotional support to the families with which they worked, either directly or through onward referrals.

Multi-Agency Training Tool - East Dunbartonshire

2.9 This project was targeted at front line staff working across a range of statutory and voluntary sector agencies. It planned to develop a set of web based diagnostic tools for each of the groups identified at being of greatest risk of homelessness in East Dunbartonshire. These tools should assist staff to identify those at risk of homelessness and make appropriate referrals to other agencies and organisations. The key focus was intended to be systemic and on improving understanding of homelessness triggers, developing organisational understanding of its own role in preventing homelessness and through this to promote more effective and active signposting on to appropriate assistance.

Domestic Abuse Resource Pack - East Lothian

2.10 This project aimed to produce an information pack for women experiencing domestic abuse in East Lothian on behalf of the East Lothian Multi-Agency Violence against Women Forum. It was intended to provide essential information for women and their children about available support and assistance, clarify expectations about the scope and limitations of available services and signpost them to further advice and support. The pack would be a printed document and also be available on the web. The pack was developed with advice from a series of focus groups of women who have experienced domestic abuse.

CAB Rent Arrears Project - East Lothian

2.11 The project aimed to provide direct on-line access to the information held within East Lothian Council's various revenues systems for staff within two East Lothian Citizens Advice Bureaux ( CABs), including access to information on rents, council tax and Housing Benefit. This should allow CAB advisors to intervene more quickly and prevent financial problems from escalating so that the financial situation of individuals does not result in homelessness presentations.

Glasgow Housing and Employment Service

2.12 The key focus of this pilot was on early intervention and reducing the number of 'looked after children' that need to present as homeless. The pilot offered formerly looked after young people work opportunities at Milnbank Housing Associations, linked with some form of college based training in parallel with working for the association. The young people may be living in a range of types of accommodation while participating in the scheme, including foster placements or supported accommodation and there was the intention to offer a Milnbank tenancy to the young people as and when appropriate, with a mentor to support them to set up and manage a tenancy. The project planned to offer 12 placements a year.

Falkirk Anger Management

2.13 This project planned to provide a practical programme of activities for participants who are either in the process of making a homeless application or those who are at risk of losing their current accommodation due to their aggressive behaviour towards others. It aimed to equip those with anger management difficulties with the skills they require to engage positively with the agencies that can help them address their homelessness and other issues. The original plan was to run a pilot scheme covering around 24 homeless people in the first instance, with the potential to expand the scheme after evaluation of the pilot to identify the success of the project.

Tayside Accommodation and Skills Project

2.14 The Tayside Accommodation and Skills project offered a programme of group work which aimed to assist clients develops the skills that will help them in both accessing and maintaining accommodation. The project was developed and run by the Resettlement, Throughcare and Addiction Services Team from the Dundee section of the Tayside Criminal Justice Partnership. It worked with people with a history of offending and who are homeless or at risk of homelessness and seeks to break the 'revolving door' cycle of prison and homelessness by addressing some of the other issues (on top of offending behaviour) which contribute to the problem. It aimed to build resilience by developing the ability of former offenders to respond effectively to housing related issues and crisis by equipping them with a broader range of the skills and information required to access and retain stable housing.

Women in Sport and Health ( WISH) Forth Valley

2.15 Women who are at risk of homelessness from across the local authority areas of Falkirk, Clackmannanshire and Stirling were the target group of this project. It aimed to involve them in a range of sports and related activities in order to build their confidence, self-esteem, health and wellbeing and social networks. The project was based on the existing Street Sport project for men and was based at Falkirk Football Club.

The Challenges for Monitoring and Evaluation

2.16 These diverse projects presented a number of challenges for monitoring and evaluation and held out the prospect of generating useful generic lessons for outcome identification and measurement of homelessness prevention.

2.17 However, assessing the impact of homelessness prevention is inherently difficult; it implies a need to make some assessment of the counter-factual or what would have happened without the intervention. This is conceptually, practically and ethically difficult 17. It is also not always possible to establish a baseline measure or to ensure data can be gathered at the 'exit' point to assess change over time. There are other challenges too arising from the complexity of the contexts and situations in which homelessness arises and the interplay of a number of other influences which are constantly changing. These make it difficult to make an attribution of observed changes to any specific or individual intervention; too many other factors are at work to disentangle the effects of any single project.

2.18 The impact of many interventions may not be expected to be seen for some considerable time, making any attempt at retrospective attribution even more problematic. This also raises issues about the timing of any evaluation assessment and further issues about evaluation design.

2.19 A more practical approach to the evaluation of interventions that aim to assist those people who are at high risk of becoming homeless, is to evaluate whether they have met their project objectives and how they have worked. This is more straightforward where interventions are aimed at people experiencing an actual or imminent crisis of homelessness, such as those about to be discharged from prison or care. This kind of intervention may work with individuals identified to be at risk, almost on a casework type basis. This might allow for more individual tracking and recording of outcomes, although this is not without difficulties.

2.20 However, evaluation is more problematic when interventions are working with individuals who may well share many of the characteristics of people who are known to be at high risk of homelessness, such as those dependent on drugs or alcohol, but for whom it is not possible to predict with any certainty such specific outcomes for those individuals. Some precautionary preventative interventions are aimed at a more generalised 'community' level where, whilst a number of people may be 'at risk' of homelessness, the identity of those individuals cannot be known, prior to the event. This raises issues about eligibility criteria for access to projects of this nature. Inevitably the benefits of such precautionary work will spill over into the wider community. This may not necessarily be unintended or unwelcome.

2.21 Evaluation design issues are also influenced by whether an intervention is working with a stock or fixed, bounded known group or cohort of people or a flow, an unbounded, fluid group in which the individuals are not usually identifiable. Where the target group is known and fixed, summative evaluation of an intervention can be valuable to support judgements about whether an approach has worked (although does not preclude formative approaches). Where the target group is a 'flow', it makes sense to work more formatively to ask how the intervention is 'working', as it proceeds (and this does not preclude a summative assessment).

2.22 These issues mean that there is a need to agree what would constitute success and to do so at the start of an intervention, in order to establish monitoring systems that can provide the required evidence to make judgements. Multi-agency interventions, like many of the HPIF projects, have the added dimension of the need to agree amongst themselves what success will look like and what indicators they will use to make such judgements.

2.23 Decisions about what measures are used to demonstrate impact or 'success' will invariably be difficult and contested amongst different stakeholders. Notionally 'objective' direct measures of success in terms of a reduction in homelessness presentations will often not be appropriate. They are themselves a result of the way that administrative judgements and procedures have been made and are therefore, subject to changes in those procedures (as much as any real change in the numbers of people finding themselves without a home).

2.24 Official homelessness statistics may also not always be a valid measure of success as some interventions may actually increase homelessness presentations. For example, greater information about options for women experiencing domestic abuse may prompt greater presentations and that may support a successful outcome if it means that through that route women are able to access safe, settled and sustainable accommodation.

2.25 This example also shows that homelessness presentation statistics are output or activity measures, not measures of outcomes or change. This raises a further dual challenge of how to ensure both that project interventions are focused on outcomes and that 'hard' and 'soft' outcomes are assessed in making judgements about impact. There is a paradox in that whilst many precautionary interventions are interested in developing more intangible or 'softer' attributes and skills amongst their target group as valid outcomes of their work, there is a lack of understanding of the validity of qualitative evidence and a lack of confidence and skills in this area.

2.26 These issues are all formidable challenges to the evaluation of homelessness prevention. Furthermore, resistance to evaluation is pervasive across public policy; understanding of different approaches to evaluation is limited and often confined to summative approaches undertaken for funding and accountability purposes which are often experienced as oppressive by those working to deliver services. In addition to these cultural factors, a key challenge for precautionary prevention projects was to develop agreed indirect or interim success factors in the absence of direct measures of prevention and in recognition of the longer term or indeterminate timescale in which impact would be expected to be seen.

2.27 The funding proposals for the HPIF included outline plans for monitoring and evaluation of each of the projects. They reflected all the kinds of difficulties of monitoring and evaluation outlined above, although they were not elaborate or detailed and do not seem to have been a factor in decision making about funding individual projects.

2.28 Few proposals discussed the inherent difficulty of assessing prevention of homelessness. Most proposals included reference to monitoring indicators; these tended to be output rather than outcome-focused and were usually quantitative. A number proposed to make reference to trends in homelessness statistics to evaluate their impact; this was perhaps suggested in the absence of any better measure even where this was acknowledged not to be a valid indicator for their project's impact. There was some acknowledgement of the difficulties of attribution of impact to a single intervention and to the measurement of outcomes, including 'softer' outcomes concerned with resilience and skill development. A number of projects were multi-agency interventions but few were proposing to evaluate partnership working.

2.29 In terms of the arrangements to undertake the evaluations, one project had an external evaluator in place from the beginning; others proposed to adopt a similar approach at a later stage, some were proposing to make use of in-house staff from research or policy teams. Some proposals were vague about when and how the evaluations would take place.

2.30 These plans certainly both reflected and endorsed the need for improved evaluation practice. They suggest that little attention or priority had been paid to the realities of monitoring and evaluation in this context. Whilst claims were made that the homelessness prevention intervention being proposed would be 'innovative' parallel attention was not required or paid to the means by which that innovation would be evidenced. In the absence of a strong lead from the funders on these difficult issues, project proposals tended to base their 'plans' on abstract and unrealistic ideas. Some projects were only receiving a small amount of funding and in these circumstances may have felt justified in not providing a detailed monitoring and evaluation plan. These issues are discussed further in Chapter Five.