We are testing a new beta website for gov.scot go to new site

Evaluation of Intensive Family Support Projects in Scotland

Listen

6. Project Support Outcomes

Chapter summary

Evidence mainly from Aberdeen and Dundee suggests that families accepted for Project assistance were typically in receipt of such help for about 9-11 months. It is not possible to derive comparable figures for the three more recently established Projects. Across all five Projects 70% of families whose cases were closed during the evaluation period successfully completed their agreed support programme. Even among those who had withdrawn or otherwise had their support programme terminated early, most had at least partially engaged with Project help.

Project staff assessed 81% of families as being at reduced risk of homelessness/eviction by the time their cases were closed. Complaints of antisocial behaviour had, at this point, been reduced in 94% of cases (Project records here closely matched those of the agencies from which referrals had originated, although the causes of reductions could be complex). In seeking to prevent family breakup Projects faced a more challenging goal. Nevertheless, in exactly half of all cases the assessed risk of family breakup had been reduced by the point of case closure, with this figure rising to almost two thirds (63%) among those who had completed support programmes.

Across each of a range of health and wellbeing indicators, the overall balance of change during Project support was assessed as positive. For example, in 62% of cases where depression was an issue at the time of the original referral, the situation was improved at the point of case closure; in only 14% of cases had the situation deteriorated in this respect. Likewise, alcohol abuse had improved in 43% of cases while deteriorating in only 10%. Children's educational progress and prospects were recorded to have improved in 66% of cases and worsened in only five%. The aspect of family problems apparently least susceptible to Project assistance was mental ill health. As one Project Manager saw it, there was a need for a stronger recognition on the part of mental health services that chaotic families have difficulty in keeping appointments.

All the Projects were acutely aware of the need to engender lasting improvements in self-esteem, lifestyles and relationships rather than focusing narrowly on the immediate suppression of antisocial behaviour. Crucially, staff member interviews in all the Projects demonstrated a concern to avoid fostering dependency among service users and to plan exit strategies for deployment at case closure. A critical factor placing the sustainment of improved lifestyles and behaviour at risk was a reversion to drug abuse. However, although evidence is limited, it would appear that only a small minority of former service users fail to sustain progress at least during the first few months following case closure. By and large, former service users interviewed by the researchers reported maintaining the gains achieved in the course of Project support and complimented Project staff on helping them 'turn their lives around'.

6.1 Chapter scope and sources

6.1 In analysing the impact of project support, this chapter draws on three main sources:

  • Statistical monitoring data relating to all 67 cases closed by the five projects between 1 December 2006 and 30 June 2008
  • Interviews with 15 former service users whose cases were closed between 1 December 2006 and 31 March 2008
  • In-depth interviews with Project staff undertaken in summer 2008 and family case summaries partly informed by caseworker testimony.

Fuller details of each of these methods are set out in Section 1.4.

6.2 The prime data source for the chapter is the case closure monitoring data. Project staff were required to complete case returns for each family having their case closed during the evaluation. Such returns were filed regardless of the reason for case closure - i.e. whether the family's support programme had been completed or whether support had been terminated due to the family's withdrawal.

6.3 However, while this system was successful in collecting quantitative evidence about support outcomes, some methodological issues need to be borne in mind in analysing and interpreting the data. Of particular importance is the nature of the cohort covered by the analysis. The vast majority of the cases in the dataset (50 of 67) relate to the Aberdeen and Dundee projects. This is because of the typically substantial duration of project support. Hence, although 53 families were accepted for support by the Falkirk, Perth & Kinross and South Lanarkshire Projects during the period 1 December 2006-30 June 2008, only in 17 of these instances had cases been closed by the end of the period. Because of their longer-established status, Aberdeen and Dundee are substantially over-represented in the dataset (see Table 6.1).

6.4 Moreover, for the recently-established BtC Projects it is possible that cases closed by 30 June 2008 were atypical in terms of their duration (i.e. cases resolved in a shorter time period than the long run 'steady state' norm). Bearing these points in mind, we have, in Table 6.2, differentiated statistics relating to the 'mature Projects' (Aberdeen and Dundee) from those for the other Projects. It is also interesting to note that families completing support programmes which had involved core block placements were likely to be cases of long duration. The median support duration of such cases was 897 days (as compared with 280 days for all those completing support programmes with AFP and 470 days for DFP).

6.5 The chapter begins by differentiating the reasons for case closure. We then look, in turn, at the impacts of project support on housing circumstances, on the incidence of anti-social behaviour, on the risk of family breakup, and on a broader range of health and welfare issues. Finally, we discuss the extent to which improvements in family lifestyles and behaviour achieved with Project assistance are likely to be sustainable in the longer term.

6.2 Reasons for case closure

6.6 As shown in Table 6.1, some 70% of cases closed over the period resulted from the completion of the family's support programme. Twenty of the 67 case closures (30%) resulted from the family's withdrawal (or the premature termination of support). The mature projects, Aberdeen and Dundee, recorded figures similar to the overall norm.

6.7 In most of the 20 'family withdrawal' cases, Project staff perceived that family members had 'failed to engage' with support - in some instances attributed to ongoing drug abuse. One family had their support terminated when all the children were taken into local authority care. Some were thought to have withdrawn in the belief that they were not in need of Project assistance.

Table 6.1 - Cases closed, 1 December 2006-30 June 2008: Circumstances of case closure

Support programme
completed

Withdrew/ programme
terminated
Total

Aberdeen Families Project

15

6

21

Dundee Families Project

20

9

29

Falkirk

4

3

7

P4 Perth

7

1

8

South Lanarkshire

1

1

2

All - number

47

20

67

All - %

70

30

100

Source: Case closure monitoring returns

6.8 Of the 67 cases closed over the evaluation period, eight involved families who had received support in the core block context. The others had been helped in outreach or (in a few cases) dispersed tenancy settings. Among the eight former core block residents two had withdrawn/disengaged from the Projects while six had completed their support programmes.

Table 6.2 - Cases closed, 1 December 2006-30 June 2008: Duration of Project support (days)

Project

Cohort

Mean*

Median**

Maximum

Minimum

Aberdeen

Support programme completed

350

280

745

115

Aberdeen

Withdrawn

232

216

405

111

Aberdeen

All

316

273

745

111

Dundee

Support programme completed

615

470

2,520

149

Dundee

Withdrawn

306

215

989

37

Dundee

All

519

330

2,520

37

All projects

Support programme completed

439

305

2,520

37

All projects

Withdrawn

259

208

989

37

All projects

All

385

280

2,520

37

Source: Case closure monitoring returns. *Statistical average **Middle value in the distribution

6.9 Many of those whose cases were closed due to 'withdrawal' were at least nominally in receipt of support over considerable periods. As shown in Table 6.2, the overall median duration of Project support for cases being closed after the completion of support programmes during the evaluation period was 305 days. However, bearing in mind the qualifications discussed in Section 6.1, it is probably more meaningful to refer to the figures for Aberdeen and, especially, Dundee - i.e. 273 and 330 days, respectively (about 9-11 months). And, focusing solely on those involving the completion of support programmes, the Dundee figure was 470 days (or 15 months) The fairly substantial difference between Aberdeen and Dundee could reflect the fact that, in spite of having been in operation since 2005, the Aberdeen Project is relatively recently established and it is possible that not all of the original cohort of families had completed their support programmes by 2008.

6.10 Monitoring returns also indicate that in some cases of 'premature' closure considerable support had been provided and three quarters of such families had at least partially complied with their Support Plan. In only five of the 67 cases in the dataset were adult family members considered to have entirely failed to engage. In analysing Project support outcomes, therefore, it seems appropriate to include all closed cases (rather than focusing solely on families recorded as having completed their support programme).

6.3 Impacts of Project support as judged at the point of case closure

6.11 As shown in Table 6.3 Project staff judged that in over 80% of cases, families' risk of eviction was lower at the point where the case was closed than had been the case when the referral had been made. Interestingly, this was perceived to be the case in the majority of cases where cases were prematurely terminated, as well as in those where support programmes were completed.

Table 6.3 - Cases closed, 1 December 2006-30 June 2008: risk of homelessness/threat of eviction at case closure (compared with position at referral)

Circumstances of case closure

Reduced

Unchanged

Increased

Total

Completed support programme

41

4

1

46

Withdrew

11

7

0

18

All - number

52

11

1

64

All - %

81

17

2

100

Source: Case closure monitoring returns. Missing data on three cases.

6.12 Largely consistent with the pattern shown in Table 6.3, the vast majority of families were reported as having reduced their involvement in antisocial behaviour in the course of Project support (see Table 6.4). However, the reasons for this reduction could be complex, for example a child being looked after and accommodated. By and large, outcomes were again positive in this respect for families which had withdrawn rather than completed their support programmes.

Table 6.4 - Cases closed, 1 December 2006-30 June 2008: incidence of complaints about antisocial behaviour at case closure (compared with position at referral)

Circumstances of case closure

Reduced

Unchanged

Increased

Total

Completed support programme

46

0

0

46

Withdrew

14

4

0

18

All - number

60

4

0

64

All - %

94

6

0

100

Source: Case closure monitoring returns. Missing data on three cases

6.13 In keeping with the pattern recorded in Table 6.4, most of the 15 former service users interviewed by the research team reported that Project support had helped them improve in their relationships with neighbours, typically to a 'no complaints at all' level. In one case, however, it was acknowledged that complaints were continuing at a level not dissimilar to that prior to Project support.

6.14 It is acknowledged that the source of the data in Table 6.4 (as well as in Tables 6.3 and 6.5) is the Projects themselves. Clearly, there could be a risk of such judgements being over-optimistic. However, analysis of Dundee cases provides reason to believe this is not so. Here, Project caseworker judgements about changes in the incidence of ASB were compared with referral agency assessments with respect to 26 former DFP service users. This confirmed a very close correlation between the two judgements for the 12 cases about which information was obtained from both parties. In none of these cases was there inconsistency in terms of a Project judgement of 'improvement' being contradicted by a referral agency assessment of 'deterioration'.

6.15 As might be anticipated, preventing family breakup appears to be an aspiration somewhat more challenging than helping families control antisocial behaviour. Nevertheless, in nearly two thirds of cases where support programmes had been completed (63%) it was judged that the risk of breakup had been reduced thanks to Project support (see Table 6.5). Even across the entire cohort of closed cases, it was judged that half of the families concerned were in less danger of family breakup than had been the case at the time of their referral (see Table 6.5).

Table 6.5 - Cases closed, 1 December 2006-30 June 2008: risk of family breakup at case closure (compared with position at referral)

Circumstances of case closure

Reduced

Unchanged

Increased

Total

Completed support programme

27

13

3

43

Withdrew

4

8

7

19

All - number

31

21

10

62

All - %

50

34

16

100

Source: Case closure monitoring returns. Missing data on five cases

6.16 Relevant to the findings set out in Table 6.5 is the evidence that, across the cohort of 67 families, ten were recorded as having one or more children removed to care during Project support. Two families had children returned from care during the period. It is likely that the significant number of children identified as needing to be looked after and accommodated during Project support at least partly reflects the greater scrutiny under which families are placed as an aspect of intensive intervention. This applies, in particular, in the core block context (see Section 4.3).

6.17 Tables 6.6(a)-(g) record caseworker assessments of Project impacts on various aspects of family health and welfare. The assessments relate to family members for whom it was judged that the issue was relevant at the outset. For example, there is data about perceived changes in the incidence of depression relating to 79 individuals (see Table 6.6(b)) whereas data about the changing incidence of drug abuse is recorded with respect to only 36 individuals (see Table 6.6(f)). It should be acknowledged that the data may not be fully comprehensive in the sense of recording perceived changes for every individual for whom a particular issue was relevant at the outset (because there was scope for interpretation in the completion of case closure pro-formas). Nevertheless, it seems reasonable to assume that the overall pattern of responses is broadly representative of the whole picture.

6.18 For every 'health and wellbeing' indicator included in Table 6.6 the overall balance of change was positive - individuals judged to have seen an improvement in their circumstances exceeded those assessed as in a worse position. In several instances, the majority of relevant individuals had seen their situation improve (on depression, drug abuse and children's educational prospects). The strongly positive outcomes regarding children's school attendance are, perhaps, particularly significant given the Projects' emphasis on breaking the (inter-generational) cycle of under-achievement and problem behaviour. While the numbers were small, the issue where the incidence of deterioration was highest was mental health (perhaps reflecting the relatively intractable nature of such problems).

Table 6.6 - Project impacts on family health and welfare: assessed change in circumstances of families having cases closed 1 December 2006-30 June 2008

(a) Physical health

Improved

Remained unchanged

Deteriorated

Total

Adults

16

31

6

53

Children

24

29

4

57

All family members - no

40

60

10

110

All family members - %

36

55

9

100

(b) Depression

Improved

Remained unchanged

Deteriorated

Total

Adults

26

11

7

44

Children

24

8

4

36

All family members - no

49

19

11

79

All family members - %

62

24

14

100

(c) Other mental health

Improved

Remained unchanged

Deteriorated

Total

Adults

8

12

5

25

Children

2

10

3

15

All family members - no

10

22

8

40

All family members - %

25

55

20

100

(d) Employment prospects

Improved

Remained unchanged

Deteriorated

Total

Adults

11

29

5

45

Children/young people

25

9

3

37

All family members - no

36

38

8

82

All family members - %

44

46

10

100

(e) Alcohol abuse

Improved

Remained unchanged

Deteriorated

Total

Adults

10

12

2

24

Children

7

7

2

16

All family members - no

17

19

4

40

All family members - %

43

48

10

100

(f) Drug abuse

Improved

Remained unchanged

Deteriorated

Total

Adults

17

6

4

27

Children

2

6

1

9

All family members - no

19

12

5

36

All family members - %

53

33

14

100

(g) Children's educational progress/prospects

Improved

Remained unchanged

Deteriorated

Total

All children - no

83

37

6

126

All children - %

66

29

5

100

Source: Case closure monitoring returns

6.4 Sustainability of improved lifestyles, relationships and behaviour

Project working practices

6.19 Section 6.3 provides fairly strong evidence of positive outcomes of Project support - even for some families whose support programmes were prematurely terminated. Questions, nevertheless, remain about the sustainability of improvements achieved while continuing to receive Project help. It was clear from interviews with former service users that many had approached the end of the support programme with some trepidation. However, as described in Section 4.3, working practices across all the Projects were clearly influenced by the aim to engender lasting improvements in self-esteem, lifestyles and relationships rather than focusing narrowly on the immediate suppression of antisocial behaviour. Crucially, staff member interviews in all the Projects demonstrated a concern to avoid fostering dependency among service users.

6.20 Across all the projects staff also stressed a recognition of the need for an 'exit strategy' for every case, with a vital component of such strategies being to link families with appropriate ongoing support from specialist and mainstream services. Case closure records show that about three quarters of families were being supported by mainstream social workers at this point. Other agencies cited as playing an active supportive role at this stage included drugs and alcohol support services, health visitors, home school support, employment services and learning disability workers.

Service user testimony

6.21 Former service users interviewed by the researchers stressed that Project staff assistance had helped them develop practical skills and more healthy social networks which provided a firm footing for sustained improvement in their lifestyles and the functioning of their families. Also significant is that most such interviewees voiced praise for their local Project and the impact of Project support on their lives. Two of the 15 respondents uttered rather restrained opinions, describing their Project as 'alright' and acknowledging that '(it) helped me a bit'. One respondent admitted that 'things have gone down hill since [the Project workers] left' and felt her situation to be 'worse now than it ever was'. Much more typical, however, were comments such as 'very helpful', 'brilliant', 'a very positive thing', '(it) changed my life around', 'made a massive difference', 'was the best thing that ever happened to us', 'was a blessing', '(it) gave us a better life, a better lifestyle, everything'.

6.22 When asked about relationships within the household, one former service user interviewee acknowledged that her family were 'not doing very well… it's not really changed'. Another described her family situation as being worse than before Project referral. In the vast majority of cases, however, respondents saw their families as more harmonious and less conflictual than before, and attributed such changes to Project support.

6.23 For additional evidence on service user perceptions of support outcomes and their sustainability see Section A1.4 ( Annex 1).

'Hard evidence'

6.24 In seeking 'hard evidence' on the sustainment of improved family lifestyles and behaviour, reliance on the records of the Aberdeen and Dundee Projects is unavoidable because of the limited number of closed cases as yet registered by the Breaking the Cycle teams. Evaluation fieldwork undertaken in summer 2008 included data collection to inform 'family case summaries'. These related to families which had consented to participate in the research. The researchers also sought to contact and interview these families directly.

6.25 Of particular relevance here are the 11 case summaries relating to 'consenting families' whose cases had been closed by the Aberdeen and Dundee Projects within the relevant timeframe. Collation of these case summaries was undertaken with the help of AFP and DFP caseworkers. Some of the cases involved support programmes completed up to 19 months earlier. Selection bias could have influenced the makeup of this cohort of cases if consenting families were atypical in some way. However, as shown in Table 1.3 the attrition rate resulting from consent refusals was relatively low, thereby reducing this risk.

6.26 As shown in Table 6.7, most of the cases closures could fairly be judged 'successful' in the sense that antisocial behaviour had not been reported as recurring following the termination of Project support. In one case it had become apparent only four months after case closure that lifestyle and behaviour improvements achieved with Project support were not being sustained (see Example 2). In eight of the 11 cases, however, it was judged that the former service users' prospects were good.

Table 6.7 - Former service users - Aberdeen and Dundee Families Projects: Post-case closure outcomes

Project

Duration of Project support (months)

Months since case closure

Case closure circumstances

Subsequent events

Sustainability assessment

Aberdeen

18

10

Support programme completed

No reports of ASB

Good

Aberdeen

18

4

Support programme completed

No reports of ASB

Good

Aberdeen

12

7

Withdrew - did not engage

No reports of ASB

Doubtful

Aberdeen

9

7

Withdrew - partial engagement only

No reports of ASB

Doubtful

Dundee

84

7

Support programme completed

No reports of ASB

Good

Dundee

36

4

Support programme completed

Reversion to drug habit and associated criminality

Nil

Dundee

36

5

Support programme completed

No reports of ASB

Good

Dundee

24

19

Support programme completed

No reports of ASB

Good (but need for long term support)

Dundee

24

9

Support programme completed

No reports of ASB

Good

Dundee

6

19

Support programme completed

ASB complaints much reduced

Good

Dundee

30

10

Support programme completed

No reports of ASB

Good (but need for long term support)

Source: Project caseworkers; service user interviews. Note: Table relates to the 11 (of 50) cases closed by the Aberdeen and Dundee Projects during the period and where the families concerned had given consent for information on their circumstances to be disclosed

6.27 In conjunction with other evidence, the family case summaries also demonstrated that key risks to post-Project sustainment of improved lifestyles and behaviour included both continuing (or a reversion to) drug abuse and the potentially malign influence of a new partner. It was also emphasised by Project staff that some former service users remained highly vulnerable (e.g. due to severe learning disabilities) and that such families would inevitably require long term (in some instances, lifelong) support (see Table 6.7).

6.28 The 11 cases detailed in Table 6.7 represent only a relatively small proportion of the 50 cases closed by AFP and DFP during the evaluation period (see Table 6.1). As acknowledged above there is no certainty that these cases are wholly representative of this broader population (or, even, the population of former service users who had completed support programmes). However, if a large proportion of families formerly in receipt of Project support subsequently reverted to problematic behaviour this would certainly be well-known to the local authorities concerned. Although it did not appear that 'Project sponsoring' councils monitored families' sustainment of improved lifestyles and behaviour in any systematic way, key managers in both Aberdeen and Dundee expressed confidence that, by and large, Project support generated lasting benefits.

6.29 The relatively short duration of the research has limited the scope for tracking former service users and thereby developing a better understanding of the medium and longer-term impacts of Project support. There would be a strong case for building on the fieldwork already undertaken by tracing post-project outcomes for a larger number of former service users. The basis for such work would be the existing cohort of service users who have already provided their consent to be involved in the research. Ideally, such a project would attempt to add to that sample by seeking to secure consents from additional households whose cases are closed in the latter part of 2008/09 and who have not yet provided this.

Example 2

A family of a mother and six children were referred to an IFSP at the point of eviction. There was a family history of domestic abuse and alcohol misuse, mental and physical health, self-esteem, parenting and neglect issues. The family fully engaged with the project and there had not been a complaint about anti-social behaviour in the six months prior to the completion of the work programme, a final review and the closing of the case in February 2008. However, following case closure there was a deterioration in the family situation with the mother reflecting that 'I regretted the day they [the Project workers] had to leave us'. At the point of the last interview she felt that 'It [her family situation] is worse now than it ever was'.

There were continuing complaints from the neighbours about the behaviour of some of the children in the family. One son was subject to an Acceptable Behaviour Agreement and had been fined for an offence. The mother was worried that the son would lose the job that the Breaking the Cycle project had assisted him to secure. Another son had received further warnings from the police and the oldest daughter, whom the project had assisted to obtain her own tenancy, had been evicted. The FIRST Team were visiting the family once a month. The family had also slipped into rent arrears. The mother had suggested to her social worker that the family be referred back to Breaking the Cycle Project but did not know whether this would be possible.