Evaluation of Sixteen Women's Community Justice Services in Scotland

This document presents the findings of an evaluation of sixteen women’s community justice services in Scotland. The evaluation was conducted by the Institute for Research and Innovation in Social Services (IRISS) during 2014-15.


10 Evaluation and Future Research

Introduction

This section describes lessons learned from the evaluation approach, and areas for future research.

Evaluating services

10.1 The Commission's Report highlighted that effective services should have in place "robust performance management and evaluation arrangements to be able to demonstrate impact and value for money".[i], [102]

10.2 WCJSs all held considerable amounts of information about the women they worked with. However, services appeared to vary considerably in their available resources, expertise and/or systems to demonstrate the difference they made. None of the 16 interventions were allocated funds to develop their performance management and evaluation arrangements, as building local evaluation capacity was within the remit of the national evaluators.

10.3 One of the aims of the evaluation was to support WCJSs to build their local capacity for self-evaluation. The scale of the national evaluation meant that the contribution to local evaluation capacity was modest, but included: logic model development sessions, support to use Excel data collection tools and a dashboard to present data, feedback on data quality and completion, presentations at the practitioner's forum, and a self-evaluation workshop to support WCJSs to 'tell their story' using evidence.

10.4 The specific enablers and barriers experienced in this evaluation are highlighted in existing literature[xvi],[xvii], [xxix] and summarised in Table 12. Barriers and enablers have been drawn from existing literature but reflect the observations of evaluators in this current evaluation, and evidence from WCJSs' secondary documents.

Table 12: The barriers and enablers to building evaluation capacity in WCJSs

Enablers

  • A shared understanding of what funders want to see in terms of measurement; clarity about what outcomes are being measured and what constitutes 'success'
  • Robust collection of data that satisfies both internal needs and external requirements
  • Using existing information already collected by services, including rich descriptions of progress in the qualitative accounts of service users and practitioners
  • Common ground and a growing body of evidence about factors linked to reoffending that enable consistent measures to be developed across services
  • Responsibility for evaluation tasks is appropriate to the practitioners role, therefore not viewed as a burden that detracts from their key work priorities
  • Staff have the expertise and/or resources (including admin support) for collating and analysing robust data and opportunities to use it.

Barriers

  • Unrealistic timescales and pressure to evidence longer-term goals for women while the service is 'bedding in' or is only funded short-term
  • Limited investment into systems of outcome measurement and administrative posts to support the upkeep of service monitoring, which has historically focused on activities/outputs rather than outcomes
  • Organisation-wide information systems and resources not responsive to practitioners information needs at the project level
  • Inherited monitoring systems that may not fit current purpose
  • "The absence of an integrated ICT system across Criminal Justice agencies makes collection of even basic data extremely challenging and often relies on manual cross reference and checking" [WCJS proposal]
  • Different audiences requiring different types of information on outcomes
  • Developing measurement approaches in isolation from other services
  • Lack of difficult-to-quantify 'soft' measures indirectly linked to desistence, and gender-responsive measures (e.g. non-criminogenic factors such as welfare, confidence, safety etc.) to support practitioner buy-in of evaluation tools

10.5 This evaluation (and that of the Reducing Reoffending Change Fund mentoring services) provided an opportunity to trial a standardised outcomes tool (the service user questionnaire). The tool captured 'distance travelled' of women against known criminogenic needs and enabled measures to be reported at a national (aggregate) level to demonstrate the contribution of WCJSs to women's desistence journey. This was a need recommended in previous evaluations of women's centres.[xvi] The lessons learned from this evaluation should inform future improvements to the tool's content and implementation.

10.6 In general, practitioners were receptive to evaluation activities. However, some expressed valid frustrations relating to the initial delays in receiving the service user questionnaire (particularly where they were funded from 2013), and ironing out unforeseen issues as a result of 'trialling' it across different services.

10.7 Challenges in introducing the service user questionnaire tended to be greater among large, established WCJSs where it introduced another layer of data gathering to existing processes and systems. New services were typically smaller, less complex, and had a 'clean slate' on which to introduce the tool. Overall, regardless of size, WCJSs experienced challenges relating to the time, knowledge and technical skills required to interpret their data and link activities to outcomes at a service level. We understand that some WCJSs intended to continue to tailor and use tools from this evaluation to inform evidence about the impact of their service in future.

Areas for further research

10.8 This evaluation identified a number of areas for future research, including:

  • Longitudinal studies to understand the impact of WCJSs on long-term outcomes, including women's reintegration into the community and desistance (this may include linking WCJS records with centrally-held criminal proceedings data, such as reconviction rates)
  • A case-control study to explore the extent to which outcomes can be attributed to particular interventions (including groupwork)
  • A comparison of breach and compliance rates by different sentence types and approaches to engagement (e.g. structured deferred sentences, and the effectiveness of voluntary versus statutory engagement in WCJSs)
  • In-depth analysis of the characteristics of those who find progress difficult or disengage from services (including for example interviews with women who have declined holistic support, have reoffended since attending WCJSs/ are in prison)
  • Further qualitative work with women and staff to understand why some outcomes appear to be hard to achieve in practice
  • Interviews with sentencers to gain an understanding of their perspectives of WCJSs and when they would use them
  • Studies that include partner or mainstream agencies to explore the impact (including cost benefit) of WCJSs upon their work.

Contact

Email: Tamsyn Wilson

Back to top