Independent legal representation and the victims, witnesses, and justice reform (Scotland) bill: A review of potential models of delivery

Chapter 9: Evaluation and Measuring Success

Key points

• ILR services can be evaluated using ‘victim measures’, ‘system measures’, or a mixture of both. Victim measures relate to how complainers feel as a result of engaging with the service. The two victim measures that were repeatedly mentioned by research participants were increased agency for complainers and improved complainer wellbeing. Both can also have a positive impact in terms of preventing complainer withdrawal from the prosecution process.
• As well as the positive benefits to complainers arising from ILR, participants noted that complainer wellbeing might be adversely affected if introducing ILR had the knock-on effect of introducing delays into the progress of the case.
• System measures relate to service outcomes, such as substantive legal outcomes, the take-up of the service, or case processing times. These can be informative but substantive legal outcomes in particular is a measure that needs to be regarded with caution, as complainers may feel an increased sense of agency even if the outcome of the hearing is not the one that they wanted.
• There is a lack of transparency around the evaluation of existing ILR schemes. Publicly available data is almost non-existent, which limits the opportunity for an international community of practice to develop.

9.1 Introduction

This chapter discusses the evaluation of ILR schemes. It draws from the interview data to examine how research participants thought that ‘success’ should be defined and measured. It then moves on to consider how ILR schemes have been evaluated in practice, although in relation to this, its main finding is that very limited data exists in the public realm.

9.2 How might ‘success’ be measured?

There are various ways in which an ILR scheme might be evaluated. These can broadly be divided into what might be termed victim measures and system measures.^[585] Victim measures here relate to how victims feel as a result of engaging with the service. This might include feelings of general wellbeing, agency or feeling more supported or knowledgeable about their rights. System measures relate to service outcomes, such as substantive legal outcomes, case processing times or the take-up of the service. The two types of measure are often related – victim satisfaction might be improved by, for example, the outcome of the case or detrimentally affected by, for example, longer case processing times. One of the very few evaluations of an ILA/ILR scheme that is publicly available used both types of measure in its assessment.^[586]

System measures are generally more easily quantifiable than victim measures. The latter might be quantitively assessed – at least to an extent – by comparing the satisfaction levels of two separate groups of complainers who have engaged with the criminal justice system with and without ILR. The difficulty would be in confidently attributing any differences in satisfaction levels to the ILR service, as it would be impossible to hold every other aspect of the complainer experience constant. As such, they are perhaps better thought of as qualitative indicators that can be explored via complainer interviews or focus groups.

9.2.1 Victim measures

Victim measures were the most commonly mentioned measures of effectiveness among our interviewees. When asked how they would assess the success of an ILR scheme, almost all respondents focused on client satisfaction as the primary evaluative criterion. One interviewee stated that the measure of success for their particular scheme would be: “that we have happy clients most of the time for the most part”.^[587] Another interviewee put it like this:

I think the only person that can evaluate it is the person themselves who’s gone through that support or that engagement. So, it would be the victim-survivor, I think they’re the only one who can evaluate whether or not it felt okay.^[588]

Complainer satisfaction in the context of an ILR scheme has at least two distinct but related components.^[589] The first is agency. Research has suggested that complainers feel that they have little or no control over what happens in the criminal justice process after they report an allegation of a serious sexual offence and that this marginalisation is very stressful.^[590] Although in the schemes that were examined, ILR is provided only in relation to a small component of the criminal justice process (admissibility of sensitive records or sexual history evidence), interviewees commonly spoke of an effective scheme being one that gave complainers a sense of agency over at least one aspect of the process that they would not otherwise have had, or one in which there can be “effective participation”^[591] by the complainer. In relation to an ILR for sensitive records scheme, one interviewee stated:

What I think is at the absolute crux of all of this is that aspect of agency around their records. There’s so many other parts that they have no control over in the criminal justice process, so the more we can give them that is definitely a focus for us.^[592]

Interviewees stated that a scheme might be successful in providing a sense of agency to complainers, even in a case where the substantive legal outcome was not the one that they desired: “it might not be the result they wanted, but they have that information and they were involved”.^[593] As one interviewee put in in relation to sensitive records ILR:

We can’t guarantee that records won’t be ordered to be produced but we will make all the arguments we possibly can, make all the submissions we can and then it will be for the judge to decide. But I think at least the comfort for clients knowing that it’s been through those processes with really careful consideration as to how the law applies at least gives them something.^[594]

Another stated that:

It’s about the experience of the client, that agency and I think testing what that has meant for them to have that agency in that process. Because I think you can’t always influence the outcome of the trial, whether the woman thinks she gets justice … But I think it’s the agency for her and what it means for her to have someone in her court.^[595]

A second aspect of complainer satisfaction relates to the broader issue of complainer wellbeing. There is overwhelming evidence that those who engage with the criminal justice system after making an allegation of rape or other serious sexual offences can find the experience distressing and re-traumatising.^[596] In this sense, an effective ILR scheme is one that does not itself contribute to re-traumatisation, which it might do if it did not operate in a trauma-informed manner.^[597] One interviewee spoke of judging the scheme as successful on the basis they are “trying to make things a human-centred approach, come through that lens and being trauma-informed … that’s what we want”.^[598] In addition to this, however, effective ILR provision may also serve to improve the wellbeing of complainers.^[599] As two of our research participants put it:

I think it’s always important when you do evaluations to be, like, qualitative because sometimes the numbers don’t tell a story. And so like, really drill down into what does that look like for this woman and what did her experience look like? And I think, you know, when we talk about chipping away and small victories, some of it is just that you have made that woman’s experience of one particular part of one particular flawed system a bit more bearable.^[600]

Sometimes I look at what I do and I think this is trivial. But it’s not trivial to the person that cares about it. So, I don’t think there is a way to measure it because … the currency is measured in the dignity of the complainants within the criminal justice system, that’s the measure. We give them options, we give them choice, we give them advice and we give them the ability to have an impact on how re-traumatising this is going to be.^[601]

The two measures are, of course, related. Re-traumatisation may be lessened by the complainer having a sense of agency in decision making.^[602] But it also relates to a sense of having the support of someone who is focused on them, rather than on the broader public interest:

It’s like in that moment, you have an ally and you have someone with you who’s going in to bat for you. And that might be the only time that that woman experiences that from statement to completion.^[603]

Another interviewee described it in the sense of complainers being able to have confidence and trust in the advice that they are being given, knowing that their best interests, rather than the best interests of the prosecution, are being considered:

What’s very, very clear is that survivors really value having someone that they trust to be giving them information from their own perspective. So quite often, we have to tell people things they don’t want to hear, we have to say, sorry, I don’t think that is going to work or I’m afraid you do have to provide that if you want the case to go forward or whatever. We … give people honest advice. Sometimes it will accord with what they want, sometimes it won’t. But even when we give people negative advice, we find people really, really appreciate it. You know, nine times out of ten people say, well, thank you, at least now I know I’ve looked into this or thank you, you’ve explained it in a way which I finally understand and nobody else had explained it to me, so it didn’t make sense.^[604]

Improved complainer wellbeing is important in itself, but it may also contribute to preventing complainer withdrawal from the prosecution process.^[605] Several interviewees mentioned the latter as an evaluative measure. This was especially the case in relation to the ILA schemes that formed part of our research.^[606] It was, however, also mentioned by some interviewees as a measure of success in relation to ILR schemes. One interviewee providing ILR to complainants in relation to sensitive records applications commented that:

What we’re seeing so far is that having the support of a lawyer … in this service really contributes to [complainant] willingness to stay engaged in that prosecution as well, given that they’ve been able to get through this hurdle with that support.^[607]

The point was also made, however, that it is important when assessing the effectiveness of ILR provision to look carefully at complainer wellbeing and the factors that affect it. If introducing ILR had the knock-on effect of introducing inordinate delays into the progress of the case, for example, this has the potential to adversely affect complainer wellbeing, negating or lessening the benefits brought by increased agency. As one of our research participants put it:

I think an effective ILR system would be one that allows the complainer to be able to provide the court with their views on [the application] in a way that allows the court to take account of these when considering the application [But] I think there also has to be a balance in terms of ensuring that it doesn’t interfere with the case proceedings in general, and there isn’t any unnecessary delay or churn involved in the ILR system, which would impact both the complainers, other witnesses, and accused in general.^[608]

9.2.2 System measures

Three system measures were mentioned by interviewees: scheme take-up, substantive legal outcomes and case processing times.

In terms of scheme take-up, several interviewees referred to the number of complainers who accessed the scheme as being an important evaluative measure, either as a measure on its own, or in terms of cost effectiveness (the number of complainers assisted divided by the cost of the scheme). However, it was also suggested that this measure was an incomplete one. This is because a successful ILR scheme might be one in which applications made by the defence (and therefore the number of complainers seeking ILR) decreased, because the knowledge that the complainer would be legally represented would deter the making of frivolous applications or ones that had little chance of success. As one of the policymakers who was involved in designing an ILR scheme in relation to sensitive records stated:

So, I think the biggest thing when we were talking about the scope of this pilot and the success we could have was that it could have a chilling effect on exploratory applications for these types of records. That was probably the biggest outcome we would hope to see … If we could achieve a resetting of the expectations from those seeking those records, that actually victims will fight back and that you’re not going to get away with a fishing expedition for records that you shouldn’t have. That would be the number one outcome we would seek.^[609]

A similar point was made in relation to one of the more established ILR schemes: “I think anecdotally, the other thing is when we’re hearing that people are avoiding accessing or trying to access the records, that’s really important”.^[610] With this in mind, another interviewee suggested that as well as the numbers assisted per se, an equally important evaluative measure is client awareness of the scheme (or more broadly the scheme’s accessibility to those who do need it): “it’s about continual improvement in terms of both the awareness for practitioners, the awareness for clients”.^[611]

A second quantitative measure mentioned by interviewees was substantive case outcomes.^[612] But here too ‘success’ is perhaps not quite as straightforward as the number of cases ‘won’ or ‘lost’. As one interviewee put it in relation to ILR for sensitive records:

I don’t think there is a, sort of, general way to measure whether you’re successful or not. Because … we’re trying to do lots of different things. Sometimes we’re trying to keep the information out, sometimes we’re trying to get the information in. Sometimes we’re trying to get some of the information in.^[613]

Additionally, as has already been discussed, client satisfaction may be positive even in case where the outcome was not the one that the complainer desired, if it increased the sense of agency that the complainer felt they had over the issue.

The final system measure that was mentioned was case processing times. This was not as frequently mentioned,^[614] but it is nonetheless a system measure that would be relevant to any evaluation of ILR.^[615] The detrimental impact of delays on complainers in sexual offence cases has been well-documented,^[616] and delays will of course also impact the accused person and other witnesses in the case.

9.3 Evaluation of existing ILR schemes

Publicly available data on the evaluation of the existing ILR schemes that were examined in this report is almost non-existent.^[617] The researchers identified a small number of evaluations that had taken place and had sight of some of these, but they were not given permission to refer to them in the public domain. In two instances, it is a matter of public note that an evaluation of the ILR scheme has taken place, but despite the best efforts of the researchers and our network of contacts we were unable to obtain a copy of the reports. This lack of transparency is regrettable, as it prevents an international community of practice developing whereby one jurisdiction can learn from another. A recent review of sexual assault victim advocacy noted the lack of an evidence base on what works well in this context and recommended the establishment of a “clearing house could allow agencies to share their evaluation data (without identifying their program), so that practitioners and researchers can identify important trends in advocacy” such as “what is most and least helpful; how well different types of clients are served etc”.^[618] The same point could justifiably be made in relation to ILR.

The only publicly available evaluation of an ILR or ILA scheme in the jurisdictions that we examined is the evaluation of the Sexual Violence Complainants’ Advocate (SVCA) scheme that operated in Northumbria between September 2018 and March 2020.^[619] This was an ILA service, which provided advice and non-court intervention for complainants in relation to requests to access their personal data.^[620]

As noted previously,^[621] the evaluation concluded that the pilot had been successful. A total of 83 eligible referrals were made,^[622] and complainants engaged with the service in 47 of these cases (57%).^[623] The most common type of support offered was advice on data requests (38 cases), followed by attending ABE interviews (20 cases) and intervention (short of court representation, which was not part of the scheme) on data requests (18 cases).^[624] The average time commitment by SCVAs per case was 155 minutes at an average cost of £725.^[625]

The evaluation found a positive effect in terms of the outcome of the request to access data. There were 22 cases where SVCAs challenged data requests, and in eight of the 22 cases where the outcome of the challenge was known, the request was withdrawn or amended to a reduced timeframe or scope.^[626] The overwhelming consensus of stakeholders was that the SVCA project led to greater consideration of third-party requests before they were made.^[627] Only three complainants gave feedback on the pilot, but their comments were all highly positive, in terms of feeling supported by someone who had the legal training to understand, explain and navigate the criminal justice system.^[628]

While not a formal evaluation, there is evaluative material in the public domain relating to the initial pilot of the New South Wales Sexual Assault Counselling Privilege (SACP) service, which took place in 2009. It is reported that solicitors acted for 32 men and women in relation to 116 subpoenas. In terms of outcomes:^[629]

Representation through the project successfully resulted in preventing or limiting access to victims’ confidential records in 91 per cent of subpoenas. Of the 80 relevant subpoenas, access to subpoenaed documents was refused in 11 matters, partial access granted in 50 matters and 12 subpoenas were withdrawn. Full access was granted to the parties in only seven subpoenas.

It was noted that the scheme was very positively received by complainants, but also that prosecutors were very positive about the availability of free representation for complainants, noting that this saved time, allowing prosecutors to focus on other issues.^[630] It was also noted, however, that timely compliance with the legislation’s notice requirements was a problem, with a lawyer involved in the pilot stating that the notice provisions were “honoured more in the breach than the observance” which “creates significant difficulties for complainants”.^[631]

Notwithstanding the absence of formal evaluation, our research participants did on occasion express views on whether they considered ILR in their jurisdiction to be successful. In those jurisdictions that operated a salaried lawyer model, their assessments were overwhelmingly positive. One commented that “there is quite a lot of good feedback and we are seeing that victims are more willing to be engaged and are saying it feels like I’ve got a voice for the first time”.^[632] Another stated that:

I think it’s [a success] and I’m basing that off the kind of feedback I’ve received from our clients … they’re just really positive and thankful for the representation that we can give them … My sense is they feel heard in this and they seem to really trust us to represent them and represent their voice. … So, I think that’s been really positive and I think a sign that it’s a success.^[633]

It does have to be said, though, that these assessments were based primarily on complainant feedback rather than other measures and in some cases were based on only the first few months of operation of the service.

Similarly positive feedback was noted in Northern Ireland about the SOLA scheme. As noted previously, there have been high levels of engagement with the scheme. Of those clients referred, 86% took up the legal support offered.^[634] Of those clients who took part in a Victim Support NI survey, 100% said that advice “allowed them to have a voice in the criminal justice process” and 75% reported improved confidence in the criminal justice system.^[635] One of our research participants stated that “the feedback from service users has been overwhelmingly positive, you know, they’ve come back to say that ... and they’ve continued with the case, that their advice meant, you know, the difference between continuing or not”.^[636] Another noted that the target of 80% of those being referred to the SOLA service remaining engaged with it has consistently been met since 2021.^[637]

In the two jurisdictions that operated a private practice model, research participants were less positive about the service. At least one interviewee in each jurisdiction expressed the view that if they had the choice to go back and create the scheme again, they would choose to operate on the basis of a salaried lawyer model.^[638]

9.4 Chapter summary

This chapter has examined the ways in which the provision of ILR might be evaluated. It drew a broad distinction between victim measures and system measures. The vast majority of our research participants focused on victim measures such as complainer satisfaction as the most important evaluative measures of an ILR scheme. In a sense, this is not surprising, given that most of the comparative interviews were undertaken with those who operated the schemes, not with those from organisations that were responsible for funding them. Nonetheless, it is notable that the two ‘success measures’ that repeatedly came up were increased agency for complainers and improved complainer wellbeing. Both of these can also have a positive impact in terms of preventing complainer withdrawal from the prosecution process. As well as the positive benefits to complainers arising from ILR, however, participants noted that complainer wellbeing might be adversely affected if introducing ILR had the knock-on effect of introducing delays into the progress of the case.

System measures relate to service outcomes, such as substantive legal outcomes, take-up of the service, or case processing times. These can be informative but the first two, in particular, need to be treated with caution. Complainer satisfaction may be positive even where the outcome was not the one that the complainer desired, if it increased the sense of agency that the complainer felt they had over the issue. Figures on the take-up of the service need to take account of the possibility that applications made by the defence (and therefore the number of complainers seeking ILR) might decrease, because the knowledge that the complainer would be legally represented would deter the making of frivolous applications or ones that had little chance of success.

A further finding from the research is that there is a lack of transparency around the evaluation of existing ILR schemes. Publicly available data is almost non-existent, which limits the opportunity for an international community of practice to develop. We have already noted that the establishment of a community of practice for lawyers delivering ILR within the Scottish jurisdiction would be helpful.^[639] In the same vein it would also be beneficial to develop a wider international community of practice among comparable jurisdictions that operate ILR schemes.