Social Security Charter Review: research findings

19. Revising the Charter Measurement Framework

This section outlines the approach to reviewing the measures that make up the framework.^[21] The work resulted in a set of revised measures which will be considered by Social Security Scotland and the Scottish Government following the parliamentary procedure to conclude the Charter review process. As such, a list of proposed changes has not been included in this report.

19.1 Identifying areas for improvement

Within Social Security Scotland, the Charter Measurement and Client Panels team lead on producing the annual framework publication. This team work closely with other analysts and policy officials to gather evidence for the framework each year. This team provided an overview of each measure in the framework and their reflections on issues which have affected reporting. This input provided a starting point for a detailed consideration of each measure, its relationship to the Charter commitments, and the data required to provide evidence to report on it. Discussions with analytical and policy staff identified a number of areas for potential improvement.

Some measures are very similar in meaning, or come at the same concern from multiple different perspectives. This adds to the number of questions required to ask clients, particularly in the Client Survey. This can feel repetitive for clients responding to the survey and requires analytical staff to spend additional time analysing and making sense of the data. This duplication also adds to the length to the published framework, potentially making it a less accessible document. For context, the 2022-23 Client Survey received over 30,000 responses and the 2022-23 framework is 130 pages long and includes over 60 tables of numerical data as well as a significant volume of qualitative data.^[22] An example of measures which cover similar themes are presented in the box below.

Measure 1: X percentage of clients said they were treated with kindness

Measure 2: X percentage of clients said they felt trusted by staff

Measure 3: X percentage of clients said staff listened to them

Measure 4: X percentage of clients said they felt staff did all they could to make them feel comfortable

Tabe 18.1 shows an example of where clients are asked multiple questions on the same theme in the Client Survey (rows) but analysis shows close consistency in responses. For example, clients are asked two questions about whether they agree or disagree Social Security Scotland are open and honest. Responses show 89% of respondents answer both questions with the same response.

Table 18.1: Examples of 2021-22 Client Survey questions exploring the same theme

Thinking about your experience with Social Security Scotland since [date], how much do you agree or disagree with the following...

Question 1

• Social Security Scotland treated me with dignity - 91% same response
• Social Security Scotland is an open organisation - 89%

Question 2

• Social Security Scotland treated me fairly - 91%
• Social Security Scotland is an honest organisation - 89%

Question 3

• Social Security Scotland treated me with respect - 91%

Analytical staff also raised concerns about some measures which do not easily match up with existing management information, or do not align with operational practice. An example of this is a measure looking at the percentage of Social Security Scotland’s Local Delivery venues which meet a checklist of accessibility requirements, as shown below.

Measure 52: X percentage of social security offices pass the accessibility checklist (Checklist to be agreed by disabled people with lived experience)

This information is unavailable for many venues and also does not reflect that venues for client meetings are selected based on the needs of the client. For example, for a client who doesn’t drive, a venue that is on a bus route may be more accessible for them than one that has accessible parking. A more meaningful measure may be whether a client’s accessibility needs were met when meeting with staff.

Another example raised by analytical staff was measures about payments and processing times for benefit applications, below.

Measure 31a: X percentage of clients paid on time from the first time

Measure 31b: X percentage of applications processed within X days

Measure 32: X percentage of clients paid correct amount from the first time

There is duplication here between measures 31a and 32. Measure 31b has been historically reported against a timescale of 10 days, which only aligns with the processing timeframe for one Social Security Scotland benefit, Funeral Support Payment. Analysts in Social Security Scotland have worked together to establish a better reporting practice for application processing times for inclusion in the frameworks to date.

Another concern about the existing framework was the multiple layers of analysis needed to report data for some measures. For example, the measure asking for examples where clients faced discrimination below.

Measure 6: X percentage of clients said staff did NOT discriminate against them

Measure 7: X percentage of clients who experienced discrimination said they felt able to challenge it

Measure 8: X percentage of clients who felt discriminated against that did challenge that discrimination

Measure 9: Examples of good practice AND examples of discrimination (if any) AND Social Security Scotland action taken

These measures require four levels of analysis as follows:

• first level: ‘did the client face discrimination’
• second level: where yes, ‘did Social Security Scotland take steps to remedy this’
• third level: where yes, ‘was the client satisfied’
• fourth level: ‘provide examples’

This type of measure poses challenges in that it requires multiple questions to be asked of clients on a single theme. It also reduces the number of respondents to such small numbers that it can be difficult to either meaningfully report on, or report in a non-disclosive way.

It is essential that individual clients cannot be identified in analytical data. Some measures would be best answered through matching client responses with client case information. For example, to explore information about complaints or re-determinations. This would however, breach the ethical protocols in place to ensure participation in research is anonymous and which separate analytical staff and staff who make decisions. Clients who take part in research are reassured that their participation is voluntary, anonymous, and staff involved in processing applications or making payments will not know they have taken part.

In addition to publishing the Charter Measurement Framework on an annual basis, the Scottish Government and Social Security Scotland also publish a range of other information about social security in Scotland.^[23] Although there is a legislative requirement for most of these publications, it is worthwhile for the review to consider how far existing published data is repeated in the annual framework.

Lastly, there were some concerns about how the measures in section four of the framework are reported. These measures evidence the client-centred, collaborative, and joined-up approach which underpins how the Scottish Government design and deliver the social security system in Scotland. These measures are less straightforward to measure and rely more on narrative explanations compared to the data used for other measures. Some staff felt this led to section four feeling more “woolly” than the rest of the framework. There was also a question about the balance in the framework between operational and policy focused measures. Measures about delivery span three separate sections, compared to just one on the policy aspirations for the social security system.

19.2 Criteria for change

The findings from discussions with staff were used to develop a set of criteria for reviewing and updating the framework. These criteria were discussed by the client core group at workshop four with broad support for the approach. These are outlined below:

• Bring the framework in-line with proposed changes to the Charter. This includes changes to the structure and wording of commitments.
• Measures should be consolidated to avoid duplication and reduce burden on clients and partner organisations who take part in research.
• Measures should be in-line with Social Security Scotland working practices. This will reduce unnecessary staff time spent on data gathering data for measures which could be evidenced in a more meaningful way.
• Measures should be updated (or removed) where data is not captured or where there is not enough data to report meaningful findings.

19.3 Proposed revisions

As outlined above, a list of proposed revisions has not been included with this report. The following information provides some context about the nature of the proposed changes to date. Final revisions will reflect changes made to the Charter as agreed by the Scottish Parliament in 2024.

Firstly, it is suggested that measures are re-ordered to reflect the structure in the proposed revised Charter. Also, the wording of measures should be updated to reflect the terminology used in the revised Charter. Proposed updates also include splitting out measures which previously covered multiple concepts. This makes it easier to evidence measures and easier to understand reporting.

Potential updates to the measures in the first three sections of the framework included revisions to measures which call for qualitative data to give examples. By stating that examples will be presented where available, the proposed changes take into account that not all experiences may be represented among those who take part in research. For example, clients who have experienced an appeal or making a complaint. Further, where participants have experience of certain processes, it might be impossible to use these examples without compromising the anonymity of participants.

Proposed revisions also update measures to align with previous reporting. This covers measures where interpretation has been used to provide meaningful reporting in previous frameworks. These measures relate to call waiting times, application processing times and staff knowledge about how clients can challenge decisions.

Input from officials who work on social security policy within the Scottish Government was used to produce recommended changes to the measures in section four of the framework. Researchers who work on the evaluation of the social security system considered the overall ambition of the section. It was considered if it is appropriate for the measures within this section to be more subjective or if changes were needed to make it more tangible. It was agreed that it is more appropriate for evidence to be more narrative in section four than in other measures. Officials also considered what changes may be needed to ‘future-proof’ the measures in this section for the next five years. Proposed revisions are mainly small changes to the wording of original measures.