Devolved disability benefits: decision making evaluation

Methodology

Logic model

As outlined in the evaluation strategy, evaluations within this programme of work are supported by the development of a theory of change. This is represented graphically within a logic model. It shows the trajectory from the direct impacts of the policy commitments (short-term outcomes) to the outcomes that arise as a result of these, and other policies related to the delivery of disability benefits (medium-term outcomes). Finally, the long-term outcomes relate to broader impacts on views and experiences with Social Security Scotland.

The logic model for this evaluation was developed collaboratively with policy colleagues and analysts, is detailed below.

Short-term outcomes:

• Individuals feel that their experience was person-centred
• Individuals feel that they are trusted
• Individuals feel that the right decision was made first time
• Individuals understand their decision and why it was made
• Individuals understand why a consultation was needed
• Individuals understand the outcome of the consultation
• Individuals feel that their consultation was proportionate, trust-based, and in line with their needs
• Individuals feel that their review period is appropriate for them
• Indefinite awards and person-centred review periods allow for financial planning
• Individuals understand the review process and engage with it
• Individuals understand the need for a review or why no review is necessary
• Individuals feel that the review was light touch and avoided unnecessary burden

Medium-term outcomes:

• Individuals’ experience is in line with dignity, fairness, and respect
• Individuals trust in our decision-making
• Individuals understand that, where entitled, receiving disability benefits is a human right
• Individuals understand what is happening and why at each stage of decision-making
• Individuals do not feel undue stress or worry
• Individuals have a positive attitude towards Social Security Scotland and the Scottish social security system
• Individuals feel more in control of their finances and can plan accordingly
• Individuals understand their responsibilities and report any change in circumstances in good time

Long-term outcomes:

• Individuals trust in Social Security Scotland
• Individuals have a positive relationship with Social Security Scotland
• Public services treat people with dignity, fairness, and respect
• There is reduced stigma regarding disabled people and people with long-term health conditions
• Individuals have greater financial security and stability

Data Sources

The evaluation draws on data from multiple data sources to evidence the outcomes outlined in the logic model. Collectively these sources aim to provide a comprehensive understanding of the experiences of clients and staff. These are outlined below.

Commissioned Research

IFF, an independent social research agency, was commissioned by the Scottish Government to conduct qualitative research with Social Security Scotland clients and staff from February to April 2025. This involved focus groups and interviews with clients as well as observations and interviews with Social Security Scotland staff. Findings were analysed and presented thematically. The report of the commissioned work is published in Annex A.

This report draws on the commissioned report but also builds on this work. To achieve this, Scottish Government analysts drew on the transcripts from the commissioned research to provide further detail and context to some of the evidence, as well as to identify evidence that aligns with the aims and objectives of this report e.g. mapping the evidence onto the anticipated outcomes.

Qualitative research was conducted with Social Security Scotland clients and staff from February to April 2025. This involved focus groups and interviews with clients as well as observations and interviews with Social Security Scotland staff. The groups and recruitment approach are described in more detail below.

Focus groups and interviews with clients

Research was conducted with clients who had experienced the decision-making process when applying for or undergoing a review of their disability payment under Social Security Scotland. A total of 74 client participants across ADP, CDP, and PADP applications, reviews and consultations, took part in 16 focus groups and seven individual interviews. This included a mix of clients with successful and unsuccessful outcomes in their applications and clients with an increase, no change, or decrease in their award at reviews. Focus groups were stratified equally by benefit type (ADP, CDP, and PADP), policy area (applications, reviews, review periods, and consultations), and by outcomes (including increase, no change or decrease in review award and successful or unsuccessful application). Scottish Government invited eligible clients to take part via Social Security Scotland’s Client Panels as well as Client Satisfaction Survey recipients, and interested clients contacted IFF directly.

Observations with Case Managers and discussions with Social Security Scotland staff

A total of 46 staff participants took part in the research, which took the form of in-depth interviews, group discussions, and case observations.

The observations consisted of Case Managers ‘thinking aloud’ their decision-making process with a researcher, while working through previous decisions they had made, including decisions on applications, reviews and consultations.

The discussions element consisted of 12 in-depth interviews with Case Managers, and eight group discussions with Quality Support Staff, Decision Support Staff, Decision Team Managers, and Health and Social Care Practitioners.

Staff were recruited to take part internally via team leaders. Interested individuals were asked to contact IFF directly.

Staff glossary

This section outlines the roles of the different staff groups that took part in the commissioned research. Note that the roles below are those relevant to this evaluation specifically. Some of the staff below have additional roles that are less directly linked.

• Case Managers – those who have overall responsibility of determining entitlement to a disability benefit. The main decision-makers.
• Decision Team Managers – line managers to Case Managers, providing a supporting role and working to ensure quality and consistency of decision-making within and across their teams.
• Quality Support Staff – randomly select cases with decisions already made and conduct quality assurance checks on the decision-making underpinning these cases. Raise anything that they flag to decision team leaders.
• Decision Support Staff – support Case Managers with decision-making, specifically, with understanding the guidance and identifying any areas that might need further insight from Scottish Government Policy and Legal colleagues.
• Health and Social Care Practitioners – referred to as ‘Practitioners’ throughout the report for ease. Can provide further support to Case Managers in decision-making by providing further professional insight into cases, for example through their health, care or social work expertise. These staff also conduct consultations to help gain a better understanding of the impacts of clients’ health conditions and needs, to aid in Case Manager decision-making.

Caveats and things to note

A full list of caveats and things to note has been included in the commissioned report in Annex A. It is worth restating here though that the qualitative research aimed to capture a variety of experiences across clients and formal representatives. However, the findings are not representative of all clients. This is because the overall sample was small, and participants were self-selecting, meaning that they actively chose to take part, as opposed to being randomly selected.

It should also be noted that the primary source of data in this work was the qualitative commissioned work. The aim of qualitative research is to explore the experiences of groups of people. Qualitative research is not designed to be representative, and therefore these findings cannot be said to be generalisable. In terms of interpretation of data throughout the report, qualitative data is generally reported thematically and not quantified. In this report, when a finding is presented along the lines of ‘clients felt’ this means it was a dominant theme, and it if it is preceded by a qualifier, for example, ‘some clients felt’, this indicates it was a less prevalent theme. However, where one participant has raised something linked to the main themes, yet still distinct, this has been referred to as, for example, ‘One client felt that…’.

Client Satisfaction Survey

The report draws on quantitative data from the Social Security Scotland Client Satisfaction Survey 2024-25 report. The main reason for using Client Satisfaction Survey data was to gain an insight into the decision-making process particularly with regards to applications and consultations. The Client Satisfaction Survey is administered to all those who have applied for benefits under Social Security Scotland or who have had their benefits transferred from the DWP. It collects equalities and socio-economic information from respondents. It also asks about their experience of Social Security Scotland and the decision-making process.

The Client Satisfaction Survey data presented in this report has a reference period of April 2024 and March 2025 and is based on responses from clients who responded to the survey between 17 July 2024 and 31 March 2025. In total, 6481-out of 13248 individuals had responded to questions that were identified as relevant to the decision-making process for the purposes of this evaluation. This is with the exception of consultation questions which were only asked of ADP clients, with responses from those who reported having a consultation ranging from 610-625. The findings here are based on data provided by these respondents.

Please also note the following technical points about how the quantitative Client Satisfaction Survey findings are presented throughout this report:

• The number of respondents providing a valid answer to each individual question/statement varied slightly. The results do not include respondents who skipped the question/statement.
• All data, including equalities and demographic data is self-reported by respondents.
• Further breakdowns were included in analyses based on our knowledge of common differences in experience as well as our commitment to understanding differences according to different equalities data. These breakdowns included: outcome success, disability benefit, gender, age, ethnicity, deprivation (SIMD), rurality, and type of complexity (see Annex B). However, to note that these breakdowns exclude ‘prefer not to say’ responses and only statistically significant equalities breakdowns are presented, where the test of statistical significance (p-value) is equal to or less than .01.
• Most results to the closed questions are rounded to whole numbers. As such, results may not sum to 100% due to rounding.
• Pension Age Disability Payment (PADP) is not included in the data as PADP was not included in the Client Satisfaction Survey until it launched nationally in April 2025 which is outside the 2024-25 collection period.
• The survey results provide rich insight into some of the experiences of clients who applied for ADP or CDP. However, we cannot assume that the results represent the views of these clients as a whole. It is reasonable though, given the number of responses, to treat the findings as indicative of general views.
• The data included in this report was for new applicants only. While the Client Satisfaction Survey collects the data, this report excludes data for those who have had their benefits transferred over to Social Security Scotland.

Client Panels – Members Survey

The report draws on quantitative data from Social Security Scotland’s Annual Client Panels Survey Report 2024. The main reason for using Client Panels data was to obtain further insight into the decision-making process particularly with regard to reviews and review periods. The Client Panels are made up of people who responded to the Client Satisfaction Survey and gave consent to be contacted for further research.

The Annual Client Panels survey was live from the 20th November 2024 to the 18th December 2024. All questions asked respondents to think about their experiences in the last 12 months. Overall, the survey received 1,583 responses, 69% of respondents had received ADP in the previous 12 months and 10% had received CDP in the same time period.

Please also note the following technical points about how the quantitative Client Panels survey findings are presented throughout this report:

• PADP clients have not been included in the Client Panels Survey 2024 as PADP was not launched until April 2025, after the research had been completed.
• For scheduled reviews, the responses reported were based on 230-235 responses. These were individuals who had indicated that they had completed a review form within the last 12 months, and they had either received a decision on that review or were still awaiting one. There are less than 100 responses from CDP respondents, so where responses from CDP and ADP respondents are mentioned separately, the proportions need to be interpreted with caution.
• For unscheduled reviews, the responses reported were based on 42-76 responses and so should be interpreted with caution. These were individuals who had indicated that they had reported a substantial change of circumstances (i.e., change(s) that can impact payments) in the last 12 months. However, it should be noted that not all of these change of circumstances had been processed at the time of the research, with two thirds (66%) of respondents saying that it had been processed and one third (35%) saying that it had not yet been processed. Results for these reviews will also be presented by benefit (e.g., CDP and ADP). The Annual Client Panels Survey 2024 report also presented these findings by whether or not individuals had their benefit case transferred in the last 12 months. For ADP though, it should be borne in mind that some non case transfer respondents may have had their case transferred more than 12 months ago. CDP case transfer breakdowns were not presented, as numbers were too small to allow for comparison. However, this evaluation report will combine these case transfer proportions and present them as e.g., XX/XX%. Note that respondents whose benefit was transferred in the last year always represented the lowest proportion of respondents and those who had not recently experienced case transfer represented the highest.
• The number of respondents providing a valid answer to each individual question/statement varied slightly. The results do not include respondents who skipped the question/statement.
• Most results to the closed questions are rounded to whole numbers. As such, results may not sum to 100% due to rounding.

Charter Research – Staff and Partner Surveys

This report also draws on quantitative and qualitative evidence from the 2024-25 Social Security Scotland Charter Research staff survey. All staff in Social Security Scotland were invited to complete a survey for this research which was issued in March and April 2025. Overall, 999 staff took part, a response rate of approximately 23%.

A survey was also sent to a range of partner organisations. Partners (also known as ‘stakeholders’) are people who, as part of their job, support clients to use Social Security Scotland’s services or collaborate with Social Security Scotland to inform how the service is delivered. The survey asked partners about their experiences with Social Security Scotland in 2024-25, The partner survey ran in May and June 2025 and received 196 responses.

Please note the following technical points about how the quantitative Charter Research survey findings are presented throughout this report:

• The survey includes responses from all staff who took part and so findings represent a much larger and more varied sample of staff within a number of different roles, including across all benefits, in comparison to those included in the commissioned research.
• A similar caveat can be drawn for partners, with experiences across benefits reported. However, almost all (97%) of partner respondents who worked directly with clients had supported clients with Adult Disability Payment and two-thirds had supported clients with Child Disability Payment (66%).
• The findings represent the views of staff and partner survey respondents only. It therefore cannot be assumed that the results represent the views of all Social Security Scotland staff or partners.
• Respondents were asked to consider their experiences during the 2024-25 financial year and therefore the findings do not reflect any development activity within the organisation since the surveys took place.

Official Statistics

Social Security Scotland collects information on applications, payments, and clients in the process of delivering the benefits. Some of this information is published online as Official Statistics in development. Official Statistics in development are new or existing statistics which will be tested with users in line with the standards of trustworthiness, quality and value in the Code of Practice for Statistics. The following Official Statistics publications are used as sources of evidence in this report:

• Social Security Scotland - Child Disability Payment statistics to 30 June 2025
• Social Security Scotland - Adult Disability Payment statistics to 31 July 2025
• Social Security Scotland - Pension Age Disability Payment statistics to 31 July 2025

To coincide with the timing of the other data sources, data was drawn from Official Statistics from April 2024 to March 2025 for CDP and ADP. For PADP, statistics presented represent the period from October 2024 to March 2025 to also coincide with the launch of the pilot of this payment which was rolled out across five Local Authorities.

This publication is hereafter referred to as “Official Statistics”. Please note the following technical points about how Official Statistics are presented throughout this report:

• Data combines proportions for applications and reviews
• Figures are rounded for disclosure control and may not sum due to rounding
• Where stated, secondary analysis has been conducted on rounded figures from published Official Statistics
• Due to its date of launch, there is currently no available published data on the number of appeals that have been made or allowed or change of circumstances reviews that have been completed for PADP.

Additional considerations

This section highlights some key considerations in addition to the ones mentioned above for each of the data sources. These are bullet pointed below.

• This report sometimes disaggregates reviews into unscheduled and scheduled reviews. Unscheduled reviews are those that are not planned and are triggered by clients reporting a change of their circumstances to Social Security Scotland. Scheduled reviews are those that are planned and the date has been determined by Case Managers when awarding a disability payment or reviewing a disability payment award. They are triggered when the review pack is sent to the client.
• Some of the outcomes have more evidence for them in comparison to other outcomes. This is highlighted for those that have the most vs. the least amount of evidence throughout the report. For the commissioned research, the reasoning for this was because some outcomes generated more discussion from individuals than others.
• The report mentions the use of balance of probabilities among the staff findings. Decisions can be made by Social Security Scotland based on the balance of probabilities. It is a way to describe the level of certainty about any decision made throughout the decision-making process, by requiring decision-makers to determine whether information is more likely than not to be a fact.
• For quotes from clients specifically, this report attributes them to the focus group that the individual was recruited to based on internal data and screening measures in place. However, where it is clear from what the individual has said that they do not fall into that category (e.g., unsuccessful applicants who have subsequently been successful), attributes have been altered to be as accurate as possible.
• Regarding unsuccessful applicants or clients who received a decrease in their award, it is important to note that these clients knew the outcomes of their applications or reviews when taking part in the commissioned research as well as the Client Satisfaction Survey 2024-25 and so these outcomes will have likely influenced their overall perceptions of their experiences.
• Throughout the report there were some clear differences in beliefs and perceptions of experiences due to the award outcomes people had received. However, this was not always clear cut. One potential reason for this was because of the way that review awards had been categorised e.g., focus groups either represented those who had received an increase or no change in their review award or those who had received a decrease in their review award. It became clear in focus groups that some of those who had received no change to their award had indeed wanted an increase in their award and therefore did not agree with the decision. It is difficult to directly separate these people out from the aggregated data and therefore analysts advise keeping this information in mind when interpreting the findings in instances where differences in outcomes are discussed.
• Clients could be someone who is acting on the individual’s behalf who is disabled or experiencing the condition or situation. For example, many CDP clients who took part were parents or guardians acting on behalf of their child. The decision was to ask about the person who applied or who undertook the review even if it was on someone else’s behalf, as those are the people experiencing the policy processes.
• Medium and long-term outcomes are likely impacted by other factors. The report aims to provide an insight into how policy changes have contributed to these outcomes where possible by linking up the policy commitments and short-term outcomes with the medium- and long-term ones. However, some long-term outcomes necessarily require more long-term data. An evaluation of these outcomes therefore needs to be explored in combination with the full suite of policy changes across the disability benefits.