Disability and Carers Benefits Expert Advisory Group - disability evaluation strategy: response from ministers

Thank you for your letter dated 14 October 2021 regarding the, now published, Disability Assistance Evaluation Strategy. I understand that you also met with officials to discuss an early draft of the strategy report.

Please accept my apologies for the delay in responding. I am grateful for your patience.

Your recommendations were considered and factored into the further drafting of the strategy report prior to its publication. Please see my responses to each of these below, in turn.

Purpose of the evaluation and scope

Recommendation 1: Set out more clearly within the disability assistance evaluation strategy document the stated purpose(s) of each form of assistance under evaluation and the ways in which the strategy will evaluate the degree to which the purpose(s) are being achieved. If this is not to be evaluated as part of this strategy, it would be helpful to include information on how this will be evaluated and how any relevant findings from this evaluation will inform that wider evaluation.

I partially accept this recommendation.

While the evaluation strategy has now been published, the research and evaluation team within the SSAFE (Social Security Analysis, Forecasting, and Evaluation) unit are still working toward a similar evaluation framework as in Wave 1. That is, we are aiming to create logic models that display policy actions and goals for each evaluation project (e.g. decision-making, case-transfer, eligibility for child disability payment, and terminal illness; see page 9 of the Evaluation Strategy). These evaluation projects reflect policy changes that have been put in place to improve people’s experience of the devolved disability benefits. The evaluation strategy focuses on how these initial changes impact on experiences of the devolved disability benefits. For each evaluation project, we will link evaluation activity to policy aims and action.

Recommendation 2: Consider where learning from the evaluation of Scottish Child Payment could be taken into account, given the similarities of volume and frequency of payment.

I accept this recommendation.

We are happy to consider lessons across all benefits, both in terms of impact and process. We will also use an iterative approach in our evaluations, where new outcomes and themes that emerge from early research activity will inform future evaluation.

Recommendation 3: Include evaluation of take up and poverty reduction for each of the disability benefits within scope of the disability evaluation strategy, or make clearer the connections that will be made with any separate evaluation of take up and poverty reduction.

I partially accept this recommendation.

The report was amended to directly reference that the evaluation strategy will utilise different sources of information that shed light on the effectiveness of benefit delivery from a policy perspective and this includes ongoing work to measure the take-up of Scottish social security benefits. Measuring take-up requires successfully identifying the eligible population for any given benefit which is complex and difficult to achieve, particularly for disability benefits due to challenges in aligning available data on disability prevalence and impact with eligibility criteria for benefits. The Department for Work and Pensions do not publish any take-up rate estimates for their disability benefits, however, this autumn we will publish an update to our estimated take-up rates for Scotland’s low income benefits and this will include a progress update of the work we are doing to develop a methodology to estimate take-up rates for Scotland’s disability and carer benefits. The qualitative commissioned research will also explore issues such as financial impacts of the disability benefits as well as confidence in Social Security Scotland and willingness to apply for other help that is available. Though these are not direct nor representative measures of poverty or take-up, they will give some insight into proximal impacts.

Recommendation 4: Include evaluation of the efficacy of signposting and provision of advocacy support for each of the disability benefits within scope of the disability evaluation strategy, or make clearer the connections that will be made with any separate evaluation of the efficacy of signposting and provision of advocacy support.

I partially accept this recommendation.

Signposting and the provision of advocacy support are not processes that are specific to disability benefits. As such, the efficacy of these processes should be included in wider Social Security Scotland evaluation work so that a more accurate insight into their effectiveness can be provided. Nonetheless, it is likely that across the thematic evaluation projects exploring experiences of how the benefit have been delivered, there will be data gathered covering these aspects. The first opportunity to do this will be in the assessment of the use of supporting information, largely learning from the first year’s operation of Child Disability Payment as well as the initial operation of Adult Disability Payment. In this project, we can investigate whether applicants were signposted to any help with supporting information or to further support with this aspect of the application process, such as advocacy support. The remaining project evaluations will allow for further exploration of signposting and provision of advocacy support.

Recommendation 5: Set out more clearly within the disability evaluation strategy document how client wellbeing will be considered, specifically within each of the thematic projects.

I partially accept this recommendation.

Improving client health and wellbeing is a key policy goal for the devolved disability benefits, and will likely feed into the medium- and long-term goals illustrated within each evaluation’s logic model. The way in which this will be addressed will likely vary across different evaluations, and will be scoped at the appropriate time for each thematic evaluation. Furthermore, the qualitative commissioned research will likely be the only research that asks about health and wellbeing in relation to disability benefits explicitly. Data from other sources, the Client Satisfaction Survey.
Management information and Official Statistics will allow further exploration of the health and wellbeing impacts of disability benefits.

Taking a human rights based approach

Recommendation 6: Provide greater clarity within the disability assistance evaluation strategy document that it is seeking to take a rights based approach and how this will be achieved.

I accept this recommendation.

Lines were added to the strategy about what we mean by person-centred approach and how this will need to be counterbalanced with the approach to fairness across the social security system. Lines were also added on addressing any issues raised in impact assessments in the initial policy development phase, picking up human rights elements where relevant.

Recommendation 7: Ensure there is space within the disability assistance evaluation strategy for personal testimony from those with disabilities.

&

Recommendation 8: Consider specific focus on the impact of the policy divergences having been co-designed with experts including this Group and those with lived experience.

I accept these recommendations.

Lines were added to the Evaluation Strategy concerning people with lived experience who participate in both the commissioned research and Social Security Scotland research. These stated that those with lived experience will be engaged both quantitatively and qualitatively in order to assess the effectiveness of Social Security Scotland by understanding those who access it. Other lines have also been added which state that impact evaluations will be conducted to understand whether the service has been designed with client needs in mind and delivered with dignity and respect.
 
Recommendation 9: Consider starting the evaluation of general decision making earlier in the overall and extending the timescale for this project in order to ensure there is enough scope to explore experiences across equality groups and the extent to which human rights have been upheld in practice.

I partially accept this recommendation.

Information that can shed light on decision making will be gathered from the earlier projects on supporting information and client consultation, ahead of more formal assessment. We will also be developing data reporting as it becomes clearer what the management information can support in terms of analysis and reliability. In the earlier evaluations, we will identify any further issues that can be monitored more effectively. This will allow us to conduct a more robust evaluation of the general decision making process when the time comes. It will also allow for more time to pass, which will mean more robust data on equality groups will be available. For example, as time passes, this will allow for more data to become available, particularly for smaller equality groups, allowing for a more representative picture to be captured within evaluations.

Recommendation 10: The Group would welcome further information on the Scottish Government’s intentions to use the disability assistance evaluation to explore opportunities for collaboration with academic institutions.

I accept this recommendation.

Academic institutions are welcome to express an interest in the likely projects, and we will endeavour to include appropriately experienced contractors, so identified, in tender exercises.

Recommendation 11: The Group would welcome further information on the intended timescales for the review of Adult Disability Payment and how that review will take on board the conclusion of the disability assistance evaluation.

I accept this recommendation.

The Disability Benefits Policy Unit will communicate information on the independent review of Adult Disability Payment to the Group at an appropriate time. It is the intention that all available evaluation findings are shared with the independent review to inform both its work programme and findings, as the review deems appropriate.

Conclusion

I trust you find these responses helpful. My officials would be happy to further engage with the Group on any of the issues discussed. Once again, I would like to take this opportunity to thank you for your continued support as we work to further develop a social security system that delivers with dignity, fairness and response.

Best regards,

Ben Macpherson