Cash-First Programme: interim evaluation - updated

2. Evaluation Approach

Aim of the research

The overall aim of the research is to assess the implementation of Cash-First activities by Delivery Partners, the role of the Learning Partner, and the impact of the Programme as a whole.

The Programme evaluation addresses the following overarching research questions:

• Has the Programme been implemented and delivered as intended?
• What has been produced as a result of the Programme?
• What difference has the Programme made and for whom?

The full list of research questions is detailed at Appendix 3.

Methodology

Approach

A mixed methodology was used to undertake the evaluation. It consisted of qualitative and quantitative methods across the eight Delivery Partners. The study team engaged with a range of individuals connected to the Cash-First Programme to gain a complete understanding of each perspective of the benefits and challenges.

There was some variation in the data collection tools employed in each area, to ensure evaluation suited the approach undertaken locally. The methods undertaken in the first year comprised:

• 18 initial scoping interviews with Delivery Partners, stakeholders and the Learning Partner at the start of the evaluation process in May 2024.
• A further 13 interviews with Delivery Partner representatives; and 17 interviews with Programme stakeholders which took place between September 2024 to December 2024 as projects became more established.
• Quantitative survey of beneficiaries (73 responses). The profile of survey respondents is included at Appendix 5.
• Follow up one to one interviews with beneficiaries to better understand long term impact of the service (10 interviews undertaken 6-8 weeks after completion of the survey). 8 of these 10 beneficiaries are taking part in longitudinal research, which will continue over Year 2 of the evaluation.
• Interviews with non-beneficiaries (17 people that have not received Cash-First payments, due to a wide range of reasons, but may be eligible for Cash-First payments). In order to reach this group, the Evaluator engaged with Delivery Partners to attend three food banks/food pantries and speak with service users on site.
• A desk-based review of administrative monitoring reports included Delivery Partners’ quarterly data submissions; relevant extracts of Delivery Partner quarterly monitoring reports to Scottish Government; Learning Partner updates to Scottish Government; and output reports from learning events.

The research in year one included the development of two case studies covering:

• Strengthening Local Referral Pathways; and
• Engaging People with Lived Experience to incorporate their views into service design and development.

The original intention was to undertake extensive qualitative research with beneficiaries. However, the methodological challenges discussed below meant that interviewee numbers were lower than anticipated. Despite this, the data gathered has allowed the Evaluator to deliver a thorough evidence base for the Year 1 evaluation.

Methodological Challenges and Considerations for Year Two

As the evaluation progressed, it became clear that not all aspects of the methodology would be appropriate for every area due to the differences between the Delivery Partner approaches and the fact that some started delivery later than others. As a result, some planned methods were not feasible in Year 1 or were more limited. The Evaluator worked closely with the Scottish Government and Delivery Partners to adapt the methods and maximise consistency in the evaluation.

Data Challenges

ekosgen provided the Delivery Partners with a data template to be completed quarterly (see Appendix 4). The template asked for data on local Delivery Partners’ activity, including referrals to and from Cash-First services, beneficiaries, and outcomes. All Delivery Partners provided sufficient data for the evaluation, but the level of detail varied. This is because each project had different processes or timescales. Also, local data collection systems were different across areas, and generally, there were challenges across all Delivery Partners monitoring food insecurity activity due to both the wide spectrum of organisations operating in this space and the informal nature of some provision. It was agreed with the Scottish Government that a comprehensive data set would not be obtainable, and that any information provided would be appropriately anonymised and caveated in reporting. As projects progress, it is anticipated that data gathering processes may become more robust and provide more complete data.

Cash-First Beneficiary Survey

The survey was provided to six of the eight Delivery Partners. It was not used in either Glasgow or Edinburgh as these Delivery Partners are taking a systems change approach and so do not typically have individual beneficiaries receiving a Cash-First payment. However, it should be noted that Edinburgh is implementing several tests of change over the duration of the Programme, one of which has resulted in small number of individual Cash-First beneficiaries. This is accounted for in the project data received from the Delivery Partners.

The six Delivery Partners able to participate in the survey were provided with a covering email to send to beneficiaries along with a link to the electronic version of the questionnaire and a privacy agreement. The survey was tested with two Cash-First beneficiaries with lived experience of food insecurity, and two individuals from external organisations with experience in supporting young people and families.

Delivery Partners were asked to distribute the survey in the most appropriate way in their area, depending on the specific structures and processes in place for Cash-First approaches. One Delivery Partner, for example, opted to distribute hard copies, which successfully elicited a good sample of responses.

The questionnaire was designed to be as easy as possible to complete and at the same time, gather the information required to answer the Scottish Government’s research questions. Whilst 73 people completed it, there has been feedback that some Cash-First cohorts did not find it accessible.

The number of responses from each area varied considerably. 85% of the 73 completed survey interviews came from two Delivery Partner areas. This was in part due to some areas having had many more beneficiaries in receipt of a Cash-First payment at the time of the research, and the different stages of development of the Programme in each area. However, some areas with a sizeable cohort of beneficiaries found it challenging to encourage beneficiaries to complete the survey. The design of the questionnaire is being revisited for Year 2 to enhance its accessibility for people who may face barriers.

Qualitative Research with Cash-First Beneficiaries

The original intention was to undertake an extensive Programme of qualitative research with people who had received support through Cash-First across the eight areas. They were to be primarily recruited through the survey but the uneven engagement with the survey meant there was a smaller cohort to draw on for the qualitative interviews.

The Delivery Partners also fed back that many beneficiaries face complex barriers to participating in the qualitative research and it was not feasible to address all of these within the scope and period of the evaluation. Examples may include lack of confidence, concerns about stigma, and issues related to digital exclusion. As a result, 10 initial interviews with service beneficiaries were completed, representing all relevant areas; and a further eight follow up interviews to identify any sustained or potential longer-term impacts of Cash-First services for individuals.

The Evaluators did not experience any challenges in accessing non-beneficiaries and engaging them in the research.

Implications for Year Two Evaluation Methods

The following steps will be taken to inform and plan the second-year evaluation activities:

• Given the accessibility issues experienced by some potential survey respondents, a survey working group will advise and co-produce the beneficiary questionnaire for Year 2. This will draw on the feedback and learning from Year 1. The survey working group will include voluntary representatives from Delivery Partners and the Learning Partner;
• Qualitative research methods and scale will be adjusted, informed by feedback from Delivery Partners and drawing on the learning from qualitative research that has been successfully conducted with beneficiaries. This is likely to include 1-2-1 interviews rather than focus groups, as it was not possible to complete focus groups in Year 1. Feedback from Delivery Partners was that many service beneficiaries are not comfortable in a focus group setting for reasons of privacy and confidentiality, and fear of stigma. The Evaluator will continue to offer focus groups as an engagement method in Year 2 and if appropriate will facilitate this. Qualitative research methods will also include, where appropriate, attending group sessions and events organised by local Delivery Partners and stakeholders; and
• Recognising the range of asks being made of Delivery Partners, there will be improved co-ordination of these asks by the Scottish Government, the Learning Partner and the Evaluator. A more proactive approach to engagement between the Learning Partner and Evaluator is being taken to improve coordination, with regular meetings arranged. A shared project timeline has been developed, which both the Evaluator and the Learning Partner can access and update, and into which Scottish Government will also input. This enables a complete overview of key dates and interactions and will support improved planning in Year 2 along with mitigating any methodological risks.