Gender identity healthcare: evaluation of the impact of Scottish Government funding

2. Research Design and Methodology

An extended description of the research design and methodology for this project can be found in Annex B, while an overview is provided below.

Complex Systems Approach

In order to understand the interactions within and across systems in which gender identity healthcare services are being delivered, we have adopted a complex systems approach. A complex system “is composed of many parts that interact with and adapt to each other and, in so doing, affect their own individual environments.”^[25] Healthcare is increasingly recognised as a complex system, whereby health interventions are based upon interactions amongst many actors, who may have different processes, values, assumptions, objectives, experiences and expertise. This approach highlights the importance of local context, whereby an intervention in one healthcare setting may not have the same outcomes in another.^[26] For this project, we have viewed the delivery of gender identity health services through a systems lens, understanding that decisions are the result of multiple influences, interactions, relationships, opinions and behaviours. This approach has informed our methodological choices, including an extensive programme of interviews with staff and service users in different parts of the system to understand the perspectives of different actors, and the hosting of an engagement event to advance cross-clinic understanding of system change.

Comparative Case Study Method

This project also employs a comparative case study approach. Comparative Analysis is a methodology that seeks to compare the objects of a study across space or time in order to identify patterns of similarity and difference.^[27] We begin from a most-similar comparative research design, which involves examining a small number of similar cases that differ in outcomes.^[28] Our rationale for this approach is that each of the GICs have been developed in accordance with NHS Scotland general national standards, clinical guidelines and Realistic Medicine objectives to improve healthcare delivery. Each of the GICs has also had the opportunity to apply for improvement funding from the SG. At the same time, there may be differences across cases in their application of national standards to local settings. The same methods are employed for each case study, enabling a comparative analysis of outcomes. The aim of this design is to identify key similarities and differences in outcomes across the GICs, and to understand the complexity of context in which the funding interventions have taken place.

Research Questions

This evaluation project comprises a process evaluation and an impact evaluation. The process evaluation explores the ways in which gender identity healthcare services have been delivered by NHS Board. It seeks to determine whether changes in services were implemented in the way intended, and if they resulted in certain outputs. The Impact Evaluation assesses the effects of funding on healthcare outcomes. It provides a comprehensive view of GIC services, evaluating specific impacts (on waiting times, access to services, patient and staff satisfaction) and general impacts (on overall patient care, collaboration and system changes). The main research questions are:

• What have been the effects of Scottish Government funding allocated to NHS Health Boards to improve access to, and delivery of, gender identity healthcare?
• How has the SG funding affected the implementation and delivery of services (process) and patient care outcomes (impact)?

Data Collection Methods

A range of primarily qualitative research tools and methods were used to collect the data required for this evaluation, which are described below.

(i) Review of literature

We conducted a review of primary and secondary literature and documents relating to gender identity services. This is set out in Annex C of this report.

(ii) Review of budgets

We conducted a review of financial information related to the SG improvement funding for the delivery of gender identity healthcare services.

(iii) Theory of Change

We developed a Theory of Change (ToC) framework to guide the evaluation project. In addition, we developed bespoke ToC models for each of the four GICs.

(iv) Questionnaire

We sent out a questionnaire to each of the four GICs requesting information on key data and delivery mechanisms for gender identity services in May 2024. These were completed and returned to us between May 2024 and August 2024.

(v) Surveys

We designed two electronic surveys. The first focused on third sector organisations, GP local medical committees and partner groups supporting trans and non-binary people seeking gender identity healthcare. It was sent to 25 potential partner and referring organisations, and it received 5 responses. In response to a question on whether respondents would like to participate in a follow-up interview, 4 people agreed (see below). The second survey focused on service users, which received 74 responses. In response to a question on whether participants would like to engage in a follow-up interview, we invited 15 respondents to speak with us (see below).

(vi) Research interviews - key stakeholders

We sent out invitations to 65 key stakeholders across each of the four local health systems to be interviewed for this project. Of these, we conducted 36 semi-structured research interviews, comprising multiple staff members from each GIC, staff overseeing the service from the Health Board, staff referring to the GIC (e.g. GPs and the Child and Adolescent Mental Health Service - CAMHS), organisations representing medical professionals (e.g. royal colleges) and organisations to whom the GIC refer for further support (e.g. third sector organisations). To ensure the utmost confidentiality and anonymity to protect staff data, we have opted not to provide references to individual interviews when quoting or paraphrasing from interviews. The interviews lasted 45-60 minutes and were conducted online via Microsoft Teams; they were recorded and managed in compliance with UK Data Protection Act and GDPR regulations.

(vii) Research interviews - service users

We invited 15 service users from our survey respondents agreeing to a follow-up interview, seeking to ensure that people of different ages, geographical locations, and gender identities were included. Of these, we conducted 9 interviews. We offered vouchers to compensate interviewees for their time in line with SG guidance.^[29] To assist this research, we put in place strict confidentiality and data protection protocols when speaking with service users, complemented by our mental health first aid-informed approach to undertaking research with vulnerable people. We set out ethical considerations around including service users in Annex D.

(viii) System Mapping of Relationships

We created visual maps to identify relationships and illustrate the impacts of the funded services on each local health system, as part of the case studies for GICs.

(ix) Case Studies

We developed five case studies on the impact of SG funding on gender identity healthcare services in NHS Lothian (adult services), NHS Highland (adult), NHS Grampian (adult), NHSGGC (adult) and NHSGGC (young people’s services). Each case study provides an overview of the GIC, an analysis of service delivery impacts, and an exploration of factors that enable or inhibit changes to services.

(x) Engagement Event

We hosted an online Learning Event with key stakeholders to advance cross-clinic learning and gain feedback on interim findings from the project. There were 16 participants, working across the four NHS Health Boards delivering gender identity healthcare services. The event presented our interim research findings and included a workshop on ‘futures thinking’ for gender identity services. The learnings from the event are integrated into this report, most notably in the last three chapters.

Ethical Approach

During the Inception Phase of this project, we contacted teams within the NHS to seek advice on our ethical approach and any clearances required. We filled out the initial decision tool on the UKRI Medical Research Council/NHS Health Research Authority website, which suggested that NHS Research Ethics Committee review and approval was not required under GAfREC. We contacted the NHS Research Ethics Committee for further advice, and were informed that ethical review was not required as it was a service evaluation. Ethical guidelines were closely followed at all times throughout the project, which are elaborated in Annex D. The SG internal ethics process was also followed and a Data Protection Impact Assessment (DPIA) was carried out.

Limitations of Methodology

This evaluation project has depended on the participation of key staff and organisations operating in each of the Health Boards. We recognised how busy and pressured services were, and sought to offer maximum flexibility to NHS staff in participating in this project. We had a 55% response rate to our requests for research interviews with key stakeholders (NHS staff, partner organisations and third sector organisations) and a 60% response rate to our requests for interviews with service users. Furthermore, we received 74 responses to our service user survey, and 5 responses to our survey with GPs and third sector organisations. Although the service user survey was self-selecting and therefore not representative of all trans and non-binary population accessing gender identity services in Scotland, the responses gathered through the survey helped to add insights into the lived experiences of service users. The project has also relied on access to key service-related data and documentation relating to the delivery and impacts of services in each GIC. Where this was unavailable from the GICs, we explored other data sources. We were aware that PHS was collecting data on GIC waiting lists during the period of this evaluation, and we have included this data in our case studies below. ^[30] The methodological design did not include counterfactuals, which may be seen as a weakness, in that causal relationships could not be established. However, we have aimed for understanding rather than causality. Finally, the evaluation project has relied strongly on qualitative methods. These are often viewed as more ‘subjective’ than quantitative methods as they tend to rely on experiences and understandings of services, and are not fully representative of the wider population. We believe that these weaknesses are outweighed by the strengths of qualitative methods in collecting rich data that can lead to new insights.

Quality Assurance

Quality assurance (QA) and assessment has been built into all stages of the project to ensure that the research is properly conducted. An external expert was appointed to ensure that the evaluation research design, methods and outcomes were of the highest quality and integrity, and the results were reported accurately and without bias, i.e. following the highest quality benchmarks when conducting surveys (clear questions, ensuring anonymity of participants, transparent data collection and results, GDPR compliance) and semi-structured research interviews (obtaining consent forms, giving clarity on the purpose of the research, ensuring confidentiality, safeguarding participants). QA advice was received throughout this project, with the QA expert regularly checking that the data and deliverables met ethical and data protection standards, and that outputs were technically correct, well laid-out and accessible.