Gender identity healthcare: evaluation of the impact of Scottish Government funding

Annex B: Research Design and Methodology - Extended

This is an extended description of the research design and methodology for this project. The project has adopted a complex systems and comparative approach to understanding the impact of the SG improvement funding to Health Boards on access to, and delivery of, gender identity healthcare services in Scotland.

Complex Systems Approach

To understand the many interactions within and across systems in which gender identity healthcare services are being delivered, we have adopted a complex systems approach. A complex system “is composed of many parts that interact with and adapt to each other and, in so doing, affect their own individual environments.^[121] Healthcare is increasingly recognised as a complex system, comprising different levels of interacting agents. This approach highlights the importance of local context, whereby an intervention in one hospital or healthcare setting may not show the same outcomes in another.^[122] The approach also underscores that health interventions are not a single mechanical process; they are also delivered into complex social systems based upon many interactions between actors. There are many different actors involved in delivering healthcare improvements with numerous pathways, with different, and equally legitimate, perspectives on how to make things better, and with different value bases, assumptions, objectives, experiences and levels of expertise. Finally, there are influencing factors around organisational norms, capacity, resources, and a range of other external pressures such as media attention and political contestation.

For this project, this has meant viewing the delivery of gender identity health services through a systems lens, understanding that decisions are the result of multiple influences, interactions, relationships, opinions and behaviours. Healthcare decisions therefore cannot be understood in isolation or as the outcomes of a linear, rational, or discrete process. Instead, it is more useful to understand health improvements as the intersection of multiple interrelated systems, whereby a system embodies “a set of elements connected together which form a whole, this showing properties which are properties of the whole, rather than properties of its component parts.”^[123]

This approach has informed our methodological choices. As we explain below, in order to understand the perspectives of different actors involved in delivering healthcare, we have undertaken an extensive programme of semi-structured interviews with staff involved in different parts of the gender identity healthcare system in Scotland. This includes collecting the views of key stakeholders outside the ‘core’ of the service delivery system (i.e., the clinics), including third-sector organisations and service users. We employed surveys and interviews to understand the care pathways created by clinics, and how these are experienced by service users. We also hosted a learning engagement event, to advance cross-clinic understanding of system challenges and enabling factors. These methods have allowed us to map out relationships in the system and explore the values, assumptions and other factors at play when health interventions are designed and delivered, providing a whole-system perspective.

Comparative Case Study Method

This project employs a comparative case study methodology to explore the impacts of the Scottish Government improvement funding to NHS Health Boards on the delivery of gender identity healthcare at Scotland’s GICs. Comparative Analysis is a methodology in social science that seeks to compare the objects of a study across space or time in order to identify patterns of similarity and difference.^[124] We begin from a most-similar comparative research design, which involves examining a small number of similar cases that differ in outcomes.^[125] Our rationale for this approach is that each of the GICs have been developed in accordance with NHS Scotland general national standards, clinical guidelines and Realistic Medicine objectives to improve healthcare delivery. Each of the GICs has also had the opportunity to apply for improvement funding from the SG, in line with the stated objectives of the funding. At the same time, there may be differences across cases in their application of national standards to local settings, regarding referral criteria and pathways, staffing, local partnerships, and delivery mechanisms. The same methods are employed for each GIC-based case study, enabling a comparative analysis of outcomes. The aim of this comparative design is to identify key similarities and differences in outcomes (the delivery of care) across the GICs, and to identify factors that may help to explain any difference in the impacts of the funding on gender identity healthcare.

The case study method enables a greater understanding of the complexity of context in which the funding interventions have taken place, an exploration of how the SG funding interventions have affected different parts of the system in each case and how they relate to one another (referral services, GICs, partner organisations, patients and their families, Health Boards, NHS Scotland), and how the system has adapted. Moreover, the comparative analysis provides detailed insight into variations in the processes, relationships and constraints facing actors when delivering services.

Research Questions

The main research questions for this project are:

• What have been the effects of Scottish Government funding allocated to NHS Health Boards to improve access to, and delivery of, gender identity healthcare?
• How has the SG funding affected the implementation and delivery of services (process) and patient care outcomes (impact)?

Beyond this, we have two sets of more detailed research questions for this project: one set for the process evaluation, and one for the impact evaluation. The Process Evaluation explores the ways in which gender identity healthcare services have been delivered by NHS Boards in Scotland. It seeks to determine whether changes implemented by services were implemented in the way they were intended, and if they resulted in certain outputs. The Impact Evaluation assesses the effects of funding on healthcare outcomes. It provides a comprehensive view of GIC services, evaluating specific impacts (on waiting times, access to services, patient and staff satisfaction) and general impacts (on overall patient care, collaboration and system changes).

The key questions from each set are listed below:

Process Evaluation Questions:

• How are the changes implemented by the individual services aligned with the original principles for allocation of funding?
• Has how the services are supported within their respective NHS Board structures, systems and frameworks changed during improvement work?
• How has the staff mix/complement changed as part of this work and how?
• How have care pathways changed?
• How is waiting list data used to inform future service planning?
• Has how the service engages with patients changed and, if so, how?
• Has the way the service engages and commissions third sector input changed?
• Are there any emergent issues that are influencing what is being done and how the principles were applied to service change?

Impact Evaluation Questions:

• Have waiting times changed since September 2022? What impact has work undertaken with SG funding had on waiting times, if any?
• Have changes taken place in partnership working between the gender identity clinic and other primary and secondary services in the host/referring Boards?
• Have changes been put in place to support patient care through the provision of expert advice from gender identity services to other primary/secondary services?
• What are the changes to patient satisfaction/feedback and staff satisfaction/feedback as a result of the improvement work?
• Have people’s experience of accessing or waiting to access the services changed as a result of this improvement work and if so, how?
• What benefits or unintended consequences arose from this improvement work for: patient care, service managers, clinical staff, CAMHs, related services?
• Are there any differences in benefits and unintended consequences of this work across different areas and different population groups?

Data Collection Methods

A range of primarily qualitative research tools and methods were used to collect the data required for this evaluation, to triangulate the evidence base and provide actionable intelligence on the impacts of SG funding on access to, and delivery of, gender identity healthcare services in different places. Data was collected for each of the four case studies for the process and impact evaluations: NHS Lothian (which provides adult gender identity services), NHS Highland (which provides adult services), NHS Grampian (which provides adult services) and NHSGGC (which provides both adult services and young people’s services).

(i) Review of literature

We conducted a review of primary and secondary documents relating to gender identity services. This included policy documents and academic literature on the design, delivery, implementation and impacts of gender identity healthcare services in Scotland. We analysed: government documents; independent reviews; analysis of services by Health Boards, third sector organisations and think-tanks; and international research. We identified the literature through a database search of documents (Web of Science, SSRN, Public Information Online, Cochrane Library) and a manual search (using Google) for grey literature. This literature review was not a systematic review of all relevant available literature or data, and thus should not be considered exhaustive or comprehensive; instead, the intention was to provide background information and context for the evaluation and to support methodological decisions.

We also collected service-related data and documentation for each local health system, to help answer our research questions. We contacted each GIC/Health Board and invited them to send us relevant documentation relating to their application for, and receipt of, SG improvement funding. We also requested additional documentation relating to the funding allocations and timelines from the SG.

(ii) Review of budgets

In addition to the review of clinic-related documents, we also conducted a review of financial information related to improvement funding for the delivery of gender identity healthcare services. We requested this information from the SG and the GICs.

(iii) Theory of Change

We developed a Theory of Change (ToC) framework to guide the evaluation project. The ToC model was based on preliminary research interviews and data analysis, including analysis of the SG’s aims for the improvement funding, the aims of the Gender Identity Healthcare Strategic Action Framework, and the aims of Realistic Medicine, supplemented by our further reading and literature review. It was refined throughout the course of the project. In addition, we developed bespoke ToC models for each of the four GICs, based on their aims for the funding and outcomes.

(iv) Questionnaire

We sent out a questionnaire to each of the four Gender Identity Clinics requesting information on key data and delivery mechanisms for gender identity services. The intention was to free up time during the research interviews (see below) to discuss more complex topics about funding impacts. We emailed the questionnaire to key staff, who completed the questionnaire and returned it to our team for analysis.

(v) Surveys

As part of our engagement work, we designed two short online electronic surveys. The first survey focused on third sector organisations, GP local medical committees and partner groups supporting trans and non-binary people seeking gender identity healthcare. The second survey focused on service users.

Both surveys were constructed using Typeform and housed on a bespoke microsite, which set out the project background and a link inviting people to complete the survey.

For the first survey, we sent this to 25 potential partner and referring organisations supporting trans and non-binary people that we identified through an online search. This included third sector organisations, GP organisations and other partner organisations across local health systems. We received 5 responses to the survey, which was live during May-June 2024. In this survey, we asked if respondents would like to participate in a follow-up interview, and 4 people agreed to an interview (see below). Those interviews explored the impact of improvement funding on the outcomes for trans and non-binary service users, and any changes in their collaborations and partnerships with clinics as a result of the improvement work.

For the second survey, we were assisted by Scottish Trans, a third sector organisation, in identifying potential research participants. The survey was sent to services users, and asked about their experiences of gender identity healthcare in Scotland. We received 74 responses to this survey, which was live during July-August 2024. This included 15 responses from service users of NHS Lothian GIC, 3 responses from service users of NHS Highland GIC, 48 responses from service users of NHSGGC GIC and 10 responses from service users of NHS Grampian GIC.^[126] As part of the survey, we asked if respondents would like to participate in a follow-up interview. We invited 15 respondents who wished to be interviewed to speak with us, with at least two people from each NHS Health Board, and 9 people were interviewed (see below).

(vi) Research interviews - key stakeholders

We sent out invitations to 65 key stakeholders across each of the four local health systems to be interviewed for this evaluation project. Of these, we conducted 36 semi-structured research interviews with respondents, comprising key stakeholders across each of the four systems, including multiple staff members from each GIC.

• 9 interviews with people from NHS Highland
• 6 interviews with people from NHS Lothian
• 8 interviews with people from NHS Greater Glasgow and Clyde
• 8 interviews with people from NHS Grampian
• 5 interviews with people from third sector and other national organisations

The interviews took place between April and October 2024. To ensure the utmost confidentiality and anonymity to protect staff data, we have opted not to provide references to individual interviews when quoting or paraphrasing from interviews. The aim of the interviews was to solicit greater insight into the delivery of gender identity healthcare improvements and impact on patient outcomes, and to gauge participants’ perspectives on the challenges and opportunities for delivering services. The interviews lasted approximately 45-60 minutes and were conducted online via Microsoft Teams. The interviews were recorded, transcribed, anonymised, stored, used and destroyed in compliance with UK Data Protection Act and GDPR regulations.

We used non-probability sampling strategies to identify potential interviewees. These included purposive sampling and snowball sampling techniques. Through purposive sampling, we identified - with SG input - a population based on already established criteria, i.e. organisations identified as providing, managing or referring to gender identity services. Through purposive sampling, interviews were sought with a selection of representatives and users in each NHS Health Board system, including staff managing and delivering care in the GIC, staff overseeing the service from the NHS Health Board, staff referring to the GIC and its services (e.g., GPs, for instance via Local Medical Committees, and the Child and Adolescent Mental Health Service - CAMHS), organisations representing medical professionals (e.g. royal colleges and societies) and organisations to whom the GIC refer for further support (e.g. third sector organisations providing social support and advice to patients). We complemented this technique with ‘snowball’ sampling. Here, we asked stakeholder interviewees in our purposive sampling list if they might recommend anyone else we should talk to, thereby enabling the identification of additional study participants. Finally, as mentioned above, for our stakeholder surveys we included a question at the end asking if respondents wished to participate in a follow-up interview. We followed-up with all of the (5) respondents from third sector, GP and partner organisations who wished to be interviewed, which led to four interviews.

(vii) Research interviews - service users

Our survey of service users, which received 74 responses, contained a question asking respondents if they wished to engage in a follow-up interview. We selected 15 service users who wished to be interviewed, seeking to ensure that people of different ages, from different geographical locations, and with different gender identities were included. We sent invitations to those 15 service users across the four local health systems, and of these, we conducted 9 semi-structured interviews. We offered a small token of appreciation (vouchers) to compensate interviewees for their time in line with SG guidance.^[127] To assist this research, we put in place strict confidentiality and data protection protocols when speaking with service users, complemented by our mental health first aid-informed approach to undertaking research with vulnerable people. (We set out ethical considerations around including service users in Annex D.) Service users were included in this project given that they are the people most affected by efforts to improve services. Their inclusion enables a fuller understanding of funding impacts. However, we acknowledge that service users on waiting lists may have not yet experienced the impact or observed any change in processes.

(viii) System Mapping of Relationships

We created visual maps to identify relationships and illustrate the impacts of the funded services on each local health system, as part of the case studies for GICs. The relationship maps are included in the case study chapters. This mapping of local health systems has enabled identification of key stakeholders and partners. These maps were refined throughout the evaluation project as we learnt more about the linkages between GICs and other teams and organisations.

(ix) Case Studies

We have developed five case studies on the impact of SG funding on gender identity healthcare services in NHS Lothian, NHS Highland, NHS Grampian and NHS Greater Glasgow and Clyde (which includes one case study on adult services and one case study on young people’s services). Each case study provides an overview of the GIC, an analysis of service delivery impacts and changes to patient outcomes, and an exploration of systemic issues that enable or inhibit changes to services.

(x) Engagement Event

We hosted an online Learning Event with key stakeholders to advance cross-clinic learning and gain feedback on interim findings from the project. There were 16 participants, working across the four NHS Health Boards delivering gender identity healthcare services. The event presented our interim research findings and included a workshop on ‘futures thinking’ for gender identity services. We informed participants at the beginning of the event that we would take notes to integrate the event learnings into the final report, and made clear to participants that the notes would only be used for the purposes of the report, and that any quotes from the event would be fully anonymised. The event encouraged participants to explore areas for service improvement within their sphere of influence across the systems they are working in. Our team wrote up and reviewed the event resources (notes from break-out groups and general discussions), to draw out perspectives on the barriers and opportunities that are seen to exist, what works, and what can be built on to improve services. The learnings from the event are integrated into this report, most notably in Chapters 9 and 10.

Ethical Approach and Clearances

During the Inception Phase of this project, we contacted several teams within the NHS to seek advice on our ethical approach and any clearances required. We filled out the initial decision tool on the UKRI Medical Research Council/NHS Health Research Authority website, which suggested that NHS Research Ethics Committee review and approval was not required under GAfREC. We then contacted the NHS Research Ethics Committee for further advice and were informed that ethical review was not required as it was a service evaluation. However, ethical guidelines were closely followed at all times throughout the project, which are elaborated in Annex D of this report. In addition, the SG internal ethics process was followed, and a Data Protection Impact Assessment (DPIA) was carried out to identify potential data protection risks.

Limitations of Methodology

This evaluation project depends on the participation of key staff and organisations operating in each of the Health Boards. We recognised how busy and pressured services were, and that there may be some reluctance to participate in this study. We sought to offer maximum flexibility to NHS staff in participating in this project. The project took place over a relatively short time period. We anticipated that research participants would be less available during the summer months (i.e., school holidays) and sought to mitigate for unavailability by collecting most data for the project prior to the end of June. We also anticipated that, in a busy sector with high workloads, not every invitee would have time to participate in this research; we reached out to as many relevant people as possible to mitigate against a potentially lower response rate.

This evaluation project also relied on access to key data and documentation relating to the delivery and impacts of services in each GIC. Waiting times data were especially important. Where service-related data was unavailable from the GICs, we explored other data sources including other NHS data. We were aware that Public Health Scotland was collecting data on GIC waiting lists during the period of this evaluation, and we have included this data in our case studies below.

The methodological design did not include counterfactuals, which may be seen as a weakness, in that causal relationships could not be established. A counterfactual, for instance, could have developed an estimate of what might have happened without the intervention (i.e. the SG improvement funding) through a quasi-experimental design that presented alternative possible scenarios. However, even with a counterfactual analysis, causality is often hard to identify when there are other multiple factors at play. In this project, therefore, we have aimed for understanding rather than causality.

Finally, the evaluation project has also relied strongly on qualitative methods. These are often viewed as more ‘subjective’ than quantitative methods as they tend to rely on experiences, understandings and perceptions of services; while the volume of data can make analysis and interpretation time-consuming. We believe that these weaknesses are outweighed by the strengths of qualitative methods in collecting rich data that can lead to new insights and unexpected findings about service delivery and impacts.

Quality Assurance

The process and impact evaluation results have been quality assured by an external expert. Quality assurance (QA) and assessment has been built into all stages of the project to ensure that the research is properly conducted. An external expert was appointed to ensure that the evaluation research design, methods and outcomes were of the highest quality and integrity, and the results were reported accurately and without bias, i.e. following the highest quality benchmarks when conducting surveys (clear questions, ensuring anonymity of participants, transparent data collection and results, GDPR compliance) and semi-structured research interviews (obtaining consent forms, giving clarity on the purpose of the research, ensuring confidentiality, safeguarding participants). Quality assurance advice was received throughout this project, with the QA expert regularly checking that the data and deliverables met ethical and data protection standards, and that outputs were technically correct, well laid-out, accessible to the public, and free of errors.