Equality evidence: Inclusive evidence resources

Introduction

Robust and comprehensive equality evidence helps us understand how events impact differentially on individuals, groups and communities. This, in turn, is vital for the design and delivery of inclusive policies and services. The COVID-19 pandemic in particular demonstrated the importance of good quality equality evidence. It showed how events can impact differentially on individuals, groups and communities by exposing deep-rooted structural inequality in our society and exacerbating the disproportionate impact on individuals and groups who already experience disadvantage.

This guide presents brief summaries of key resources on improving inclusivity in the collection of evidence. Equality data collectors from across the public sector are encouraged to access the full resources for more detail on how they could improve their practice.

The focus of this guide is on collecting equality evidence in an inclusive and appropriate way, however we recognise that data protection is a concern for anyone collecting equality evidence. To support understanding in this area we would recommend the following resources: The public sector equality duty and data protection | Equality and Human Rights Commission (equalityhumanrights.com) and Data protection and anonymity considerations for equality research and data | Advance HE (advance-he.ac.uk).

The Scottish Government (2023) Participation Framework

The Participation Framework was produced by the Scottish Government to fulfil commitments in its Open Government Action Plan 2018-20. It provides a guide to good practice for SG staff who are taking decisions about participation work. This is part of SG’s commitments to provide high quality and inclusive opportunities for the Scottish public to be involved in the development of policy, service design and decision making that affects them.

The Participation Framework is currently being updated to provide greater focus on equalities and inclusive practice.

The framework lays out the spectrum of participation, which contains five types of activity:

• inform: to provide the public with balanced and objective information
• consult: to obtain feedback on analysis, alternatives, proposals and/or decisions
• involve: to work directly with participants throughout the policy/decision making process to ensure that their concerns and aspirations are consistently understood and considered
• collaborate: to partner with participants in each aspect of the decision, including defining the issue, developing alternatives and identifying preferred solutions
• delegate: to place final decision-making in the hands of the participants

The framework also sets out the National Standards for Community Engagement:

• inclusion: we will identify and involve the people and organisations that are affected by the focus of the engagement
• support: we will identify and overcome any barriers to participation
• planning: there is a clear purpose for the engagement, which is based on a shared understanding of community needs and ambitions
• working together: we will work effectively together to achieve the aims of the engagement
• methods: we will use methods of engagement that are fit for purpose

• communication: we will communicate clearly and regularly with the people, organisations and communities affected by the engagement
• impact: we will assess the impact of the engagement and use what has been learned to improve our future community engagement

This publication may help you to: 

• develop your understanding of participatory approaches
• design and commission research that uses participatory methods
• design and produce accessible communications and reports
• collect data from participatory methods  

For further information and advice contact: Doreen.Grove@gov.scot

Scottish Government (2022) Collecting Equality Data

In 2022 the Scottish Government published guidance notes on the recommended questions to ask when collecting information on: age, disability, ethnic group, gender, religion or belief and sexual orientation. In addition to this guidance is also provided on the analysis and presentation of data.

This publication may help you to:

• design research
• collect data
• analyse and present data

For further information and advice contact: social-justice-analysis@gov.scot

Scottish Government (2022) Using intersectionality to understand structural inequality in Scotland: Evidence synthesis

In 2022 the Scottish Government published a report summarising literature on the concept and applications of intersectionality. It is intended to be used by public sector analysts and policymakers to build knowledge and expertise on how to analyse, report and use equality data to develop effective services for those with intersecting protected characteristics.

The report aims to contribute understanding of:

• what intersectionality means and how it can be applied to policymaking
• how intersectionality can be integrated throughout the analytical process
• examples of how the concept of intersectionality has been used
• available resources that the reader can use to further deepen understanding

In the report intersectionality is defined by three key elements:

• a recognition that people are shaped by simultaneous membership of multiple interconnected social categories
• the interaction between multiple social categories occurs within a context of connected systems and structures of power (e.g. laws, policies, governments). A recognition of inequality of power is key to intersectionality
• structural inequalities, reflected as relative disadvantage and privilege, are the outcome of interconnected social categories, power relations and contexts

In order to integrate the concept of intersectionality throughout the analytical process analysts should aim to address the power imbalance in research, they can do this in the following ways:

• reflectivity: a researcher should ask themselves questions about their own social positions, values, assumptions, interests and experiences and how these can shape the research process, as well as putting the research into context
• public involvement in research: there should be active contribution from people with lived experience in research projects and in research organisations
• ensuring marginalised groups are reached: analysts need to consider and take practical steps to aid diverse research recruitment
• sampling techniques and reporting: sampling techniques used in research are important considerations when taking an intersectional approach. For example, a completely random sampling approach or use of a sampling frame designed to be representative of a whole population may lead to problems with low sample size for lower-frequency groups in the population or among those who may be more reluctant to take part in or contribute to research. If this is the case, the validity and applicability of other sampling techniques should be considered to assist coverage

This publication may help you to: 

• design intersectional research
• understand and apply methods to produce intersectional data
•  report on the findings of intersectional research

For further information and advice contact: social-justice-analysis@gov.scot

The Scottish Government (2021) – Understanding equality data collection in the Scottish public sector

Research published by the Scottish Government in 2021 and produced by Jennifer Waterton Consultancy explores the collection of equality data within a sample of 27 Scottish public sector organisations and networks.

The study involved:

1. a desk-based review of organisations' websites and publications
2. the collection of descriptive information about each selected data collection
3. qualitative interviews to explore issues related to the collection of equality data
4. discussions with public sector equality networks

The research explored the following topics:

• what equality data are collected across equality characteristics
• characteristics of equality data collections (e.g. surveys, administrative data), and their purpose and uses
• the collection and processing of equality data, including the methods used to collect data (e.g. online, telephone, face-to-face methods)
• barriers and challenges in collecting equality, including: the personal and sensitive nature of equality data, practical, operational and/or methodological issues, difficulties relating to definitions and terminology, and shortcomings in the organisational culture, capacity and/or capability
• factors which enable and facilitate equality data collection, including: mainstreaming equality, the importance of being clear and the purpose of collecting data, using data collection methods that work, building data management and analytical capacity, and improving guidance and developing networks for support

In addition, in December 2021 the Scottish Government published a series of case studies produced by Jennifer Waterton Consultancy focused on good practice in the collection of equality data. These case studies focused on a series of public sector bodies: sportscotland; Highland Council Children’s Services; Skills Development Scotland; The Open University in Scotland; Social Security Scotland, and the Scottish Children’s Reporter Administration.

This publication may help you to:

• design research
• understand the barriers and facilitators to good data collection

For further information and advice contact: social-justice-analysis@gov.scot

Office for National Statistics (2021) - Inclusive Data Taskforce report: Leaving no one behind. How can we be more inclusive in our data?

In October 2020, the National Statistician convened an independent Taskforce to recommend how best to make a step-change in the inclusivity of UK data evidence. The Taskforce’s findings are presented in the report ‘Leaving no one behind: how can we be more inclusive in our data?’, which was published in September 2021. The Taskforce considered four key questions:

1. how can we improve inclusiveness in our approach to the collection, analysis and reporting of data and evidence?
2. how can we make most effective use of existing data, such as administrative, census and survey data to understand equalities and inclusion?
3. what are the critical data gaps that hinder our understanding of equalities and inclusion and how can we address them?
4. how can we build on our own and others’ experiences in improving our approach to equalities and inclusion going forward?

A range of consultation activities were carried out by the ONS, on behalf of the Taskforce, between January and May 2021 to inform the findings, and these included:

• a 12 week online open consultation on CitizenSpace

• seven roundtable discussions and six in-depth interviews with central and local government representatives, and those in the devolved nations
• four roundtable discussions and two in-depth interviews with academics and representatives of learned societies
• discussions with over 80 civil society leaders working in 15 different equalities areas
• discussions with over 90 members of the public with lived experience of equalities issues

Key findings from the consultation activities spanned many themes, including:

• improving trustworthiness: there was a perception among several participants that the government are not always viewed as trustworthy. This was particularly, though not exclusively, the view among under-represented groups, specifically described as affecting those from Gypsy, Roma and Traveller communities, other minority ethnic groups and documented and undocumented migrants. Improving trustworthiness is important to enable data, and any subsequent policy decisions, to better reflect these populations’ needs and experiences

• ensuring that the data collected meet respondent and user needs: the labels used to capture individual characteristics within data collection were perceived as critically important to enable people to select categories in surveys and on forms that reflect their personal characteristics and circumstances and to ensure that the data allows for an accurate understanding
• ensuring data collected are of sufficient quality to accurately count everyone in society and monitor their outcomes: various quality issues in relation to data collection were identified, particularly in terms of conceptual challenges and lack of harmonisation and coherence

The Taskforce made 42 recommendations categorised under 8 Inclusive Data Principles:

1. create an environment of trust and trustworthiness which allows and encourages everyone to count and be counted in UK data and evidence
2. take a whole system approach, working in partnership with others to improve the inclusiveness of UK data and evidence
3. ensure that all groups are robustly captured across key areas of life in UK data and review practices regularly
4. improve the UK data infrastructure to enable robust and reliable disaggregation and intersectional analysis across the full range of relevant groups and populations, and at differing levels of geography
5. ensure appropriateness and clarity over the concepts being measured across all data collected
6. broaden the range of methods that are routinely used and create new approaches to understanding experiences across the population of the UK
7. harmonised standards for relevant groups and populations should be reviewed at least every five years and updated and expanded where necessary, in line with changing social norms and respondent and user needs
8. ensure UK data and evidence are equally accessible to all, while protecting the identity and confidentiality of those sharing their data

Following the publication of the Inclusive Data Taskforce’s recommendations the Office for National Statistics (ONS) published an implementation plan in January 2022, which summarises known current and planned initiatives across the UK statistical system. The activities are grouped based on the eight Inclusive Data Principles, discussed above. The first annual review of progress towards implementing these recommendations was published in May this year (2023).

This publication may help you to: 

• design inclusive research
• analyse data
• report on findings

For further information and advice contact: equalities@ons.gov.uk

European Commission (2021) - European handbook on equality data

This guidebook from the European Commission offers advice on various aspects of data collection including: why equality data should be collected; planning and organising the collection of equality data, as well as advice on collecting different forms of equality data (e.g. official statistics, qualitative data, diversity monitoring). Key findings on increasing response rates include:

• sources of error in survey: in order to achieve reasonable response rates it is important to ensure the survey design takes into account the specific needs and characteristics of the target population. This might involve, for example, having the questionnaires in several languages and/or using interviewers who speak the language of the respondents; utilising face-to-face interviews instead of telephone interviews or postal surveys
• self-administered data collection tools may be more effective for collecting data on sensitive behaviours, for example discrimination
• the context of the survey -  the ‘packaging’ of the survey, i.e. the apparent topic of the survey, the organisation responsible for collecting the data, the letterhead used on advance letters and similar details may affect how individuals perceive the purpose of the survey, and potentially affect the way respondents interpret the questions

This publication may help you to:

• design research, particularly surveys

For further information see: European handbook on equality data - Publications Office of the EU (europa.eu)

Laura Wilson, Emma Dickinson (2022) Respondent Centred Surveys: Stop, Listen and then Design, London: SAGE

This handbook on writing respondent centred surveys is produced by researchers from ONS’s Research and Design Team. It applies the concept of ‘user centred design’ from the online user experience design industry. This involves gathering insights and needs directly from research with the people are using the product, and then using these insights to design the solutions, and not the views of the product’s owner. This contrasts with the usual approach to survey development, whereby the data user traditionally drives the design. The authors present the following framework:

1. gather the data user needs: before any research and testing takes place with respondents, the data users must be consulted to establish what they are trying to learn and analyse at every question and derived variable. Once you have established the data user needs, you then need to take time to familiarise yourself with them and review them through out the development process
2. understanding mental modes: a mental model provides an understanding of how someone thinks, feels and then later acts. They help you understand a respondent’s thought process which means you can then incorporate that information into your designs. We can learn about the following from exploring mental models: tone of voice and words/language to be used in our communication and questions; what the communication and survey content will contain, explain and ask; the appearance and style of the communications and interfaces; the order and flow of topics and themes within the questionnaire; how concepts and questions are understood and relate to other topics. You can learn about mental modes by: gathering the data user needs; insight sessions (with interviewers); observing interviewers; conducting in-depth interviews with respondents
3. understand respondent experience and needs: respondent needs can be established by speaking to respondents, observing them and learning from their experiences
4. use data and insights to design: utilise the quantitative and qualitative data you have collected to inform your design decisions, including about the ordering and flow of your survey
5. create using appropriate tone, readability and language: you should create content which is conversational in tone – ask ‘is this how you would speak to a friend about this?’ Tone – when researching tone it is important to ask the respondents to share how the content makes them feel. Readability – think about average reading age (9 in the UK); avoid use of localised and regionalised words or phrases; avoid use of contractions; use short sentences. Language – through research with respondents we can learn about the language that they use to describe their circumstances and interests
6. design without relying on help: don’t depend on additional guidance to support question comprehension. Instead of relying on guidance break one question up into many – make the questions clearer with only one question being focused on
7. take an ‘optimode’ approach to design: for materials an optimode approach means adapting the content to reflect the journey and needs of the respondent in that mode. Cogability testing of materials and questions in each mode ensures that the content is understood as intended and the desired action or response is achieved
8. use adaptive design: adaptive design aims to deliver the best respondent experience based on the device being used, by adapting the display to the context of use
9. conduct ‘cogability’ testing: combine cognitive testing and usability testing in the same session. It is important to combine the two methods in order to allow us to fully evaluate the effectiveness of the design. The cognitive testing side explores how a respondent comprehends a question through probing and ‘think aloud’ techniques during a one-to-one session. The usability testing side explores how easy or difficult a product is to use, for example, following letter instructions, online navigation, finding help, fixing errors
10. design inclusivity: this means including and learning from people with a range of perspective, including autistic people, people who use assistive technologies, people with low vision and people with dyslexia

This publication may help you to:

• design research, particularly surveys
• understand the concept of user centred design and how to apply it survey design

For more details see: User-centred design approach to surveys – Government Analysis Function (civilservice.gov.uk)

Roberta Sammut, Odette Griscti, Ian J. Norman (2021) ‘Strategies to improve response rates to web surveys: A literature review’, International Journal of Nursing Studies 123

The aim of this literature review was to evaluate strategies to increase the response rate to web surveys. 45 papers were included in this review which found that:

• e-mail pre-notification, e-mail invitation, 2 reminders, and a simple design which is easy for respondents to follow increase response rates to web surveys
• web surveys that take no longer than 10 min to complete are more likely to achieve a higher response rate than longer surveys
• a semi-automatic log-in and blank subject line or a subject line of interest to potential participants improves response rates as does entry to a draw with a cash prize considered to be of value to respondents drawn and awarded at the end of the study
• progress indicators are not helpful and may actually decrease response rates to web surveys.

The review suggested that the following strategies could be used to increase response rates:

• prenotification, invitation and reminders: findings of the studies generally suggested that a pre-notification increases response rate
• studies found that when it comes to the actual invitation to participate in a survey the response rate tends to be higher when an email invitation was used versus a post-card invitation. Highest response rate where email is used in combination with an SMS prenotification
• the studies included in this review generally indicate that reminders are important in increasing the response rate to web surveys; a number of studies have explored different modes of delivering reminders, the number of reminders which need to be sent and when these should be sent
• several studies have investigated the impact of incentives on response rates to a web survey. Three main categories of incentives have been studied - monetary incentives, gifts (including gift vouchers) and lotteries/entry into a prize draw. The results of these studies indicate that incentives are effective in increasing the response rate to web surveys

This publication may help you to:

• design research
• increase survey response rates

For more details see: Strategies to improve response rates to web surveys: A literature review - ScienceDirect

Equality and Human Rights Commission, Scottish Government, Improvement Service (2012) Improving Local Equality Data

The Equality and Human Rights Commission, the Scottish Government, and the Improvement Service (IS), worked in partnership to deliver the Improving Local Equality Data (ILED) project; an action-learning project to support four councils to meet the challenge of developing the equality evidence base to use within their outcomes approach. The ILED took place between February 2011 and February 2012.

The ILED report sets out the project’s key findings, learning, and recommendations to improve the availability and use of equality evidence in Scotland. This paper also makes links to published papers, workshop material and other relevant, practical information that may be useful to those collecting equality evidence.

Key findings include:

• the importance of harmonised questions: using harmonised questions when gathering equality information is a practical way of improving both the quality of data gathered and the potential to compare results both locally and nationally
• good quality qualitative information can provide richness to an organisation’s knowledge about their local community. Qualitative information can demonstrate the complexities of some issues and the day-to-day impact this has on people’s lives, as well as highlighting what might appear to be obvious issues, but which are not picked up through quantitative methods. Qualitative and quantitative information complement each other and can be used together to provide a rich picture of a local area
• partnership working within and between organisations is crucial to improve the gathering, use and sharing of equality evidence

This publication may help you to:

• design both qualitative and quantitative research

For more details see: scotland@equalityhumanrights.com, info@improvementservice.org.uk