Ethnicity in the justice system: evidence review

Appendix B: Methodology for Scottish Crime and Justice Survey pooled sample analysis

Background

All analysis in this report from the Scottish Crime and Justice Survey (SCJS) on ethnicity comes from a pooled sample. This is a further analysis project of existing survey data that aimed to present findings on ethnicity for the first time.

The SCJS is a large scale social survey that asks people in Scotland about their attitudes and experiences of certain issues relating to crime and justice. The respondents are chosen at random among all adults aged 16 and over living in private residences in Scotland. Each survey sweep typically runs over a twelve month fieldwork period, and results of each sweep are reported separately in a long form findings document.

Given it is not feasible to interview all people in Scotland, a sample of addresses are selected for each sweep at random and members of these households are asked to take part in an interview. The results of these interviews are collected and analysed in order to make inferences about the Scottish population as a whole, as well as break the data down to examine different groups within society. Currently, findings are presented by gender, age, deprivation and disability status among other things.

However, it is not possible to necessarily provide breakdowns of the data for all groups, and one of the most notable cases where this occurs is for ethnicity. Given that participants are chosen completely at random, it cannot be guaranteed that the fieldwork period will collect a sufficiently large amount of data from people from minority ethnic backgrounds in order to report reliable and accurate data. Full details of the sampling process used in the SCJS can be found online in the technical report.

In Scotland's 2011 census, still the most recent at time of writing, we can see in Table B.1 that the ethnicity makeup of Scotland is very imbalanced. As a result, the composition of ethnicities in the SCJS results will reflect this.

The most recent SCJS sweep completed 5,570 interviews across the country, while this was sufficiently large to make accurate inferences about the population as a whole and of certain subgroups, it was not possible to do the same for many minority ethnic groups. In order to overcome this issue, a new analysis project had to be undertaken to be able to present findings for these groups.

Pooled sample methodology

To overcome the issue of having too few respondents from minority ethnic backgrounds, a new approach was taken to bring together the results of multiple SCJS sweeps into one pooled sample. This would bolster the sample size both overall and for lesser represented groups too, meaning it would be possible to present findings for the survey by ethnicity for the first time. The methodology to implement this pooled sample is described below.

Firstly, each survey sweep results in a certain number of interviews being completed, and this varies from sweep to sweep. At the beginning of the SCJS, the sample size (n) target was to complete approximately 16,000 interviews, however in the more recent years this target has been roughly 6,000. Table Y shows the exact number of interviews completed across the 9 sweeps.

Selection for participation in the SCJS is random, however it is not completely random for the entire country as a whole. The design framework of the survey is to be representative at the level of the 13 Police Division areas in Scotland. This deviation from a true random sample is expressed in what is known as the 'design effect' (Deff). The exact value of the design factor used in each survey sweep is provided by the contractors who run the fieldwork process of the SCJS.

Using the design factor, we can calculate the 'effective sample size' of each survey. The effective sample size (neff) is the overall sample size divided by the design effect, and is an expression of the number of responses needed under a pure random sample to obtain the same results as the actual sample framework.

Having done this, the proportion of the overall effective sample across the 9 sweeps can be calculated for each individual sweep, denoted as 'pool_neff' in Table B.2. This will be used later on when discussing weighting, and is an important step in ensuring that potential bias from uneven sample sizes across sweeps is reduced.

The only remaining step is to implement the pooled design effect on the weightings for the data. Survey data is weighted in order to ensure that the data better reflects what it is trying to measure. Each observation is assigned a weighted value to indicate how important it is to the analysis. The procedures for the implementation of the weighting methodology were developed by the Scottish Government with the Methodology Advisory Service at the Office for National Statistics (ONS), full detail on this can be found in the SCJS technical report. Multiplying the pooled effect sample by the weight results in the pooled weight for the survey which is then used throughout the analysis.

Limitations of pooled sample

While the pooled sample methodology has been an effective project for implementing analysis of the SCJS by ethnicity for the first time, there are some limitations to the power of the analysis.

The main limitation is that any sense of change over time is lost by combining survey years. The pooled sample is able to give a sense of the overall picture across all surveys, it is not able to tell whether there have been improvements or changes from one survey to another. Given some of the data from the pooled sample is over a decade old, it remains unclear whether the picture presented in the pooled sample is still representative of how it is today. It would be possible to perform further analysis yet and split the pooled sample data into two or more time periods, allowing for a 'before and after' analysis, but the analysis as presented in this report focusses only on one overall picture.

The other limiting factor of the pooled analysis is that it has not fully overcome sample size issues present when looking at ethnicity data. The SCJS is a constantly evolving product, which is adapted to keep up with the changing nature of crime and justice and of emerging topics. Some questions in the survey have not been asked across every sweep, so the sample size is smaller. This means that some questions still cannot be analysed by ethnicity despite the pooled sample methodology. Further, even for questions which have been present in the questionnaire for a significant length of time, it was still not always possible to analyse questions at the most granular level. As a result, the findings are presented using groupings of ethnicities to maintain consistency of analysis throughout the report.