Annex 3: Data Strengths and Limitations
A.3.1 Overview of Scottish Crime and Justice Survey strengths and limitations
The table below outlines some of the main strengths and limitations associated with the SCJS (and crime and victimisation surveys more generally). Further detail is provided on some issues in the sections below. For further details on the respective advantages and limitations of SCJS data see the introduction to the 2014/15 Main Findings Report and the Technical Report.
Section A.3.4 discusses the strengths and weaknesses associated with respondent self-completion aspects of surveys.
Captures information about crimes that are not reported to the police (including sensitive issues such as domestic abuse or drug abuse).
Does not cover all crimes ( e.g. homicide or 'victimless' crimes such as speeding).
Provides information on multiple and repeat victimisation (up to 5 incidents in a series).
Does not cover the entire population ( e.g. children, homeless people or people living in communal accommodation).
Good measure of trends since 2008/09.
Unable to produce robust data at lower level geographies.
Analyses risk for different demographic groups and victim-offender relationships.
Difficult to measure trends between survey sweeps in rarer forms of crime (such as more serious offences).
Provides attitudinal data ( e.g. fear of crime or attitudes towards the justice system).
Estimates are subject to a degree of error.
A.3.2.1 Comparison with police recorded crime
One of the main strengths of crime and victimisation surveys such as the SCJS is that they provide a complementary measure of crime to police recorded crime statistics. Police recorded crime is a measure of crime that police come into contact with. However, it is well established that people may be unwilling to report crimes for a range of reasons, including a perceived lack of benefit, fear of reprisal, vulnerability, an inability to identify assailants or unwillingness to bring the victims conduct to the attention of the police. People are also less likely to report some types of crime than others (for example, people are less likely to report sexual than property crimes). Police practices can also influence recorded crime, for example, officers may not record all crimes reported by the public.
These factors are unlikely to affect SCJS data. For example, people's attitudes toward the police are unlikely to affect SCJS data, nor are the data affected by police recording practices. SCJS data also provide a measure of prevalence, that is, the risk of experiencing different types of crime in a given time period. By contrast, police recorded crime can only measure incidence or the number of crimes. In addition, the SCJS collects demographic information, providing richer insights into who is experiencing crime. Furthermore, follow-up questions about incidents allow the SCJS to capture respondent's attitudes to the criminal justice system, including reasons for not reporting crimes to the police.
A.3.3.1 Sampling and crime type limitations
The main limitations of the SCJS result from sampling, and the types of crimes surveyed. In terms of sampling, the survey is of adults aged 16 and over, living in private residential households only. As such, the survey excludes persons under the age of sixteen, the homeless, and populations living in residences such as care homes, halls of residences, hospitals, prisons or other communal accommodation.
A.3.3.2 Survey Error
As discussed in the report Introduction, the SCJS gathers information from a sample rather than from the whole population and survey results are always estimates, not precise figures. This means that they are subject to a level of uncertainty. To estimate the extent of this uncertainty, 95% confidence intervals for the statistics are calculated to define bands within which the 'true' value of survey estimates are likely to lie ( i.e. that value which would be obtained if a census of the entire population was undertaken). These confidence intervals are particularly important when making comparisons of SCJS estimates over short timescales.
A.3.3.3 Non-quantifiable errors: recall and accuracy
SCJS estimates are also subject to a margin of non-quantifiable error. For example: there may be errors in the recall of participants as to when certain incidents took place; respondents may have claimed to have reported a crime to police when they had not, feeling that this was the socially acceptable response; some incidents could also be inaccurately recorded by interviewers, or miscoded by the wider survey team. Although a number of steps in the design and implementation of the survey are taken to reduce such errors, they can never be fully eliminated.
There may be errors in the recall of participants as to when certain incidents took place, or the number of incidents that took place. This is particularly relevant to collecting data on more senisitive topics, which may be cumulative and ongoing. For example, it may be difficult for respondents to recall the exact number of incidents. Also, respondents may not want to either remember or report some experiences.
A.3.3.4 Survey design changes
The collection of survey data on crime and victimisation in Scotland has undergone several major changes in methodology. Changing crime survey methodology in Scotland has implications for making comparisons across survey designs. As previous surveys had smaller sample sizes, estimates from earlier surveys are subject to a higher degree of uncertainty, and this report therefore focuses on the period from 2008-09 onwards.
A.3.4 Self-report data strengths and limitations
As with the reporting of crime experienced, the SCJS self-complete section is unlikely to be influenced by attitudes towards the police and are unaffected by police recording procedures. It is designed to allow respondents to answer questions on more sensitive and personal topics privately. Self-report data can capture crimes and experiences of a sensitive nature that respondents may be unwilling to report to the police, or to disclose in a face-to-face survey situation.
However, a number of factors may act as limitations on the self-report data (as well as other types of survey data). These include the wording of questions and the presence or skills of the interviewer. The presence of other people in the house may also influence results. Although the self-completion module allows respondents to answer in relative privacy, respondents may be unwilling to disclose personal or distressing details.
The nature of self-report means that these estimates of drug use capture what respondents intended to take, or believed they had taken. Given the evolving nature of the drug market however, those who have taken illicit drugs, or 'legal highs' may not be sure about what they have taken.
Another important limitation to consider is where self-completion interviews on sensitive topics are administered by the interviewer at the request of respondents who, for example, do not wish to use the laptop/tablet to complete the interview themselves. In 2014/15 87% of respondents completed the self-completion section; 71% entered their answers directly in to the tablet PC themselves and 16% asked the interviewer to administer the questionnaire for them. Of those where the interviewer administered the self-completion, in five per cent of cases, the respondent completed the section themselves after the first few questions being administered by the interviewer. Steps are taken by the SCJS trained interviewers to ensure that the number of self-complete interviews that are interviewer-administered are minimised and this is monitored closely by the SCJS team and our survey contractors.
Under-reporting and under-representation is also a concern of this survey. For example, it is likely that there will be an under-representation of some groups, e.g. those who take drugs. In part, this will be due to the fact that some people who use drugs may live in accommodation not covered by a survey of private households (such as the SCJS). The survey is likely to under-represent those with the most problematic or chaotic drug use, some of whom may live in accommodation outwith the scope of the SCJS and some of whom may live in private households covered by the survey, yet they may be rarely be at home or be unable to take part in an interview due to the chaotic nature of their lives.
Despite using Computer Assisted Self-completion Interviewing ( CASI) for this module, it is likely there will be some underreporting of (illicit) drug use, partner abuse and sexual victimisation and stalking among survey respondents due to the sensitivity and legality of these issues, despite reassurances about confidentiality and anonymity.
Questions cover past use over varying periods ( ever, in the last year and in the last month) and it is possible that some respondents may simply forget experiences, particularly if they last took a particular drug a long time ago.
While under-reporting of drug use on surveys such as the SCJS is likely, it should be noted that the issues discussed above are unlikely to apply equally across all types of drugs. While a survey such as the SCJS is likely to provide an insight into the more commonly used drugs, in particular cannabis, it may be less effective in providing information for some of the Class A drugs such as opiates or crack cocaine, where a sizeable number of those using these drugs may be concentrated in small sub-groups of the population not covered by the survey (Smith and Flatley, 2011).
It is also recognised that some people may report taking particular drugs when they have not actually done so. Respondents were therefore asked whether they had ever taken a fictitious drug 'semeron'. Fourteen respondents reported having ever taken semeron and these respondents have been excluded from the analysis in this report (making the overall sample 9,972).
Section 7.7 of the Technical Report also discusses the self-complete section of the questionnaire in more detail.
There is a problem
Thanks for your feedback