Publication - Statistics

Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS): technical report 2018

Published: 26 Nov 2019
Directorate:
Population Health Directorate
Part of:
Education, Health and social care, Law and order
ISBN:
9781839603327

Information on the fieldwork and data processing for the 2018 Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS).

Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS): technical report 2018
Precision of results and measurement of change

Precision of results and measurement of change

Survey respondents represent only a sample of the total population of 13 and 15 year old pupils in Scotland, and results are therefore subject to sampling error. The sampling error is the amount by which the value of a sample estimate for a particular parameter is expected to differ from its true value in the population sampled from. This means that observed differences between sub-groups may have occurred by chance. Throughout the report we only comment on differences that are statistically significant, where we can be 95% confident that such a difference has not occurred by chance (p<0.05).

The formula for calculating the sampling error (SE) of a simple random sample is:

The formula for calculating the sampling error (SE) of a simple random sample

where p = the estimate of the parameter and n=sample size.

The formula for calculating the sampling error for the differences between two estimators (p1 and p2) derived from two independent samples (assuming a simple random sample) is:

The formula for calculating the sampling error for the differences between two estimators (p1 and p2) derived from two independent samples (assuming a simple random sample)

Rather than using a simple random sample, whole classes were sampled within the schools that agreed to participate. Therefore, classes were clustered within those schools, and pupils were clustered within those classes. Pupils within the same class and within the same school were more likely to be similar to each other, and therefore values cannot be assumed to be independent of one another. Further details on the calculation of standard errors and design effects are provided below.

It is important to recognise that sampling error is only one of the sources of error that affect the accuracy of survey results. Other sources of error include response bias (previously discussed) and over/under reporting[16], both of which are difficult to quantify.

Calculating standard errors and design effects

The sample design of SALSUS is complex, involving stratification by local authority and school type (state or independent), as well as clustering within schools. In addition, weights were applied when obtaining survey estimates.

Complex design and weighting affects standard errors for survey estimates, which are generally higher than the standard errors that would be derived from an unweighted simple random sample (SRS) of the same size. For example, clustering reduces the precision of estimates, whereas stratification can increase precision. Weighting can also reduce the precision of estimates.

The ratio of the standard error of the complex sample to that of a simple random sample of the same size is known as the design factor. Put another way, the design factor (or 'Deft') is the factor by which the standard error of an estimate from a simple random sample has to be multiplied to give the true standard error of the complex design.

A Design Factor greater than 1.0 indicates a confidence interval wider than it would be with simple random sampling, meaning that the precision of estimates is reduced. A Design Factor of less than 1.0 indicates a narrower confidence interval and greater precision.

The true standard errors and Defts for SALSUS 2018 were calculated using the Taylor series linearization method as for previous waves of the survey. Thirty-five strata were included for the calculation of standard errors and Defts, one for each combination of local authority and school type (state or independent sector). For eight local authorities: Argyll and Bute, Dundee City, East Renfrewshire, Fife, Inverclyde, Perth and Kinross, South Lanarkshire and Stirling, the variance between clusters could not be calculated for independent schools because there was only one independent school sampled in each of these local authorities. The independent schools in these eight local authorities were combined into one stratum to calculate sampling errors and Defts. There were 1,126 clusters used in the calculations, one for each class that participated in the survey[17].

The Deft values applied and adjusted true standard errors (which are themselves estimates subject to random sampling error) are shown in Tables 5 to 10 for six key variables from the survey along with 95% confidence intervals.

When comparing the 2018 key variables with the 2015 key variables, significance tests were applied based on the 'pooled standard error' for each variable (a weighted sum of the true standard errors[18] for each year). Table 11 shows the six key variables for 2015 and 2018, with the true standard errors for each year and whether or not the difference is significant at the 5% level.

Table 5: Linearised standard errors and 95% confidence intervals for the proportion of pupils who are regular smokers, by age and gender: Scotland 2018

Sample Size Proportion Linearised Standard error Binomial Wald 95% CI Defts
Lower CI Upper CI
13 year olds
Boys 5951 1.52% 0.22% 1.14% 2.02% 1.388
Girls 5869 1.73% 0.19% 1.39% 2.14% 1.114
15 year olds
Boys 4995 7.59% 0.44% 6.77% 8.50% 1.176
Girls 5037 6.10% 0.47% 5.23% 7.09% 1.397

Table 6: Linearised standard errors and 95% confidence intervals for the proportion of pupils who drink alcohol at least once a week, by age and gender: Scotland 2018

Sample Size Proportion Linearised Standard error Binomial Wald 95% CI Defts
Lower CI Upper CI
13 year olds
Boys 6022 4.31% 0.34% 3.68% 5.03% 1.307
Girls 5877 4.15% 0.31% 3.59% 4.80% 1.184
15 year olds
Boys 5023 17.10% 0.65% 15.85% 18.41% 1.226
Girls 5031 15.59% 0.68% 14.30% 16.97% 1.330

Table 7: Linearised standard errors and 95% confidence intervals for the proportion of pupils who drank alcohol in the last week, by age and gender: Scotland 2018

Sample Size Proportion Linearised Standard error Binomial Wald 95% CI Defts
Lower CI Upper CI
13 year olds
Boys 5605 6.51% 0.43% 5.71% 7.40% 1.303
Girls 5520 5.93% 0.39% 5.21% 6.76% 1.239
15 year olds
Boys 4834 20.38% 0.74% 18.97% 21.87% 1.274
Girls 4901 19.67% 0.79% 18.17% 21.26% 1.384

Table 8: Linearised standard errors and 95% confidence intervals for the proportion of pupils who have ever used drugs, by age and gender: Scotland 2018

Sample Size Proportion Linearised Standard error Binomial Wald 95% CI Defts
Lower CI Upper CI
13 year olds
Boys 5684 7.37% 0.43% 6.56% 8.27% 1.253
Girls 5634 4.97% 0.36% 4.31% 5.71% 1.227
15 year olds
Boys 4719 24.14% 0.79% 22.63% 25.72% 1.261
Girls 4844 17.49% 0.64% 16.26% 18.79% 1.181

Table 9: Linearised standard errors and 95% confidence intervals for the proportion of pupils who have used drugs in the last year, by age and gender: Scotland 2018

Sample Size Proportion Linearised Standard error Binomial Wald 95% CI Defts
Lower CI Upper CI
13 year olds
Boys 5684 6.41% 0.42% 5.64% 7.27% 1.278
Girls 5634 4.20% 0.34% 3.59% 4.91% 1.257
15 year olds
Boys 4719 21.42% 0.74% 20.01% 22.90% 1.235
Girls 4844 15.73% 0.63% 14.52% 17.01% 1.213

Table 10: Linearised standard errors and 95% confidence intervals for the proportion of pupils who have used drugs in the last month, by age and gender: Scotland 2018

Sample Size Proportion Linearised Standard error Binomial Wald 95% CI Defts
Lower CI Upper CI
13 year olds
Boys 5684 4.25% 0.36% 3.59% 5.02% 1.356
Girls 5634 2.73% 0.27% 2.25% 3.31% 1.239
15 year olds
Boys 4719 14.87% 0.61% 13.71% 16.11% 1.178
Girls 4844 9.16% 0.49% 8.23% 10.18% 1.193

Table 11: Statistical significance of comparisons between 2015 and 2018 results for key variables

2015 SALSUS 2018 SALSUS T test (2 sided) P – Value Significant at the 5% level?
% True standard error Sample size % True standard error Sample size
Regular smokers
13 yr old boys 1.52 0.188 6439 1.52 0.220 5951 0.000 1.000 No
13 yr. old girls 1.65 0.211 6712 1.73 0.189 5869 -0.282 0.778 No
15 yr. old boys 7.39 0.478 5657 7.59 0.441 4995 -0.308 0.758 No
15 yr. old girls 7.21 0.478 5595 6.10 0.471 5037 1.654 0.098 No
Drink at least once a week
13 yr. old boys 2.25 0.237 6522 4.31 0.342 6022 -4.953 0.000 Yes
13 yr. old girls 2.38 0.239 6743 4.15 0.308 5877 -4.541 0.000 Yes
15 yr. old boys 11.96 0.593 5714 17.10 0.651 5023 -5.836 0.000 Yes
15 yr. old girls 12.98 0.676 5607 15.59 0.680 5031 -2.722 0.006 Yes
Drank in the last week
13 yr. old boys 3.83 0.342 6166 6.51 0.429 5605 -4.883 0.000 Yes
13 yr. old girls 4.22 0.334 6431 5.93 0.394 5520 -3.311 0.001 Yes
15 yr. old boys 15.89 0.674 5485 20.38 0.738 4834 -4.491 0.000 Yes
15 yr. old girls 18.91 0.78 5479 19.67 0.786 4901 -0.686 0.492 No
Taken drugs in last month
13 yr. old boys 3.14 0.286 6193 4.25 0.363 5684 -2.402 0.016 Yes
13 yr. old girls 2.88 0.269 6531 2.73 0.269 5634 0.394 0.693 No
15 yr. old boys 13.44 0.667 5476 14.87 0.610 4719 -1.582 0.114 No
15 yr. old girls 8.65 0.51 5490 9.16 0.495 4844 -0.718 0.473 No
Taken drugs in last year
13 yr. old boys 4.92 0.354 6193 6.41 0.415 5684 -2.731 0.006 Yes
13 yr. old girls 4.15 0.335 6531 4.20 0.336 5634 -0.105 0.916 No
15 yr. old boys 18.78 0.762 5476 21.42 0.738 4719 -2.489 0.013 Yes
15 yr. old girls 14.2 0.644 5490 15.73 0.635 4844 -1.692 0.091 No
Taken drugs ever
13 yr. old boys 5.8 0.393 6193 7.37 0.434 5684 -2.681 0.007 Yes
13 yr. old girls 4.92 0.369 6531 4.97 0.355 5634 -0.098 0.922 No
15 yr. old boys 20.89 0.797 5476 24.14 0.786 4719 -2.904 0.004 Yes
15 yr. old girls 15.94 0.695 5490 17.49 0.645 4844 -1.635 0.102 No

Contact

Email: salsus@gov.scot