Scottish Schools Adolescent Lifestyle and Substance Use Survey (SALSUS): technical report 2018

Precision of results and measurement of change

Survey respondents represent only a sample of the total population of 13 and 15 year old pupils in Scotland, and results are therefore subject to sampling error. The sampling error is the amount by which the value of a sample estimate for a particular parameter is expected to differ from its true value in the population sampled from. This means that observed differences between sub-groups may have occurred by chance. Throughout the report we only comment on differences that are statistically significant, where we can be 95% confident that such a difference has not occurred by chance (p<0.05).

The formula for calculating the sampling error (SE) of a simple random sample is:

where p = the estimate of the parameter and n=sample size.

The formula for calculating the sampling error for the differences between two estimators (p1 and p2) derived from two independent samples (assuming a simple random sample) is:

Rather than using a simple random sample, whole classes were sampled within the schools that agreed to participate. Therefore, classes were clustered within those schools, and pupils were clustered within those classes. Pupils within the same class and within the same school were more likely to be similar to each other, and therefore values cannot be assumed to be independent of one another. Further details on the calculation of standard errors and design effects are provided below.

It is important to recognise that sampling error is only one of the sources of error that affect the accuracy of survey results. Other sources of error include response bias (previously discussed) and over/under reporting^[16], both of which are difficult to quantify.

Calculating standard errors and design effects

The sample design of SALSUS is complex, involving stratification by local authority and school type (state or independent), as well as clustering within schools. In addition, weights were applied when obtaining survey estimates.

Complex design and weighting affects standard errors for survey estimates, which are generally higher than the standard errors that would be derived from an unweighted simple random sample (SRS) of the same size. For example, clustering reduces the precision of estimates, whereas stratification can increase precision. Weighting can also reduce the precision of estimates.

The ratio of the standard error of the complex sample to that of a simple random sample of the same size is known as the design factor. Put another way, the design factor (or 'Deft') is the factor by which the standard error of an estimate from a simple random sample has to be multiplied to give the true standard error of the complex design.

A Design Factor greater than 1.0 indicates a confidence interval wider than it would be with simple random sampling, meaning that the precision of estimates is reduced. A Design Factor of less than 1.0 indicates a narrower confidence interval and greater precision.

The true standard errors and Defts for SALSUS 2018 were calculated using the Taylor series linearization method as for previous waves of the survey. Thirty-five strata were included for the calculation of standard errors and Defts, one for each combination of local authority and school type (state or independent sector). For eight local authorities: Argyll and Bute, Dundee City, East Renfrewshire, Fife, Inverclyde, Perth and Kinross, South Lanarkshire and Stirling, the variance between clusters could not be calculated for independent schools because there was only one independent school sampled in each of these local authorities. The independent schools in these eight local authorities were combined into one stratum to calculate sampling errors and Defts. There were 1,126 clusters used in the calculations, one for each class that participated in the survey^[17].

The Deft values applied and adjusted true standard errors (which are themselves estimates subject to random sampling error) are shown in Tables 5 to 10 for six key variables from the survey along with 95% confidence intervals.

When comparing the 2018 key variables with the 2015 key variables, significance tests were applied based on the 'pooled standard error' for each variable (a weighted sum of the true standard errors^[18] for each year). Table 11 shows the six key variables for 2015 and 2018, with the true standard errors for each year and whether or not the difference is significant at the 5% level.

Table 5: Linearised standard errors and 95% confidence intervals for the proportion of pupils who are regular smokers, by age and gender: Scotland 2018

	Sample Size	Proportion	Linearised Standard error	Binomial Wald 95% CI		Defts
				Lower CI	Upper CI
13 year olds
Boys	5951	1.52%	0.22%	1.14%	2.02%	1.388
Girls	5869	1.73%	0.19%	1.39%	2.14%	1.114
15 year olds
Boys	4995	7.59%	0.44%	6.77%	8.50%	1.176
Girls	5037	6.10%	0.47%	5.23%	7.09%	1.397

Table 6: Linearised standard errors and 95% confidence intervals for the proportion of pupils who drink alcohol at least once a week, by age and gender: Scotland 2018

	Sample Size	Proportion	Linearised Standard error	Binomial Wald 95% CI		Defts
				Lower CI	Upper CI
13 year olds
Boys	6022	4.31%	0.34%	3.68%	5.03%	1.307
Girls	5877	4.15%	0.31%	3.59%	4.80%	1.184
15 year olds
Boys	5023	17.10%	0.65%	15.85%	18.41%	1.226
Girls	5031	15.59%	0.68%	14.30%	16.97%	1.330

Table 7: Linearised standard errors and 95% confidence intervals for the proportion of pupils who drank alcohol in the last week, by age and gender: Scotland 2018

	Sample Size	Proportion	Linearised Standard error	Binomial Wald 95% CI		Defts
				Lower CI	Upper CI
13 year olds
Boys	5605	6.51%	0.43%	5.71%	7.40%	1.303
Girls	5520	5.93%	0.39%	5.21%	6.76%	1.239
15 year olds
Boys	4834	20.38%	0.74%	18.97%	21.87%	1.274
Girls	4901	19.67%	0.79%	18.17%	21.26%	1.384

Table 8: Linearised standard errors and 95% confidence intervals for the proportion of pupils who have ever used drugs, by age and gender: Scotland 2018

	Sample Size	Proportion	Linearised Standard error	Binomial Wald 95% CI		Defts
				Lower CI	Upper CI
13 year olds
Boys	5684	7.37%	0.43%	6.56%	8.27%	1.253
Girls	5634	4.97%	0.36%	4.31%	5.71%	1.227
15 year olds
Boys	4719	24.14%	0.79%	22.63%	25.72%	1.261
Girls	4844	17.49%	0.64%	16.26%	18.79%	1.181

Table 9: Linearised standard errors and 95% confidence intervals for the proportion of pupils who have used drugs in the last year, by age and gender: Scotland 2018

	Sample Size	Proportion	Linearised Standard error	Binomial Wald 95% CI		Defts
				Lower CI	Upper CI
13 year olds
Boys	5684	6.41%	0.42%	5.64%	7.27%	1.278
Girls	5634	4.20%	0.34%	3.59%	4.91%	1.257
15 year olds
Boys	4719	21.42%	0.74%	20.01%	22.90%	1.235
Girls	4844	15.73%	0.63%	14.52%	17.01%	1.213

Table 10: Linearised standard errors and 95% confidence intervals for the proportion of pupils who have used drugs in the last month, by age and gender: Scotland 2018

	Sample Size	Proportion	Linearised Standard error	Binomial Wald 95% CI		Defts
				Lower CI	Upper CI
13 year olds
Boys	5684	4.25%	0.36%	3.59%	5.02%	1.356
Girls	5634	2.73%	0.27%	2.25%	3.31%	1.239
15 year olds
Boys	4719	14.87%	0.61%	13.71%	16.11%	1.178
Girls	4844	9.16%	0.49%	8.23%	10.18%	1.193

Table 11: Statistical significance of comparisons between 2015 and 2018 results for key variables

	2015 SALSUS			2018 SALSUS			T test (2 sided)	P – Value	Significant at the 5% level?
	%	True standard error	Sample size	%	True standard error	Sample size
Regular smokers
13 yr old boys	1.52	0.188	6439	1.52	0.220	5951	0.000	1.000	No
13 yr. old girls	1.65	0.211	6712	1.73	0.189	5869	-0.282	0.778	No
15 yr. old boys	7.39	0.478	5657	7.59	0.441	4995	-0.308	0.758	No
15 yr. old girls	7.21	0.478	5595	6.10	0.471	5037	1.654	0.098	No
Drink at least once a week
13 yr. old boys	2.25	0.237	6522	4.31	0.342	6022	-4.953	0.000	Yes
13 yr. old girls	2.38	0.239	6743	4.15	0.308	5877	-4.541	0.000	Yes
15 yr. old boys	11.96	0.593	5714	17.10	0.651	5023	-5.836	0.000	Yes
15 yr. old girls	12.98	0.676	5607	15.59	0.680	5031	-2.722	0.006	Yes
Drank in the last week
13 yr. old boys	3.83	0.342	6166	6.51	0.429	5605	-4.883	0.000	Yes
13 yr. old girls	4.22	0.334	6431	5.93	0.394	5520	-3.311	0.001	Yes
15 yr. old boys	15.89	0.674	5485	20.38	0.738	4834	-4.491	0.000	Yes
15 yr. old girls	18.91	0.78	5479	19.67	0.786	4901	-0.686	0.492	No
Taken drugs in last month
13 yr. old boys	3.14	0.286	6193	4.25	0.363	5684	-2.402	0.016	Yes
13 yr. old girls	2.88	0.269	6531	2.73	0.269	5634	0.394	0.693	No
15 yr. old boys	13.44	0.667	5476	14.87	0.610	4719	-1.582	0.114	No
15 yr. old girls	8.65	0.51	5490	9.16	0.495	4844	-0.718	0.473	No
Taken drugs in last year
13 yr. old boys	4.92	0.354	6193	6.41	0.415	5684	-2.731	0.006	Yes
13 yr. old girls	4.15	0.335	6531	4.20	0.336	5634	-0.105	0.916	No
15 yr. old boys	18.78	0.762	5476	21.42	0.738	4719	-2.489	0.013	Yes
15 yr. old girls	14.2	0.644	5490	15.73	0.635	4844	-1.692	0.091	No
Taken drugs ever
13 yr. old boys	5.8	0.393	6193	7.37	0.434	5684	-2.681	0.007	Yes
13 yr. old girls	4.92	0.369	6531	4.97	0.355	5634	-0.098	0.922	No
15 yr. old boys	20.89	0.797	5476	24.14	0.786	4719	-2.904	0.004	Yes
15 yr. old girls	15.94	0.695	5490	17.49	0.645	4844	-1.635	0.102	No