### Precision of results and measurement of change

Survey respondents represent only a sample of the total population of 13 and 15 year old pupils in Scotland, and results are therefore subject to sampling error. The sampling error is the amount by which the value of a sample estimate for a particular parameter is expected to differ from its true value in the population sampled from. This means that observed differences between sub-groups may have occurred by chance. Throughout the report we only comment on differences that are statistically significant, where we can be 95% confident that such a difference has not occurred by chance (p<0.05).

The formula for calculating the sampling error (SE) of a simple random sample is:

where p = the estimate of the parameter and n=sample size.

The formula for calculating the sampling error for the differences between two estimators (p1 and p2) derived from two independent samples (assuming a simple random sample) is:

Rather than using a simple random sample, whole classes were sampled within the schools that agreed to participate. Therefore, classes were clustered within those schools, and pupils were clustered within those classes. Pupils within the same class and within the same school were more likely to be similar to each other, and therefore values cannot be assumed to be independent of one another. Further details on the calculation of standard errors and design effects are provided below.

It is important to recognise that sampling error is only one of the sources of error that affect the accuracy of survey results. Other sources of error include response bias (previously discussed) and over/under reporting^{[16]}, both of which are difficult to quantify.

**Calculating standard errors and design effects**

The sample design of SALSUS is complex, involving stratification by local authority and school type (state or independent), as well as clustering within schools. In addition, weights were applied when obtaining survey estimates.

Complex design and weighting affects standard errors for survey estimates, which are generally higher than the standard errors that would be derived from an unweighted simple random sample (SRS) of the same size. For example, clustering reduces the precision of estimates, whereas stratification can increase precision. Weighting can also reduce the precision of estimates.

The ratio of the standard error of the complex sample to that of a simple random sample of the same size is known as the design factor. Put another way, the design factor (or 'Deft') is the factor by which the standard error of an estimate from a simple random sample has to be multiplied to give the true standard error of the complex design.

A Design Factor greater than 1.0 indicates a confidence interval wider than it would be with simple random sampling, meaning that the precision of estimates is reduced. A Design Factor of less than 1.0 indicates a narrower confidence interval and greater precision.

The true standard errors and Defts for SALSUS 2018 were calculated using the Taylor series linearization method as for previous waves of the survey. Thirty-five strata were included for the calculation of standard errors and Defts, one for each combination of local authority and school type (state or independent sector). For eight local authorities: Argyll and Bute, Dundee City, East Renfrewshire, Fife, Inverclyde, Perth and Kinross, South Lanarkshire and Stirling, the variance between clusters could not be calculated for independent schools because there was only one independent school sampled in each of these local authorities. The independent schools in these eight local authorities were combined into one stratum to calculate sampling errors and Defts. There were 1,126 clusters used in the calculations, one for each class that participated in the survey^{[17]}.

The Deft values applied and adjusted true standard errors (which are themselves estimates subject to random sampling error) are shown in Tables 5 to 10 for six key variables from the survey along with 95% confidence intervals.

When comparing the 2018 key variables with the 2015 key variables, significance tests were applied based on the 'pooled standard error' for each variable (a weighted sum of the true standard errors^{[18]} for each year). Table 11 shows the six key variables for 2015 and 2018, with the true standard errors for each year and whether or not the difference is significant at the 5% level.

**Table 5: Linearised standard errors and 95% confidence intervals for the proportion of pupils who are regular smokers, by age and gender: Scotland 2018**

Sample Size | Proportion | Linearised Standard error | Binomial Wald 95% CI | Defts | ||
---|---|---|---|---|---|---|

Lower CI | Upper CI | |||||

13 year olds |
||||||

Boys | 5951 | 1.52% | 0.22% | 1.14% | 2.02% | 1.388 |

Girls | 5869 | 1.73% | 0.19% | 1.39% | 2.14% | 1.114 |

15 year olds |
||||||

Boys | 4995 | 7.59% | 0.44% | 6.77% | 8.50% | 1.176 |

Girls | 5037 | 6.10% | 0.47% | 5.23% | 7.09% | 1.397 |

**Table 6: Linearised standard errors and 95% confidence intervals for the proportion of pupils who drink alcohol at least once a week, by age and gender: Scotland 2018**

Sample Size | Proportion | Linearised Standard error | Binomial Wald 95% CI | Defts | ||
---|---|---|---|---|---|---|

Lower CI | Upper CI | |||||

13 year olds |
||||||

Boys | 6022 | 4.31% | 0.34% | 3.68% | 5.03% | 1.307 |

Girls | 5877 | 4.15% | 0.31% | 3.59% | 4.80% | 1.184 |

15 year olds |
||||||

Boys | 5023 | 17.10% | 0.65% | 15.85% | 18.41% | 1.226 |

Girls | 5031 | 15.59% | 0.68% | 14.30% | 16.97% | 1.330 |

**Table 7: Linearised standard errors and 95% confidence intervals for the proportion of pupils who drank alcohol in the last week, by age and gender: Scotland 2018**

Sample Size | Proportion | Linearised Standard error | Binomial Wald 95% CI | Defts | ||
---|---|---|---|---|---|---|

Lower CI | Upper CI | |||||

13 year olds |
||||||

Boys | 5605 | 6.51% | 0.43% | 5.71% | 7.40% | 1.303 |

Girls | 5520 | 5.93% | 0.39% | 5.21% | 6.76% | 1.239 |

15 year olds |
||||||

Boys | 4834 | 20.38% | 0.74% | 18.97% | 21.87% | 1.274 |

Girls | 4901 | 19.67% | 0.79% | 18.17% | 21.26% | 1.384 |

**Table 8: Linearised standard errors and 95% confidence intervals for the proportion of pupils who have ever used drugs, by age and gender: Scotland 2018**

Sample Size | Proportion | Linearised Standard error | Binomial Wald 95% CI | Defts | ||
---|---|---|---|---|---|---|

Lower CI | Upper CI | |||||

13 year olds |
||||||

Boys | 5684 | 7.37% | 0.43% | 6.56% | 8.27% | 1.253 |

Girls | 5634 | 4.97% | 0.36% | 4.31% | 5.71% | 1.227 |

15 year olds |
||||||

Boys | 4719 | 24.14% | 0.79% | 22.63% | 25.72% | 1.261 |

Girls | 4844 | 17.49% | 0.64% | 16.26% | 18.79% | 1.181 |

**Table 9: Linearised standard errors and 95% confidence intervals for the proportion of pupils who have used drugs in the last year, by age and gender: Scotland 2018**

Sample Size | Proportion | Linearised Standard error | Binomial Wald 95% CI | Defts | ||
---|---|---|---|---|---|---|

Lower CI | Upper CI | |||||

13 year olds |
||||||

Boys | 5684 | 6.41% | 0.42% | 5.64% | 7.27% | 1.278 |

Girls | 5634 | 4.20% | 0.34% | 3.59% | 4.91% | 1.257 |

15 year olds |
||||||

Boys | 4719 | 21.42% | 0.74% | 20.01% | 22.90% | 1.235 |

Girls | 4844 | 15.73% | 0.63% | 14.52% | 17.01% | 1.213 |

**Table 10: Linearised standard errors and 95% confidence intervals for the proportion of pupils who have used drugs in the last month, by age and gender: Scotland 2018**

Sample Size | Proportion | Linearised Standard error | Binomial Wald 95% CI | Defts | ||
---|---|---|---|---|---|---|

Lower CI | Upper CI | |||||

13 year olds |
||||||

Boys | 5684 | 4.25% | 0.36% | 3.59% | 5.02% | 1.356 |

Girls | 5634 | 2.73% | 0.27% | 2.25% | 3.31% | 1.239 |

15 year olds |
||||||

Boys | 4719 | 14.87% | 0.61% | 13.71% | 16.11% | 1.178 |

Girls | 4844 | 9.16% | 0.49% | 8.23% | 10.18% | 1.193 |

**Table 11: Statistical significance of comparisons between 2015 and 2018 results for key variables**

2015 SALSUS | 2018 SALSUS | T test (2 sided) | P – Value | Significant at the 5% level? | |||||
---|---|---|---|---|---|---|---|---|---|

% | True standard error | Sample size | % | True standard error | Sample size | ||||

Regular smokers | |||||||||

13 yr old boys | 1.52 | 0.188 | 6439 | 1.52 | 0.220 | 5951 | 0.000 | 1.000 | No |

13 yr. old girls | 1.65 | 0.211 | 6712 | 1.73 | 0.189 | 5869 | -0.282 | 0.778 | No |

15 yr. old boys | 7.39 | 0.478 | 5657 | 7.59 | 0.441 | 4995 | -0.308 | 0.758 | No |

15 yr. old girls | 7.21 | 0.478 | 5595 | 6.10 | 0.471 | 5037 | 1.654 | 0.098 | No |

Drink at least once a week | |||||||||

13 yr. old boys | 2.25 | 0.237 | 6522 | 4.31 | 0.342 | 6022 | -4.953 | 0.000 | Yes |

13 yr. old girls | 2.38 | 0.239 | 6743 | 4.15 | 0.308 | 5877 | -4.541 | 0.000 | Yes |

15 yr. old boys | 11.96 | 0.593 | 5714 | 17.10 | 0.651 | 5023 | -5.836 | 0.000 | Yes |

15 yr. old girls | 12.98 | 0.676 | 5607 | 15.59 | 0.680 | 5031 | -2.722 | 0.006 | Yes |

Drank in the last week | |||||||||

13 yr. old boys | 3.83 | 0.342 | 6166 | 6.51 | 0.429 | 5605 | -4.883 | 0.000 | Yes |

13 yr. old girls | 4.22 | 0.334 | 6431 | 5.93 | 0.394 | 5520 | -3.311 | 0.001 | Yes |

15 yr. old boys | 15.89 | 0.674 | 5485 | 20.38 | 0.738 | 4834 | -4.491 | 0.000 | Yes |

15 yr. old girls | 18.91 | 0.78 | 5479 | 19.67 | 0.786 | 4901 | -0.686 | 0.492 | No |

Taken drugs in last month | |||||||||

13 yr. old boys | 3.14 | 0.286 | 6193 | 4.25 | 0.363 | 5684 | -2.402 | 0.016 | Yes |

13 yr. old girls | 2.88 | 0.269 | 6531 | 2.73 | 0.269 | 5634 | 0.394 | 0.693 | No |

15 yr. old boys | 13.44 | 0.667 | 5476 | 14.87 | 0.610 | 4719 | -1.582 | 0.114 | No |

15 yr. old girls | 8.65 | 0.51 | 5490 | 9.16 | 0.495 | 4844 | -0.718 | 0.473 | No |

Taken drugs in last year | |||||||||

13 yr. old boys | 4.92 | 0.354 | 6193 | 6.41 | 0.415 | 5684 | -2.731 | 0.006 | Yes |

13 yr. old girls | 4.15 | 0.335 | 6531 | 4.20 | 0.336 | 5634 | -0.105 | 0.916 | No |

15 yr. old boys | 18.78 | 0.762 | 5476 | 21.42 | 0.738 | 4719 | -2.489 | 0.013 | Yes |

15 yr. old girls | 14.2 | 0.644 | 5490 | 15.73 | 0.635 | 4844 | -1.692 | 0.091 | No |

Taken drugs ever | |||||||||

13 yr. old boys | 5.8 | 0.393 | 6193 | 7.37 | 0.434 | 5684 | -2.681 | 0.007 | Yes |

13 yr. old girls | 4.92 | 0.369 | 6531 | 4.97 | 0.355 | 5634 | -0.098 | 0.922 | No |

15 yr. old boys | 20.89 | 0.797 | 5476 | 24.14 | 0.786 | 4719 | -2.904 | 0.004 | Yes |

15 yr. old girls | 15.94 | 0.695 | 5490 | 17.49 | 0.645 | 4844 | -1.635 | 0.102 | No |

### Contact

Email: salsus@gov.scot