Annex A: Statistical Reliability
The respondents to the questionnaire are only a sample of the total 'population'. We cannot therefore be certain that the figures obtained are exactly those we would have if everybody had been interviewed (the 'true' values). However, we can predict the variation between the sample results and the 'true' values from a knowledge of the size of the samples, on which the results are based and the number of times that a particular answer is given.
The confidence with which we can make this prediction is usually chosen to be 95% - that is, the chances are 19 in 20 that the 'true' value will fall within a specified range. The table below illustrates the predicted ranges for different sample sizes and percentages results at the '95% confidence interval'; based on a random sample.
Table A.1: Predicted ranges for different sample sizes at the 95% confidence interval
Size of sample on which survey result is based | Approximate sampling tolerances applicable to percentages at or near these levels |
---|
10% or 90% + | 30% or 70% + | 50% + |
---|
100 interviews | 5.9 | 9.0 | 9.8 |
---|
200 interviews | 4.2 | 6.4 | 6.9 |
---|
300 interviews | 3.4 | 5.2 | 5.7 |
---|
500 interviews | 2.6 | 4.0 | 4.4 |
---|
1,015 interviews | 1.8 | 2.8 | 3.1 |
---|
Source: MORI
*For example, on a question where 50% of the people in a sample of 500 respond with a particular answer, the chances are 95 in 100 that this result would not vary by more than four percentage points, plus or minus from a complete coverage of the entire population using the same procedures. However, while it is true to conclude that the "actual" result (95 times out of 100) lies anywhere between 44% and 56%, it is proportionately more likely to be closer to the centre of this band ( i.e. at 50%).
Tolerances are also involved in the comparison of results from different parts of a sample. A difference, in other words, must be of at least a certain size to be considered statistically significant. The following table is a guide to the sampling tolerances applicable to comparisons.
Table A.2: Sampling tolerances
Size of samples compared | Differences required for significance at or near percentage levels |
---|
10% or 90% + | 30% or 70% + | 50% + |
---|
100 and 100 | 8.4 | 12.8 | 13.9 |
---|
200 and 200 | 5.9 | 9.0 | 9.8 |
---|
200 and 400 | 5.1 | 7.8 | 8.5 |
---|
200 and 500 | 4.9 | 7.5 | 8.2 |
---|
500 and 500 | 3.7 | 5.7 | 6.2 |
---|
1,000 and 500 | 3.2 | 4.9 | 5.4 |
---|
Source: MORI
Table A.3: Demographic sub-group comparisons
Size of samples compared | Differences required for significance at or near percentage levels |
---|
10% or 90% + | 30% or 70% + | 50% + |
---|
Males vs. females (459 vs. 556) | 3.7 | 5.7 | 6.2 |
---|
Age 44 and under vs. 45+ (418 vs. 597) | 3.8 | 5.7 | 6.3 |
---|
20% Least deprived areas in Scotland vs. 20% most deprived areas in Scotland (217 vs. 259) | 5.4 | 8.3 | 9.0 |
---|
Users vs. non-users (403 vs. 606) | 3.8 | 5.8 | 6.3 |
---|
Source: MORI