iMatter Health & Social Care Staff Experience Survey 2025

Appendix 6: Statistical Notes

Significance Testing

Significance testing has been carried out on the iMatter data, to assess whether year to year movements in results are likely to be ‘true’, rather than ‘chance’. Specifically, a series of t-tests have been used to examine the size of change needed to give us a very high level of confidence that a ‘true’ change has happened. The key element here is the number of responses – the larger the number of responses, the smaller the minimum change that can be deemed statistically significant (meaning that the change is highly likely to be ‘true’). Additionally, it should be noted that significance test results will vary by question as the variability in responses given impacts whether the differences are significant or not.

Significance testing and the example percentage point changes below assume a random sample of different people every year. In reality, this is unlikely to hold for 2 main reasons.

1. Those who complete the survey may hold characteristics different to the health staff group as a whole, reducing randomness. This leads to results being more likely to be displayed as significant than in reality, and larger movement needed for a significant change.

2. Individuals are also likely to complete in the survey in more than one year, reducing independence. This has the opposite effect leading to results less likely to be displayed as significant than in reality, and smaller movements needed for a significant change.

Overall Health and Social Care Level Data

• A change of 0.3, or even 0.2 at times, is significant across Health and Social Care as a whole. This generally means where reported scores have changed in 2025 from 2024, it is likely to be a ‘true’ change.

Board Level Data

The number of respondents (the achieved sample size) is key to the level of movement year on year that is significant. Therefore, for individual Boards, in general, significant movements are:

• Boards with less than 800 responses per year: movements of 3 points are generally significant
• Boards with between 800 and 2,800 responses per year: movements of 2 points are generally significant
• Boards with over 2,800 responses per year: movements of 1 point are generally significant

Movements of 3 points are significant

• Healthcare Improvement Scotland
• NHS Western Isles
• The State Hospital
• NHS Orkney

Movements of 2 points are significant

• NHS Shetland
• NHS 24
• NHS Golden Jubilee
• NHS Borders
• Public Health Scotland
• NHS Education for Scotland
• NHS National Services Scotland

Movements of 1 point are significant

• NHS Tayside
• NHS Grampian
• NHS Lothian
• NHS Greater Glasgow and Clyde
• Scottish Ambulance Service
• NHS Forth Valley
• NHS Highland
• NHS Fife
• NHS Ayrshire & Arran
• NHS Lanarkshire
• NHS Dumfries & Galloway

Full details of the significance testing can be found in the iMatter 2025 Data File^[19]. As can be seen with the National data there are incidences where significant movements are not evident in data reported at whole integer level only.

Note: where response rate is particularly high the extent of movement that is significant will be lower than those shown here.