Part 1: Response Rate Analysis
All questions on the iMatter questionnaire are mandatory to eliminate partial loss of data. Only those questionnaires that have every question answered can be included within the analysis. The Response Rate shows the number of staff issued with the questionnaire (Recipients) and the number of staff who responded (Respondents) as an overall percentage. A response rate of 60% is required for teams of 5 or more and 100% for teams of 4 or less to generate a report. This is to ensure anonymity and the higher the response rate, the more realistic the feedback of how staff feel about working in their team. In total 177,619 questionnaires were issued and 104,254 usable responses were received. This equates to an overall response rate of 59%.
Comparing 2017 to 2018 Response Rates
Response rates at individual Board level vary considerably from 51% in NHS Highland and NHS Western Isles to 91% in NHS Health Scotland. All Boards with a response rate of 60% or over have received a report.
|Organisation||Response rate 2017||Response Rate 2018||Response Rate Movement (pp)|
|Health and Social Care||63%||59%||-4|
|Golden Jubilee Foundation||68%||63%||-5|
|Healthcare Improvement Scotland||80%||86%||+6|
|NHS Ayrshire & Arran||63%||59%||-4|
|NHS Dumfries & Galloway||63%||59%||-4|
|NHS Education for Scotland||81%||84%||+3|
|NHS Forth Valley||65%||62%||-3|
|NHS Greater Glasgow & Clyde||58%||54%||-4|
|NHS Health Scotland||85%||91%||+6|
|NHS National Services Scotland||76%||77%||+1|
|NHS Western Isles||52%||52%||0|
|Scottish Ambulance Service||64%||64%||0|
|The State Hospital||78%||77%||-1|
Whilst the overall response rate dropped 4 percentage points (pp), there is considerable variation at individual Board level. NHS Orkney achieved a 10 percentage point increase in response rate from an already strong 73% to one of the highest at 83%.
NHS Fife saw the biggest drop in response rate from 62% to 53%. As the 2018 response rate was less than 60% NHS Fife did not receive a report for 2018. Other Boards with large drops in response rates were:
- NHS Borders -8 pp
- NHS Highland -7 pp
- NHS Tayside -7 pp
- NHS Shetland -5 pp
There are 3 Boards that did not receive a report in either 2017 or 2018. They are:
- NHS Greater Glasgow & Clyde
- NHS Highland
- NHS Western Isles
As would be expected from the overall response rate data, it is generally the Boards with a higher share of paper questionnaires that are less likely to get a report.
Three Boards did not achieve the 60% response rate threshold for online or for paper questionnaires:
- NHS Shetland
- NHS Highland
- NHS Western Isles
The difference between online and paper response rates
Overall 84% of the questionnaires were issued via email and 16% were on paper. The response rates differ greatly between the two methods, with online generating a far higher response rate than paper questionnaires.
|Health and Social Care 2018||Volume issued||Usable Response Volume||% Response Rate|
This difference in response rate by method resulted in 92% of the total usable responses coming from online completions, and only 8% from paper questionnaires.
The variability in response rates across Boards is evident in both online and paper methods: The online response rate ranges from 91% for Health Scotland to 55% for NHS Shetland. The paper response rate ranges from 54% for Golden Jubilee Foundation to 6% for NHS Borders.
|Health and Social Care 2018||Volume issued||% of Volume issued||Volume returned||% of Volume issued|
Response Rates by Board
Note: 4 Boards (Health Improvement Scotland. NHS 24, NHS Education for Scotland, NHS Health Scotland) distributed all questionnaires electronically
|Health and Social Care||64%||31%||59%|
|Golden Jubilee Foundation||64%||54%||63%|
|Healthcare Improvement Scotland||86%||N/A||86%|
|NHS Ayrshire & Arran||64%||40%||59%|
|NHS Dumfries & Galloway||62%||32%||59%|
|NHS Education for Scotland||84%||N/A||84%|
|NHS Forth Valley||65%||24%||62%|
|NHS Greater Glasgow & Clyde||62%||22%||54%|
|NHS Health Scotland||91%||N/A||91%|
|NHS National Services Scotland||77%||15%||77%|
|NHS Western Isles||59%||16%||52%|
|Scottish Ambulance Service||64%||31%||64%|
|The State Hospital||79%||45%||77%|
Shaded boards did not get 2018 iMatter EEI report
The table above shows the response rate for questionnaires returned by email and by post for each Board. It illustrates that the postal response rate is always lower than the email response rate. However, it also shows that there is no clear relationship between the response rates for the two methods within each Board. Nor did we see any relationship between size of Board and response rates.
Geographic and National Boards
However, differences can be seen between the National and Geographic Boards (see below for lists of each type of Board) Response rates are higher among the National Boards, both for email and paper. Only 1% of paper questionnaires distributed were to staff in National Boards. Given higher response rates for online questionnaires generally, this increases still further the overall response rates for the National compared with Geographic Boards.
|Response Rates and Volumes||Paper||Total|
|Share of questionnaires issued|
|Share of responses|
|Geographic Boards||National Boards|
|NHS Ayrshire & Arran||Golden Jubilee Foundation|
|NHS Borders||Healthcare Improvement Scotland|
|NHS Dumfries & Galloway||NHS 24|
|NHS Fife||NHS Education for Scotland|
|NHS Forth Valley||NHS Health Scotland|
|NHS Grampian||NHS National Services Scotland|
|NHS Greater Glasgow & Clyde||Scottish Ambulance Service|
|NHS Highland||The State Hospital|
|NHS Western Isles|
Response rate comparison 2017 to 2018
The overall response rate has dropped by 4 percentage points (pp) from 63% in 2017 to 59% in 2018. Within this the online response rate has dropped 4 pp, but the paper response rate has dropped by 7 pp.
|Health and Social Care 2017||2017 Volume issued||Usable Response Volume 2017||% Response Rate 2017||% Response Rate 2018||Change in Response Rate (pp) 2017-18|
Compared to last year, the share of online questionnaires issued has increased marginally from 82% (to 84%) and the proportion of questionnaires completed online in 2017 was also lower at 89% compared to 92% in 2018.
There were 8,625 recipients of the online questionnaire who clicked on the link but failed to finish the questionnaire.
A prerequisite of iMatter is that every question on the questionnaire is answered. If there are any errors on the questionnaire then it is not processed. For the online questionnaire this is monitored within the script and so non-completion is not an issue.
On the paper questionnaire it is not possible to monitor in this way and so it is only when returned questionnaires are processed that incomplete or incorrectly completed responses are identified.
The table below shows details of the reasons for paper responses being rejected and the incidence of each type.
|Paper Questionnaires||Number||Percentage of Questionnaires Issued||Percentage of Questionnaires Returned|
In order to achieve the required 60% response rate, only 1429 additional completed questionnaires would have been needed. Given 2645 paper responses were rejected, the 60% response rate target could conceivably have been met.
Partial Response Analysis
As shown above 1,278 returned paper questionnaires were only partial responses, and so have not been included in the results. This comprises 5% of all paper questionnaires issued, and 11% of those questionnaires returned. Further analysis shows that the level of missing questions is very low.
29% do not answer the thermometer question (overall experience). Of the Staff Governance Standards the My Team section is the most likely to be left completely blank.
The individual questions most likely to be left unanswered are as shown below. Many of these questions are ones that the respondent may feel unable to answer.
For example, some staff may feel that particular elements do not apply or are not relevant to them (e.g. relating to career progression, training etc.). Others are statements that the respondent may feel unable to answer because they do not have enough knowledge or perhaps do not fully understand the question.
I am given the time and resources to support my learning and growth
I am confident my ideas and suggestions are listened to
I am confident performance is well managed in my team
I get the information I need to do my job well
I feel senior managers responsible for the wider organisation are sufficiently visible
I am treated with dignity and respect as an individual
I have confidence and trust in senior managers responsible for the wider organisation
I feel involved in decisions relating to my job
I am confident performance is managed well within my organisation
I get the help and support I need from other teams and services within the organisation to do my job
I have sufficient support to do my job well
I am confident my ideas and suggestions are acted upon
Overall the response rate fell just short of the required 60% for a National Report to be issued, a drop of 4 percentage points from last year. Nine Boards also failed to reach the threshold, an increase from the three without reports last year.
Response rates vary considerably across the Boards, with the National Boards typically achieving higher response rates than Geographic Boards.
There is considerable difference in the response rates between the two methodologies. Online, which accounts for 84% of the volume of questionnaires issued, achieves a 64% response rate. Paper questionnaires, whist only a small proportion of the questionnaires, only achieved a 31% response rate this year, down 7 percentage points from last year. The requirement for 100% completion of the questionnaire leads to a notable volume of returned questionnaires being rejected. This report recommends further investigation into the paper questionnaire, both in terms of how it is distributed and how it is processed, in order to improve the response rate.
Whilst this report explores where the response rate has moved it cannot assess the reasons for that movement. However, given the variations at Board level both in absolute response rates and movement from last year, further work with staff is recommended, in order to understand why staff have chosen not to participate, or otherwise feel unable to do so. In particular, this might explore further why a small proportion of staff do not feel able to fully complete the questionnaire. It should be recalled that a fully-independent academic evaluation of the model has been commissioned, this will report in 2019 and may present findings in respect of these issues.