Publication - Research and analysis

Health and social care staff experience: report 2018

Published: 1 Feb 2019

The Scottish Government has commissioned Webropol Ltd, an independent company to undertake the work to report staff experience in Health and Social Care in 2018. Their report provides detailed information and analysis of staff experience gleaned from our iMatter Continuous Improvement Model.

69 page PDF

2.7 MB

69 page PDF

2.7 MB

Contents
Health and social care staff experience: report 2018
Part 1: Response Rate Analysis

69 page PDF

2.7 MB

Part 1: Response Rate Analysis

All questions on the iMatter questionnaire are mandatory to eliminate partial loss of data. Only those questionnaires that have every question answered can be included within the analysis. The Response Rate shows the number of staff issued with the questionnaire (Recipients) and the number of staff who responded (Respondents) as an overall percentage. A response rate of 60% is required for teams of 5 or more and 100% for teams of 4 or less to generate a report. This is to ensure anonymity and the higher the response rate, the more realistic the feedback of how staff feel about working in their team. In total 177,619 questionnaires were issued and 104,254 usable responses were received. This equates to an overall response rate of 59%.

Comparing 2017 to 2018 Response Rates

Response rates at individual Board level vary considerably from 51% in NHS Highland and NHS Western Isles to 91% in NHS Health Scotland. All Boards with a response rate of 60% or over have received a report.

Organisation Response rate 2017 Response Rate 2018 Response Rate Movement (pp)
Health and Social Care 63% 59% -4
Golden Jubilee Foundation 68% 63% -5
Healthcare Improvement Scotland 80% 86% +6
NHS 24 67% 70% +3
NHS Ayrshire & Arran 63% 59% -4
NHS Borders 61% 53% -8
NHS Dumfries & Galloway 63% 59% -4
NHS Education for Scotland 81% 84% +3
NHS Fife 62% 53% -9
NHS Forth Valley 65% 62% -3
NHS Grampian 64% 60% -4
NHS Greater Glasgow & Clyde 58% 54% -4
NHS Health Scotland 85% 91% +6
NHS Highland 58% 51% -7
NHS Lanarkshire 65% 62% -3
NHS Lothian 65% 63% -2
NHS National Services Scotland 76% 77% +1
NHS Orkney 73% 83% +10
NHS Shetland 61% 56% -5
NHS Tayside 65% 58% -7
NHS Western Isles 52% 52% 0
Scottish Ambulance Service 64% 64% 0
The State Hospital 78% 77% -1

Whilst the overall response rate dropped 4 percentage points (pp), there is considerable variation at individual Board level. NHS Orkney achieved a 10 percentage point increase in response rate from an already strong 73% to one of the highest at 83%.

NHS Fife saw the biggest drop in response rate from 62% to 53%. As the 2018 response rate was less than 60% NHS Fife did not receive a report for 2018. Other Boards with large drops in response rates were:

  • NHS Borders -8 pp
  • NHS Highland -7 pp
  • NHS Tayside -7 pp
  • NHS Shetland -5 pp

There are 3 Boards that did not receive a report in either 2017 or 2018. They are:

  • NHS Greater Glasgow & Clyde
  • NHS Highland
  • NHS Western Isles

As would be expected from the overall response rate data, it is generally the Boards with a higher share of paper questionnaires that are less likely to get a report.

Three Boards did not achieve the 60% response rate threshold for online or for paper questionnaires:

  • NHS Shetland
  • NHS Highland
  • NHS Western Isles

The difference between online and paper response rates

Overall 84% of the questionnaires were issued via email and 16% were on paper. The response rates differ greatly between the two methods, with online generating a far higher response rate than paper questionnaires.

Health and Social Care 2018 Volume issued Usable Response Volume % Response Rate
Email 149,557 95,693 64%
Paper 28,062 8,561 31%
Total 177,619 104,254 59%

This difference in response rate by method resulted in 92% of the total usable responses coming from online completions, and only 8% from paper questionnaires.

The variability in response rates across Boards is evident in both online and paper methods: The online response rate ranges from 91% for Health Scotland to 55% for NHS Shetland. The paper response rate ranges from 54% for Golden Jubilee Foundation to 6% for NHS Borders.

Health and Social Care 2018 Volume issued % of Volume issued Volume returned % of Volume issued
Email 149,557 84% 95,693 92%
Paper 28,062 16% 8,561 8%
Total 177,254 104,254

Response Rates by Board

Note: 4 Boards (Health Improvement Scotland. NHS 24, NHS Education for Scotland, NHS Health Scotland) distributed all questionnaires electronically

 

Response rates
Email Paper Total
Health and Social Care 64% 31% 59%
Golden Jubilee Foundation 64% 54% 63%
Healthcare Improvement Scotland 86% N/A 86%
NHS 24 70% N/A 70%
NHS Ayrshire & Arran 64% 40% 59%
NHS Borders 62% 6% 53%
NHS Dumfries & Galloway 62% 32% 59%
NHS Education for Scotland 84% N/A 84%
NHS Fife 60% 18% 53%
NHS Forth Valley 65% 24% 62%
NHS Grampian 66% 38% 60%
NHS Greater Glasgow & Clyde 62% 22% 54%
NHS Health Scotland 91% N/A 91%
NHS Highland 57% 27% 51%
NHS Lanarkshire 66% 41% 62%
NHS Lothian 67% 43% 63%
NHS National Services Scotland 77% 15% 77%
NHS Orkney 83% 14% 83%
NHS Shetland 56% 33% 56%
NHS Tayside 62% 25% 58%
NHS Western Isles 59% 16% 52%
Scottish Ambulance Service 64% 31% 64%
The State Hospital 79% 45% 77%

Shaded boards did not get 2018 iMatter EEI report

The table above shows the response rate for questionnaires returned by email and by post for each Board. It illustrates that the postal response rate is always lower than the email response rate. However, it also shows that there is no clear relationship between the response rates for the two methods within each Board. Nor did we see any relationship between size of Board and response rates.

Geographic and National Boards

However, differences can be seen between the National and Geographic Boards (see below for lists of each type of Board) Response rates are higher among the National Boards, both for email and paper. Only 1% of paper questionnaires distributed were to staff in National Boards. Given higher response rates for online questionnaires generally, this increases still further the overall response rates for the National compared with Geographic Boards.

Response Rates and Volumes Email Paper Total
Geographic Boards
Issued questionnaires 136,376 27,651 164,027
Responses 86,238 8,372 94,610
Response Rate 63% 30% 58%
National Boards
Issued questionnaires 13,181 411 13,592
Responses 9,455 189 9,644
Response Rate 72% 46% 71%
Share of questionnaires issued
Geographic Boards 91% 99% 92%
National Boards 9% 1% 8%
Share of responses
Geographic Boards 90% 98% 91%
National Boards 10% 2% 9%
Geographic Boards National Boards
NHS Ayrshire & Arran Golden Jubilee Foundation
NHS Borders Healthcare Improvement Scotland
NHS Dumfries & Galloway NHS 24
NHS Fife NHS Education for Scotland
NHS Forth Valley NHS Health Scotland
NHS Grampian NHS National Services Scotland
NHS Greater Glasgow & Clyde Scottish Ambulance Service
NHS Highland The State Hospital
NHS Lanarkshire
NHS Lothian
NHS Orkney
NHS Shetland
NHS Tayside
NHS Western Isles

Response rate comparison 2017 to 2018

The overall response rate has dropped by 4 percentage points (pp) from 63% in 2017 to 59% in 2018. Within this the online response rate has dropped 4 pp, but the paper response rate has dropped by 7 pp.

Health and Social Care 2017 2017 Volume issued Usable Response Volume 2017 % Response Rate 2017 % Response Rate 2018 Change in Response Rate (pp) 2017-18
Email 141,990 96,631 68% 64% -4
Paper 30,597 11,599 38% 31% -7
Total 172,587 108,230 63% 59% -4

Compared to last year, the share of online questionnaires issued has increased marginally from 82% (to 84%) and the proportion of questionnaires completed online in 2017 was also lower at 89% compared to 92% in 2018.

Unusable responses

There were 8,625 recipients of the online questionnaire who clicked on the link but failed to finish the questionnaire.

A prerequisite of iMatter is that every question on the questionnaire is answered. If there are any errors on the questionnaire then it is not processed. For the online questionnaire this is monitored within the script and so non-completion is not an issue.

On the paper questionnaire it is not possible to monitor in this way and so it is only when returned questionnaires are processed that incomplete or incorrectly completed responses are identified.

The table below shows details of the reasons for paper responses being rejected and the incidence of each type.

Paper Questionnaires Number Percentage of Questionnaires Issued Percentage of Questionnaires Returned
Questionnaires Issued 28,062
No Response 16,856 60%
Valid Response 8,561 31% 76%
Rejected Questionnaire 2,645 9% 24%
Partial Response 1,278 5% 11%
Completion Errors 670 2% 6%
Duplicate 245 1% 2%
Past Deadline 452 2% 4%

In order to achieve the required 60% response rate, only 1429 additional completed questionnaires would have been needed. Given 2645 paper responses were rejected, the 60% response rate target could conceivably have been met.

Partial Response Analysis

As shown above 1,278 returned paper questionnaires were only partial responses, and so have not been included in the results. This comprises 5% of all paper questionnaires issued, and 11% of those questionnaires returned. Further analysis shows that the level of missing questions is very low.

29% do not answer the thermometer question (overall experience). Of the Staff Governance Standards the My Team section is the most likely to be left completely blank.

Whole Sections Unanswered

The individual questions most likely to be left unanswered are as shown below. Many of these questions are ones that the respondent may feel unable to answer.

For example, some staff may feel that particular elements do not apply or are not relevant to them (e.g. relating to career progression, training etc.). Others are statements that the respondent may feel unable to answer because they do not have enough knowledge or perhaps do not fully understand the question.

Statement

I am given the time and resources to support my learning and growth
I am confident my ideas and suggestions are listened to
I am confident performance is well managed in my team
I get the information I need to do my job well
I feel senior managers responsible for the wider organisation are sufficiently visible
I am treated with dignity and respect as an individual
I have confidence and trust in senior managers responsible for the wider organisation
I feel involved in decisions relating to my job
I am confident performance is managed well within my organisation
I get the help and support I need from other teams and services within the organisation to do my job
I have sufficient support to do my job well
I am confident my ideas and suggestions are acted upon

Summary

Overall the response rate fell just short of the required 60% for a National Report to be issued, a drop of 4 percentage points from last year. Nine Boards also failed to reach the threshold, an increase from the three without reports last year.

Response rates vary considerably across the Boards, with the National Boards typically achieving higher response rates than Geographic Boards.

There is considerable difference in the response rates between the two methodologies. Online, which accounts for 84% of the volume of questionnaires issued, achieves a 64% response rate. Paper questionnaires, whist only a small proportion of the questionnaires, only achieved a 31% response rate this year, down 7 percentage points from last year. The requirement for 100% completion of the questionnaire leads to a notable volume of returned questionnaires being rejected. This report recommends further investigation into the paper questionnaire, both in terms of how it is distributed and how it is processed, in order to improve the response rate.

Whilst this report explores where the response rate has moved it cannot assess the reasons for that movement. However, given the variations at Board level both in absolute response rates and movement from last year, further work with staff is recommended, in order to understand why staff have chosen not to participate, or otherwise feel unable to do so. In particular, this might explore further why a small proportion of staff do not feel able to fully complete the questionnaire. It should be recalled that a fully-independent academic evaluation of the model has been commissioned, this will report in 2019 and may present findings in respect of these issues.


Contact

Email:

Workforce Practice Admin HWFadmin@gov.scot
Liz Reilly Elizabeth.Reilly@gov.scot
Catriona Hetherington Catriona.Hetherington@gov.scot