ISBN 0 7559 2891 1 (Web only publication)
This document is also available in pdf format (439k)
This document summarises the key findings from the second Scottish Executive Stakeholder Survey. The results are positive overall and there has been some improvement in perceptions since the first survey was conducted in autumn 2004.
Just over three quarters (76%) of respondents were satisfied with the dealings they have had with the Executive over the previous year, a higher proportion than in 2004 (72%). Additionally, almost two in five (39%) said they would speak highly of the Executive and a similar proportion felt the organisation's approach to engagement has improved in the past year.
Looking at specific aspects of engagement, the Scottish Executive is seen to be performing particularly well on:
- being committed to achieving its objectives
- listening to others
- being focussed on delivery
- having sensible policies
On each of these measures perceptions have improved significantly since the 2004 survey. There has also been a notable increase in the proportion of stakeholders who feel that the Scottish Executive is joined-up, although stakeholders still feel that this is an issue for the Executive.
Significant minorities of respondents, however, continue to be dissatisfied with their dealings with the Scottish Executive (8%), say they would speak critically of the organisation (18%) and feel that engagement has worsened over the past year (9%). In line with the 2004 survey, some stakeholders still feel the Executive is unclear about its priorities (29%) and is reluctant to innovate (24%).
The Executive is seen to be performing well against many of its critical success factors but there has been a slight worsening of perceptions since the 2004 survey in some areas, especially understanding the constraints facing stakeholders, which showed a significant decline from 55% in 2004 to 47% in 2005. However, there has been a notable increase over the past year in the number of stakeholders who feel that the Executive is innovative in its policy design.
Analysis of the importance stakeholders gave to the critical success factors and their ratings of Executive performance on these factors, suggests that key areas for improvement include:
- providing timely and appropriate responses
- understanding constraints facing stakeholders
- involving stakeholders at an early stage in developing policy
- being willing to take stakeholders' views on board
- openly sharing information
- being clear about how much is open to change in policy making
MORI Scotland was commissioned by the Office of the Chief Researcher on behalf of the Permanent Secretary to undertake the second comprehensive survey of the Scottish Executive's stakeholders. The main aims of the survey were to:
- assess stakeholders' overall perceptions of the Executive
- identify how stakeholders engage with the Executive and what stakeholders feel about these relationships
- provide ratings across a number of critical success factors in relation to key aspects of stakeholders' relationships with the Executive
- provide recommendations for improving stakeholder engagement in and across the Executive
Between 7th September and 14th October 2005, MORI Scotland conducted an online and telephone survey with stakeholders across all sectors, based on lists provided by Executive departments. A total 1,099 stakeholders responded to the survey. The stakeholders surveyed included representatives from non-departmental public bodies ( NDPBs), local government, voluntary groups, registered charities, private sector organisations, central government departments and academic institutions.
The stakeholder organisations ranged in size from less than 10 people to over 500. Almost three in five (57%) said that their organisation delivered services on behalf of the Executive; while a large proportion said it advised (36%) or lobbied (30%) the Executive on specific issues. Just over four in five said their organisation was funded wholly or in part by the Executive (24% and 60% respectively). Thirty-four percent of respondents had been with their organisation for 10 years or less, while 32% had been there for 5-10 years and 34% for more than 10 years. The profile of respondents in the 2005 survey was very similar to that in 2004.
Where appropriate, this paper refers to benchmarking findings (provided by ORC International) comparing results from this survey with those from business-to-business customer satisfaction surveys conducted by other government departments and agencies over the past three years.
GENERAL IMPRESSIONS OF THE SCOTTISH EXECUTIVE
Just over three-quarters of stakeholders were satisfied with the dealings they have had with the Executive over the previous 12 months, while 8% were dissatisfied and 15% were neutral. Since 2004, both overall satisfaction and the percentage of those very satisfied have increased significantly (Table 1). However, the Executive's satisfaction score (76%) is lower than the average level of satisfaction (88%) for other government departments and agencies benchmarked, though the gap has narrowed between 2004 and 2005.
Table 1: Overall satisfaction with the Scottish Executive
Q In the last 12 months, how satisfied or dissatisfied have you been with the dealings you have had with the Scottish Executive?
Base: All respondents
Neither satisfied nor dissatisfied
Change in the last 12 months
As Figure 1 below shows, around two in five (43%) respondents felt that the Executive's approach to engagement had improved in the last 12 months, while a similar proportion (44%) felt there had been no change and 9% felt it had got worse. The Scottish Executive's score for perceived improvement in the last 12 months (43%) is higher than the average for other government departments and agencies benchmarked (39%).
Public/private (58%) sector organisations and academic institutions (52%) were among those most likely to say that the Executive had improved, as were stakeholders who had dealt with the ELLD1 (48%) and the Development Department ( DD) (48%).
Figure 1: Perceived progress in stakeholder engagement in the last 12 months
The survey sought to assess the extent to which stakeholders went further than being satisfied and would act as advocates for the Executive by asking respondents to what extent they would be willing to speak highly of the organisation among their peers.
Thirteen per cent were 'advocates', saying they would speak highly of the organisation without being asked. Another quarter (26%) said that they would be positive about the Executive if asked for their opinion. Meanwhile, 15% said they would be critical if asked and 3% said they would be critical without being asked. The number of 'advocates' has decreased since 2004 but, overall, the number of stakeholders saying they would speak positively or negatively about the Executive has remained consistent. Again, stakeholders in contact with ELLD were more likely to be positive than respondents as a whole.
Experiences of Working with the SE
Just under a third of stakeholders (32%) dealt with the Executive on at least a weekly basis, while 37% dealt with the organisation monthly and 30% three to four times a year or less often. The main methods of contact were e-mail (88%), face-to-face meetings (76%) and telephone conversations (73%). A majority have had frequent contact for seeking information or advice, responding to a consultation over a specific policy, receiving information on grant processing, receiving assistance interpreting legislation and providing advice to the Executive. The most common reason for contacting the Executive was to receive information, although almost all of the organisations had contact for each of the reasons at some time.
To assess stakeholders' views on specific aspects of working with the Executive, respondents were invited to comment on their experiences of just one of the departments with which they had been in contact. Specifically, they were presented with a list of phrases and asked whether or not these fitted with their selected department. Of the 7 positive phrases presented to respondents, there was most agreement that the department was committed to achieving its objectives and least agreement that it delivers value for money and is joined-up:
- committed to achieving objectives (88%)
- listens to others (77%)
- focussed on delivery (73%)
- has sensible policies (70%)
- understands my organisation (69%)
- delivers value for money (39%)
- joined-up (36%)
Of the 5 negative phrases presented to respondents, there was most agreement that the department was unclear about its priorities and least agreement that it lacked effective leadership and was remote and impersonal:
- unclear about priorities (29%)
- reluctant to innovate (24%)
- out of touch (17%)
- lacks effective leadership (15%)
- remote and impersonal (15%)
Those who were satisfied with the Executive overall were significantly more likely to respond positively than those who were dissatisfied. Additionally, ELLD was consistently rated more positively on each of the measures than the Executive as a whole.
When compared with the 2004 findings, the results suggest that the Executive has made progress in the past year. In particular, there has been a rise in the number of stakeholders saying the Executive is joined-up, committed to achieving its objectives, listens to others, focused on delivery, understands the stakeholder organisation and has sensible policies. However, issues remain concerning clarity of priorities, reluctance to innovate and to a lesser extent being out of touch, lacking effective leadership and being remote and impersonal.
Respondents who felt the Executive was not joined-up were asked what problems, if any, this had caused for them. The most commonly-mentioned themes included poor communication and dissemination of information, policies not being linked, confusion, a lack of consistency, internal disagreements, time wasting and duplication of effort. These themes were discussed in relation to join up both across the Executive and within individual departments.
Critical Success Factors
Much of the assessment of departments will reflect the outcomes for stakeholders and the extent to which policies, funding and delivery mechanisms adopted by the SE reflect the priorities, aspirations and needs of different stakeholder groups. However, while it might be inevitable that some are disappointed by outcomes, there should be more unanimity about the Executive's performance in terms of the process of stakeholder engagement. The Executive has identified 19 'critical success factors' against which it wishes to measure its performance in the area of stakeholder engagement. For each factor, respondents were asked to indicate first, how important it was to them in their dealings with their selected department and, second, how they felt their department was currently performing against that factor.
Perceived Importance of the Critical Success Factors
Each of the success factors was regarded as important by almost all respondents, which was also the case in the 2004 survey. However, as in 2004, there was some variation in the degree of importance attached to each. The factors rated most important were: providing timely and appropriate responses, having staff that are knowledgeable about their area of work, understanding what stakeholders are trying to achieve and the constraints that they are required to work within. Factors seen to be of least importance were consistency of personnel and innovation in terms of policy design. The relative importance of each factor expressed as a z-score 2 is shown in Figure 2, ranked from top to bottom.
Figure 2: Relative importance of the critical success factors
Perceived Performance on Key Critical Success Factors
Respondents were also asked to rate performance on the same success factors. Overall, performance was good on many of the success factors (ranking illustrated in Table 2), especially the following:
- Having staff who treat you with courtesy and respect (91% rated 'good' to 'excellent')
- Having staff who are easy to contact (79% rated 'good' to 'excellent')
- Having staff who are knowledgeable about their area of work (78% rated 'good' to 'excellent')
- Providing accurate and reliable information (74% rated 'good' to 'excellent')
Despite these high scores, benchmarking data suggests that the SE's performance in all four areas is not as good as in other government departments and agencies.
- Courtesy and respect rated good to excellent from 91% of SE stakeholders compared to 97% for staff at other government departments and agencies
- Ease of contact rated good to excellent from 79% of SE stakeholders compared to 84% for staff at other government departments and agencies
- Knowledge of area of work rated good to excellent from 79% of SE stakeholders compared to 89% for staff at other government departments and agencies
- Accurate and reliable information rated good to excellent from 74% of SE stakeholders compared to 86% for staff at other government departments and agencies
Table 2: Performance of the SE departments on the critical success factors
Q. I'm now going to read out the same lists of attributes and I'd like you to tell me how you would rate the performance of the [department] in your dealings with it. Could you tell me whether your experience has been excellent, very good, good, average, poor or very poor?
Good to excellent
Poor to very poor
Base: all respondents
Having staff who treat you with courtesy and respect
Having staff who are easy to contact
Having staff who are knowledgeable about their area of work
Providing accurate and reliable information
Being focussed on achieving outcomes and objectives
Having staff who deal with your requests promptly
Provides clear information and corrspondence
Understanding what you are trying to achieve
Being willing to take your views on board
Openly sharing information
Providing timely and appropriate responses
Promoting regular dialogue with other organisations
Involving you at an early stage in developing policy
Promoting a consistent approach to dealing with organisations
Being innovative in its policy design†
In policy making, being clear about how much is open to change
Keeping you informed how thinking is developing
Understanding constraints facing stakeholders
Staff not moving around too much
† In 2004, this read 'Being innovative and willing to take risks'
The Scottish Executive also received high 'good' to 'excellent' ratings on: being focused on achieving outcomes and objectives (73%), dealing with requests promptly (73%) and providing clear information and correspondence (70%).
Performance was rated relatively less positively in the following areas (figure in brackets refers to % rating performance on these aspects as 'good' to 'excellent'):
- maintaining consistency of personnel (47%),
- understanding constraints facing stakeholders (47%)
- informing stakeholders of how their thinking is developing (51%)
- informing stakeholders of how much is open to change in policy making (51%)
Three critical success factors show improvements since 2004. Being innovative in policy design exhibits the largest increase, rising significantly from 43% in 2004 to 53% in 2005. Improvements are also seen in involving stakeholders at an early stage in developing policy and, to a lesser extent, being focused on achieving outcomes. Compared with the 2004 results, however, a number of critical success factors show a decline in positive ratings with 'understanding the constraints facing stakeholders' witnessing a significant decline from 55% in 2004 to 47% in 2005.
Stakeholders who were satisfied with the Scottish Executive overall tended to give their department better ratings on the critical success factors than those who were dissatisfied. Five variables on engagement were found to make a significant contribution to overall satisfaction with the Scottish Executive (39% of the variation in overall satisfaction was explained by the combined influence of these variables). The variables were, in order of importance:
- understanding what you are trying to achieve
- providing timely and appropriate responses
- being willing to take you views on board
- providing accurate and reliable information
- being innovative in its policy design
Areas for Improvement
Investigating the relationship between the perceived importance of the critical success factors and the actual performance of the Executive on these factors, analysis identified 6 areas where performance is at or below average compared with the importance attributed to the characteristic. These suggest key areas for improvement for the Executive as defined by its stakeholders.
1. Providing timely and appropriate responses -most important attribute, but performance below average
2. Understanding constraints facing stakeholders - fourth most important attribute, but performance below average
3. Involving you at an early stage in developing policy - fifth most important attribute, but performance below average
4. Being willing to take your views on board - eighth most important attribute, but performance below average
5. Openly sharing information - ninth most important attribute, but performance is below average
6. In policy making, being clear about how much is open to change - tenth most important attribute but performance below average.
The attribute with the poorest performance, 'staff not moving around too much' had the lowest importance rating. This implies that while performance can be improved, it should not be the highest priority.
These results were also broadly similar to 2004. However, 'providing timely and appropriate responses' and 'being willing to take your views on board' have emerged as more problematic areas in 2005. Meanwhile, 'keeping you informed about how its thinking is developing' and 'promoting a consistent approach to dealing with organisations' were still below average in terms of performance but they were not rated as high on the importance scale as in 2004.