Developing the Young Workforce (DYW) regional groups: formative evaluation

5. Monitoring and reporting

Chapter Summary

• The Scottish Government identified an overarching set of KPIs for the DYW programme of activity.
• After their formation, each DYW Regional Group agreed an individual set of KPIs with the National Group. These were based on the National KPIs and tailored to suit their specific areas of focus.
• The result is that the four groups included within the evaluation report against a combined total of 30 unique KPIs, even although they cover very similar measures.
• It is not possible to accurately assess how well or otherwise each of the groups are performing based on analysis of their KPI reports due to issues in the way data has been collected and reported.
• National and Regional consultees report that they would welcome a more streamlined and consistent set of KPIs for the DYW Regional Groups to report against. This would form the basis for any future impact assessment.
• The monitoring and reporting arrangements for the Groups were found to work well at the regional level, mainly due to the comprehensive CRM systems used by the Chambers, but were less effective for enabling reporting at the national level.

Introduction

5.1 This chapter looks at how DYW activity is being monitored and reported by the Regional Groups. It begins with an overview of the KPIs that the four groups included within the evaluation are currently reporting against. This is followed by a summary of the feedback received on the extent to which the current monitoring and reporting arrangements are meeting the information needs of regional and national stakeholders.

Key Performance Indicators (KPIs)

The Scottish Government identified an overarching set of KPIs for the DYW Regional Groups.

5.2 DYW Regional Groups are required to report against an agreed set of KPIs biannually in April and October. Prior to the formation of the Regional Groups, a number of KPIs were identified by the Scottish Government as being key to supporting wider DYW programme delivery^[10]. These were:

• % of employers (all employers and small / micro businesses) offering work placements to young people (school, college and other programmes)
• % of employers (all employers and small / micro businesses) offering work inspiration activities ( e.g. school visits, workplace visits, mentoring)
• Number of secondary schools in partnership with employers
• % of employers recruiting Modern Apprentices aged 16-24
• % of small and micro businesses recruiting Modern Apprentices aged 16-24
• Number of employers achieving Investors in Young People accreditation.

5.3 Following their formation, each of the DYW Regional Groups agreed a unique set of KPIs with the National Group that they would report against. These were based on those set out in the National Framework, but also tailored to the reflect planned areas of activity / focus within each of the regions.

The four DYW Regional Groups are reporting against a total of 30 unique KPIs.

5.4 The four DYW Regional Groups included within the evaluation each report against between six and 12 KPIs – a combined total of 30. Whilst there is a lot of similarity across the four groups in terms of the types of things they are measuring, their KPIs are all worded differently and therefore not directly comparable.

5.5 The 30 KPIs were categorised according to theme (Table 5‑1). They point to a strong focus on employer and school engagement and the brokerage of links between these. The groups also all report on employer recruitment of Modern Apprentices and Investors in Young People accreditation. There are several KPIs that are unique to individual groups reflecting their priority areas of activity.

Table 5‑1: Categorisation of current KPIs used by the four DYW Regional Groups

Category of KPI	No. of KPIs
Employer engagement and / or participation	9
Employer recruitment of Modern Apprenticeships	8
School engagement and / or participation	6
School and Employer Partnerships	4
Investors in Young People accreditation for Employers	4
Other employment outcomes for young people (not MAs)	1
Parent / Guardian Engagement	1
Innovation	1

Source: SQW review of 4 DYW Regional Group KPIs
*Base: 30 individual KPIs (some have been allocated to multiple categories)

There is limited scope for read-across and comparison of performance against the KPIs that the Regional Groups currently report against.

5.6 Whilst there is a reasonable level of consistency across the four sets of KPIs in terms of the broad themes that they relate to, the emphases differ between the groups and from the original set of national KPIs identified by the Scottish Government. A further issue is that there are variations between the groups in terms of how progress towards their KPIs is being measured and reported and the baselines are not always clear.

5.7 Table 5-2 details some other issues with the KPIs that that Regional Groups are currently reporting against, along with some examples to illustrate each point.

Table 5‑2: Key issues with the KPIs being reported against by DYW Regional Groups

Issues	Examples
Focussed on activities / outputs rather than outcomes / impacts	"Number of employers offering work placements to young people" "Number of parents / guardians engaged" "Number of employers offering apprenticeships"
They are not SMART (Specific, Measurable, Attainable, Relevant, Time-bound)	"Number of schools in membership of local business support organisation e.g. Chamber of Commerce" "Ensure all secondary pupils experience meaningful work placements"
Variations in how similar indicators are described	"Number of secondary schools in a partnership with an employer" "To achieve meaningful SME secondary school partnerships with employers"
Indicators that are dependent on factors outside the direct influence of DYW Regional Groups	"To increase the number of employers achieving IiYP accreditation tenfold in 3 years" "To increase total number of Modern Apprenticeships (MAs) – 16 – 24yr olds"

Source: SQW review of DYW Regional Group KPIs

There is appetite from regional stakeholders for greater clarity and consistency in the KPIs they are reporting against.

5.8 A general comment from regional stakeholders was that the KPIs they are currently reporting against do not reflect the broad range of activities that they deliver. A further complaint was that many of them are out with their direct influence and therefore not appropriate for them to be reporting against. For example, they all report against employer recruitment of apprenticeships. However, it would be very difficult for them to demonstrate a causal impact on this indicator given the range of factors that influence it.

"Our KPIs are other organisations' performance indicators. We can have an influence on some outputs, but we are by no means responsible for them. We can raise awareness of Invest in Young People (IIYP) accreditation, but we are not the driving force behind it and therefore shouldn't have KPIs relating to it. The same applies for MAs and FAs."
DYW Regional Consultee.

5.9 At the time of the evaluation, the Scottish Government was in the process of developing a new streamlined set of four National KPIs that all Regional Groups would be required to report against. The intention was that this would bring greater clarity and consistency, enabling read-across and reporting at the National level. A draft of these was shared with the Regional Groups in advance of the workshops. Whilst the feedback on these was generally positive, there were some concerns raised. In particular, some of the terminology used (such as "strategic partnerships") was felt to be open to interpretation. The consensus was that any new KPIs introduced would need to be accompanied with clear guidance on definitions.

Feedback on monitoring and reporting

Board members reported high levels of satisfaction with the monitoring and reporting arrangements at the regional level.

5.10 More than half (52%) of Board Members that responded to the survey said they thought the current performance monitoring and arrangements of the Regional Group were very effective and a further 44% reported that they were effective (Figure 5‑1). This suggests high levels of satisfaction at the regional level with current arrangements for reporting locally.

5.11 This was confirmed in the discussions at the regional workshops. When probed on the success factors, Board Members referenced the comprehensive CRM systems used by the Chambers of Commerce. These have enabling detailed tracking of all DYW-related activity, including all contact made with schools and employers within each region. They also enable comprehensive reports of activity, which form an agenda at the Regional Group Board Meetings.

Figure 5‑1: How effective do you think the current performance monitoring and reporting arrangements are for the DYW Regional Group?

Source: SQW survey of Regional Group Board Members
Base: 25

National consultees described information on the performance and impact of the Regional Groups as "patchy".

5.12 Feedback from National Group members on the monitoring and reporting arrangements for the Groups was generally less positive. Several referenced the high volume of unique KPIs that are currently being reported against by the Regional Groups (150+). This makes it impossible to read across and get an accurate picture of what is being delivered nationally. It is also thought to be contributing to a general lack of clarity around what the Groups as there to achieve.

5.13 In addition to reporting against their KPIs biannually, Regional Groups also submit a narrative progress report to the Scottish Government every two months. These reports are condensed into short summaries for circulating to all National Group Members. Link Members also provide a verbal update on progress and any issues arising within their respective regions at each National Group meeting. Again, there is limited potential for read-across and identification of common themes across these narrative reports, particularly the condensed versions that are accessed by National Group members.

Regional Groups report that they would like more feedback on progress and performance.

5.14 The narrative progress reports that the Regional Groups submit to the Scottish Government every two months detail progress towards their objectives, as well as details of what is working well and where there are issues / challenges. The Groups also submit a performance report twice a year reporting on progress towards their agreed KPIs. The consensus amongst regional consultees was that this tends to be a one-way flow of information, with little coming back in terms of feedback.

5.15 The view amongst the Regional Groups is that they would like more of a two-way conversation with the Scottish Government on progress and performance. They would like assurances that they are on track with what they are delivering. They would also like to know how they are performing relative to the other Regional Groups or the national picture.

Evidence on the scale of the in-kind contribution from employers that has been leveraged through the Regional Groups was described as a key gap.

5.16 An information gap identified by several consultees relates to the value of the contribution of employers to DYW activities. This is potentially huge and is not currently being captured anywhere. It is a significant added value component to DYW (as compared to other public sector interventions). One suggestion was to introduce a consistent way of measuring employer contributions – for example, by hours.

5.17 Several consultees also commented on the need to report on the profile of employers engaged, for example by size. The consensus was that it was 'easier' to get larger employers on board, as they have available resources, but that there was higher added value from engaging small and medium sized firms. This was in fact identified as a priority area of activity within the National Framework for the establishment of the Groups, but is not currently being reported against.

"It's all very well getting large employers on board. In a way, that is easy as they have more time and resource, as well as corporate objectives around CSR / recruitment. However, the major cultural change comes from getting SMEs on board. This is a much harder sell and therefore a bigger achievement." DYW National Consultee