We are testing a new beta website for gov.scot go to new site

Evaluation of the Scottish union learning fund— Year one (2000-2001)

Listen

Evaluation of the Scottish Union Learning Fund - Year 1 (2000-2001)

Chapter three: project outputs

Introduction

3.1 Chapter three introduces the main quantitative outputs from the 12 round one projects. The quantitative outputs were collected by the SULF questionnaire which was designed and distributed by York Consulting. A 100% response rate to the questionnaire was achieved.

3.2 Quantifying the overall performance of SULF projects masks the variety of project models, activities and partnerships involved and the different contexts in which many projects have developed. However, obtaining an overview of the outcomes gives an indication of the extent to which capacity to deliver learning has been enhanced. The following indicators were used to measure the range of SULF project outcomes:

  • number of people attending awareness/briefing sessions
  • number of training needs analyses undertaken
  • numbers of learners enrolled
  • number of Individual Learning Accounts opened
  • number of accredited courses developed/accessed
  • number of other courses/materials developed
  • number of learning centres developed/enhanced
  • number of people achieving a qualification
  • number of employers involved.

3.3 This framework captures the majority of, but not all, activities. The indicators are also limited in the extent to which more strategic impacts can be measured. To address this, four projects from round one have been followed-up as case studies to assess impacts in qualitative terms. These are used as the basis for illustrating key themes and issues identified in Chapters 5-8.

3.4 The quantitative outputs are analysed and presented under the following headings:

  • target setting
  • outputs achieved
  • achievements against targets
  • barriers to achieving targets
  • cost per output
  • output per project.

Target Setting

3.5 A third of all of the projects did not outline quantifiable targets in their initial project proposals. Two thirds of projects in Year One set at least one quantifiable target.

3.6 Table 3.1 analyses the number of projects setting quantifiable targets for each of the different indicators and the number of projects reporting outcomes against the indicators. A total of 8 projects set at least one quantifiable target. The targets were set mainly related to the number of learning reps trained or the number of learners.

Table 3.1 The number of projects setting quantifiable targets against each of the targets and number of projects reporting outcomes on this indicator

Activity

Number of projects setting quantifiable targets

(n=12)

Number of projects reporting an output against the target

(n=12)

Awareness Raising

0

10

Identifying learner needs

2

10

Learners starting courses

5

7

LRs trained

6

9

LRs and learners achieving qualifications

0

8

Number of accredited courses/qualifications that were developed

1

6

Number of other courses developed

1

5

Number of Learning Centres developed

0

4

Number of employers involved

0

12

3.7 The table clearly shows that projects are generally not setting quantifiable targets (except for the number of LRs and learners trained) yet quantifiable outputs are being recorded. This is an issue that should be addressed by SULF fund managers in both the bidding process and the assessment of bids. The bidding criteria should be clear that quantitative targets must be set and the assessment of the bids should ensure that such targets are set, which can subsequently be using for monitoring progress. Data should be collected on the actual outputs and monitored against targets. Clarification of how monitoring data will be collected, when, and by whom is a key task that needs to be agreed in the early stages of the project. Whilst the achievement of targets is not the only objective of the Scottish Union Learning Fund, the ability of unions to plan, monitor and achieve outputs is a key factor, which helps to demonstrate that union capacity to develop and deliver learning services has increased.

Outputs achieved

3.8 Table 3.2 summarises the cumulative outputs as reported by all of the Year One projects in July 2002 3.

Table 3.2 Cumulative outputs from the Round One Projects

Outputs
(n=12)

Number of learners starting courses

1,266

Number of LRs starting courses

196

Number attending awareness sessions

1,776

Number of learning representatives trained - initial

147

Number of LRs trained - further

52

Number achieving qualifications

433

Number of accredited courses developed

9

Number of other materials developed

21

Number of employers directly involved

99

Number of learning centres established

15

Number of ILAs opened

1,116

Number of ILAs used

510

Number of training needs analyses 4

5,768

3.9 The cumulative outputs reported in the table are the total outputs from all 12 of the projects. Significant achievements of the fund include the number of learners and learning representatives starting courses, the number of people attending awareness raising sessions and the number of training needs analyses undertaken. In addition, the development of 15 learning centres is also a significant achievement and it is to be hoped that learners will continue to be able to access learning after the SULF project ends.

3.10 Reflecting on past experience and from the case studies, it is likely that the 99 employers reported as directly involved in the SULF projects is an inflated figure. As part of the English ULF evaluation an employer impact study was undertaken, during the study it became apparent that employers that had submitted letters of support often had no direct involvement in the projects and project managers could not provide details on all of the employers involved. The figure of 99 employers directly involved should be treated with caution, particularly since during the employer survey the number of employers for whom details were available was very low 5.

3.11 In addition, on contacting the non-case study employers for which project managers had supplied details, a significant amount of the employers had no direct involvement with the SULF project. Of the 5 employers surveyed, 4 had no direct involvement.

Achievements to target

3.12 Table 3.3 presents the cumulative targets and cumulative performance of the Year One projects. The table indicates that all of the targets set have been achieved. Some of the targets, such as the number of training needs analyses have been exceeded significantly. Project managers have recognised their relative inexperience in setting targets as shown by the mismatch between the actual figures and the targets set. This was also an issue that arose in the evaluation of the Wales Union learning Fund 6.

Table 3.3 Cumulative outputs against Cumulative Targets

Outputs

Cumulative Targets

Cumulative Outputs

(n=12)

(n=12)

Number of learners starting courses

388

1,266

Number of LRs starting courses

179

196

Number attending awareness sessions

-

1,776

Number of learning representatives trained - initial

-

147

Number of LRs trained - further

-

52

Number achieving qualifications

-

433

Number of accredited courses developed

1

9

Number of other materials developed

7

21

Number of employers directly involved

-

99

Number of learning centres established

-

15

Number of ILAs opened

320

1,116

Number of ILAs used

-

510

Number of training needs analyses

245

5,768

Performance of Projects Covering Different Themes

3.13 The analysis of performance by project theme shows that no specific type of project is significantly more likely to achieve its targets, although there are a number of key factors that help ensure success. Lack of success in achieving targets is related to the following:

  • a tendency for some project bids to list a range of themes and targets trying to 'cover all bases'
  • setting unrealistic targets
  • notable external factors affecting achievement include the closure of the national ILA framework.

Cost per output

3.14 Table 3.4 presents an analysis of cost per output for the key indicators over the funding round. The data shows the average cost of producing one unit of output. The calculation only includes the projects that report this output, but does not account for the proportion of the project budget used in attaining these outputs. These costs are calculated by dividing the total costs of projects that have reported an output against the indicator by the total outputs reported against the indicator.

3.15 Cost per output is presented for illustrative purposes only and for future comparisons to analyse if the cost per unit of output has fallen, which is to be anticipated as project managers become more experienced. The costs per unit of output appear high, which is to be expected as a high cost/lower outputs are associated with the initial development of projects.

Table 3.4 Cost per output

Cost per Output

SULF

()

Number of learners starting courses

297

Number of LRs staring courses

2,367

Number attending awareness sessions

261

Number of learning representatives trained - initial

1,909

Number of LRs trained - further

3,521

Number of learners achieving qualifications

561

Number of LRs achieving qualifications

2,377

Number of accredited courses developed

31,584

Number of other materials developed

8,387

Number of employers directly involved

5,673

Number of learning centres established

12,059

Number of ILAs opened

208

Number of ILAs used

258

Number of training needs analyses

80

Key Points

  • Two thirds of projects set at least one quantifiable target
  • 1,266 learners and 196 learning representatives were trained
  • All of the cumulative targets set were achieved
  • Costs per output appear high, but it is to be hoped this will reduce as project managers gain experience.