Procurement Efficiencies: Monitoring and Evaluation of Devanha Phase 2

13 Evaluation Conclusions and Lessons Learned

13.1 This chapter summarises the overall evaluation of Devanha and discusses transferable lessons learned from the Devanha experience. It focuses on:

• Value for Money
• Process Improvement
• Collaboration
• Programme Management
• Governance
• Strategic Relationships
• Training & Employment

Value for Money

Evaluation of Performance against Cost and HAG Targets

13.2 We considered performance against cost and grant targets and looked at some high level disaggregation of this performance: framework agreement projects and section 75 developers; tenure; and tranche. Data showed that there was no discernible pattern in performance between tranches and so no breakdown at this level is presented. Figures shown in this section relate to the 35 schemes for which actual costs were available. While the detailed figures vary a little between the 35 schemes and the full 51 schemes, the differences are not significant and do not alter the conclusions.

Table 13.1 Actual Works Cost Compared to Offer of Grant Target

Category	Variation From Offer of Grant Target	Met Target (Yes/No)
Whole Programme	+4.22%	No
Rental Schemes	+4.36%	No
Shared Equity Schemes	+3.60%	No
Framework Schemes	+9.36%	No
Non Framework Schemes	-2.78%	Yes

Based on 35 completed schemes where Actual Cost data was available.

Table 13.2 Actual HAG Level Compared to Offer of Grant Target

Category	Variation From Offer of Grant Target	Met Target (Yes/No)
Whole Programme	-9.22%	Yes
Rental Schemes	-8.98%	Yes
Shared Equity Schemes	-10.56%	Yes
Framework Schemes	-0.33%	Yes
Non Framework Schemes	-20%	Yes

Based on 35 completed schemes where Actual HAG data was available.

While the programme as a whole and all of the sub categories met the Devanha Offer of Grant target for HAG, only the non-framework schemes met this target for works cost.

Value for Money

13.3 The evaluation of value for money is complicated by the extreme changes which occurred in the economy as a whole. It is further complicated by a completely changed grant system which means that grant levels experienced by Devanha are well in excess of those being awarded today. Qualitative opinion from Devanha and its partners suggested that value for money may have been obtained through pain/gain share but that in the particular economic circumstances pertaining at the time, further savings could have been achieved by going out to the market to tender for individual schemes.

13.4 Works Cost: While commenting that Devanha works cost targets do not appear to have been based on evidence that they were achievable, Devanha has failed to deliver value for money against this benchmark. An exception has been the non-framework schemes which outperformed this target.

13.5 HAG: Against the benchmark set out in the Offer of Grant, Devanha has achieved value for money for the programme as a whole and for all of the sub categories of rented/shared equity and framework/non-framework schemes.

13.6 Adjusted Target: It is arguable whether the final adjusted target is a more useful benchmark of procurement efficiency. It was the basis of gain/pain share calculation in the 35 schemes where actual costs are known. Sixteen schemes performed better than this target and were in a gain situation and 14 met the target. Only 5 schemes experienced pain. On the basis of this measure of value for money the majority of schemes have performed to the adjusted target or better at the actual cost stage.

13.7 However, it can be argued that it is in the contractors' interests to revise this adjusted target upward through the compensation events mechanism. Of course, their view was that it was only used for legitimate extras as intended by NEC to recover actual additional costs incurred through no fault of their own. Conversely the procurement consultant's view was that contractors used this as an old fashioned claims approach and were not committed to achieving lowest cost possible. The Devanha consultants' view was that they saw the Scottish Government Validated target as the 'real' target.

13.8 Scottish Government Validated Target: Comparing actual costs with Scottish Government validated works cost targets shows that, at aggregate level, schemes have actually been delivered within this target (table 6.3 refers). This raises the question of what is the 'real' or most appropriate and realistic target against which to gauge performance. The Offer of Grant target was widely considered to be unrealistically low and generally unachievable, but conversely the SG validated targets were a crude benchmark that did little to challenge performance or incentivise contractors to minimise costs. Furthermore the gap between target setting and starting on site could be a matter of months, by which time the market had changed considerably. It was observed that some of the more "successful" schemes were the ones where the target was set at the peak of the marketplace.

13.9 The following table highlights transferable lessons from Devanha in relation to assessing value for money.

Table 13.3 Transferable Lessons: Value for Money

Devanha Experience	Lessons
a. A single works cost target benchmark was used in the Offer of Grant. This was simple and easy to express and understand but too simplistic to be meaningful at scheme level as it doesn't account for site specific risks.	A more refined mechanism for establishing scheme level target cost would be necessary.
b. Although Devanha agreed the ambitious Offer of Grant target it was quickly regarded as unachievable. There was a lack of awareness amongst those delivering the projects and it was not used to drive performance.	There should be more emphasis on involvement of framework partners in understanding cost targets to improve buy-in.
c. A number of KPIs were used in the contract to incentivise performance and inform gain share/pain share on schemes. However, KPIs were not well defined or understood at the outset to clearly drive performance.	Establish and clearly communicate KPIs to all partners in a framework arrangement from the outset to give clearer direction. Promote better understanding of target cost concept of NEC.
d. Value for Money criteria were limited to capital cost target. This was simple and readily understood and measured. However, it lacks important value criteria including design quality, sustainability, life cycle costing.	Consider a number of VFM metrics in addition to capital cost.
e. Devanha introduced a new procurement approach and way of working not previously experienced in the area. The Devanha legacy has been to develop procurement skills and experience in the North East construction sector.	Although the value of this is not measurable within Devanha terms of reference, an innovative approach to procurement increases skills and capacity in the industry.

Evaluation of Process Improvement

13.10 Higher levels of collaboration than experienced through traditional procurement were achieved. Consequent benefits of better communication, trust, open book costing and sharing of buildability experience were attained.

13.11 On the other hand, lack of understanding and experience of the NEC procurement process that is fundamental to operational aspects of Devanha procurement are clear, despite the engagement of a procurement consultant from the outset. Difficulties relating to the operational aspects of implementing the NEC Devanha approach hampered Devanha for a large part of the duration of the initiative.

13.12 Incremental improvements in efficiency gains/cost savings year on year from the Devanha framework approach were not evident. As such the influence of the Devanha process on VFM and cost savings at scheme level is difficult to evaluate.

13.13 Table 13.4 shows transferrable lessons from the Devanha experience in relation to procurement process issues.

Table 13.4 Transferable Lessons: Procurement Process

Devanha Experience	Lessons
a. The selection process for Devanha partners didn't sufficiently consider compatibility with a framework approach. There was a sense that not all partners fully bought into NEC target cost approach and behaved as if traditional procurement.	Amend the selection and appointment process to be more specifically suited to framework contracts.
b. All programme workload was allocated to contractors at outset. This is a comparatively simple and straightforward process but limits ability to review performance and competitiveness mid programme.	Consider staged allocation of framework workload based on performance.
c. Devanha procurement consultant also carried out assessments used to set validated works costs targets. While carried out to professional standard this lacks independence as procurement consultant has a vested interest in inflation assessment.	Consider independent third party assessment of any inflation adjustment to cost targets.
d. Price Data for inflation assessment largely derived from within Devanha partners. Price Data does not effectively represent open market inflation conditions.	While this was unavoidable due to lack of available comparators, greater confidence in qualitative assessments needed would have been given by using a third party consultant.
e. There was an absence of knowledge and experience of NEC within Devanha partners. Cultural readiness and operational capability therefore took a long time to develop.	Carefully consider whether the NEC procurement approach is appropriate for the programme. Possibly modify/adapt to suit.
f. Many schemes were partially or fully developed before framework approach adopted. Opportunity was therefore lost at early stages for Core Group input limiting effectiveness of NEC approach.	Ensure early involvement of supply chain in framework approach to harness benefits.
g. Devanha framework approach adopted the NEC 3 form of contract. CWC and Devanha considered a variety of forms of contract at a workshop. NEC3 enabled a level of collaboration and integration in the Core Group not seen in traditional procurement but significant risks are still retained by the client under NEC3.	Carry out thorough assessment of alternative procurement routes and contracts. Pay particular attention to risk aspects of all contractual arrangements which are considered.
h. Some aspects of an identifiable Devanha specification evolved over time. This allowed for limited economy of scale benefits to be achieved for some elements. On the other hand, lack of comprehensive Devanha specification meant that full bulk buying potential was not achieved. Opportunity for more strategic involvement of subcontractors could not be achieved.	There should be more integration of partners at outset to establish and define an agreed specification.

Evaluation of Collaboration

13.14 Devanha was partially successful in implementing collaborative working. Some aspects of the NEC approach were implemented but only partially and they were still being developed late in the programme. Table 13.5 indicates learning points from this experience.

Table 13.5 Transferable Lessons: Collaboration

Devanha Experience	Lessons
a. The Strategic Client Group promoted collaborative systems from a client perspective e.g. in implementing the framework agreement. However, this was criticised as taking too long to get underway.	There is a need for focus on developing collaborative working from an early stage. This requires leadership, professional support and commitment from the Board.
b. Procurement consultants were engaged to advise on NEC approach. They provided access to the required knowledge and experience of NEC approach which was absent. However, training workshops achieved little; knowledge transfer was superficial and did not extend to practice and application.	Carefully consider remit and terms of engagement of procurement consultant in future. Emphasise the mentoring role in supporting operational and implementation aspects. Ensure that the client has a dedicated project manager to oversee the procurement consultant project. Consider whether scheme PM role should be extended where experience in team is absent.
c. The Core group provided a route for communication for partners including sharing learning. The procurement consultant and Programme Director considered it could have been more proactive.	A core group involving the client and partners is essential for enhancing collaborative working and sharing of ideas which can improve quality. It should be set up early in the programme to advance good working practices and involve partners in specification issues e.g. related to buildability. It is important that staff delivering development are involved.
d. Open book costing and NEC framework meant sharing of savings but in securing longer term collaboration some market testing and price advantages in a falling market were lost.	There is a trade off between competition and collaboration. In a falling market the client may lose out on windfall gains. Nevertheless, the wider benefits of collaboration are accepted and partners should operate framework agreements to maximise efficiency, e.g. through commitment to beating target costs and staged allocation of contracts.
e. There was a generally non-confrontational approach which was assisted by the NEC contract use of compensation events. This did, however, leave the client with significant responsibility for risk.	Carefully consider the contract to be adopted including suitability for collaborative approach and impact on risk.
f. Collaboration was almost exclusively with first tier contractors.	Efforts should be made to expand collaborative working within the supply chain.

Evaluation of Performance in Programme Management

13.15 Overall, Devanha is in the closing stages of managing a substantial programme to a successful completion based on delivery of 1,383 units while meeting grant targets. Table 13.6 shows transferable lessons from the Devanha experience in relation to Programme Management.

Table 13.6 Transferable Lessons: Programme Management

Devanha Experience	Lessons
a. Four year programme.	Multi year programme gives opportunities for continuity of work using a framework agreement.
b. Interim year-on-year grant planning targets.	Interim targets are useful for overall programme management but should be used with as much flexibility as possible.
c. Innovative approach to project approval associated with project delays. Delays caused damage to Devanha's reputation.	Light touch approval gives greater control to RSLs and should be encouraged. However, there is a need for a common understanding of the parameters of scheme assessment to be established early as delays in reaching agreement on individual schemes can have a major impact on programme success. Processes should be tested before the start of the main programme.

Evaluation of Devanha's Governance

13.16 Devanha's governance structure and systems have reflected the tensions between ceding control to the Devanha Board and RSLs' desire to retain control. Over the course of Devanha, the RSLs became more comfortable working together in Devanha but difficult issues remained in relation to apportionment of risk. Devanha also recognised that lack of strong leadership had hampered its performance in a number of ways.

13.17 Table 13.7 highlights transferable lessons in relation to governance. These relate largely to roles and responsibilities, leadership, risk, monitoring and control.

Table 13.7 Transferable Lessons: Governance

Devanha Experience	Lessons
a. Board of 10 directors - two each from each RSL.	The Board composition should be designed to enable more independent scrutiny. It is reasonable for the RSLs to nominate members for the Devanha board, but independence of the board should be enhanced. This should be achieved by a minority membership from RSL executives, and inclusion of independent Board members with no affiliation to any of the RSLs. This should help minimise conflicts of interest and make decision making more efficient.
b. Governance sub-structure with 6 subgroups allowed RSLs to stay involved but was slow set up. Strategic Client Group particularly useful. Others were periodically used.	The sub-structure should be simplified. An efficient Devanha executive and staffing structure would reduce the requirement for a bureaucratic governance sub-structure. Ad-hoc sub-groups could be convened where expertise and individual RSL policy views are required but on an exception basis. More use should be made of the Core Group as an operational rather than strategic group.
c. Senior leadership - Devanha failed to appoint a managing director and this weakness was demonstrated repeatedly in many aspects of Devanha's operation.	Any future collaborative programme should have a leader - Managing Director - from the start with a development team, probably seconded from the existing RSLs to enable effective implementation. Reporting lines of the development staff would be through the Devanha MD.
d. Employment of Programme Director.	This role will not be necessary if there is a managing director and development team in place.
e. Day-to-day control of projects by RSLs allowed use of existing expertise and local control but did not give clear lines of responsibility, reduced opportunities for standardisation and caused confusion for external bodies with lack of branding of Devanha.	All projects which are part of the collaborative programme should be the responsibility of Devanha. Staff working on the projects should be responsible to and managed by Devanha (as above). Responsibility to the RSLs should be through a service level agreement between Devanha and the RSLs.
f. Strategic responsibility split. Difficult in practice to separate "development strategy" (RSLs) and "strategic decisions about execution of the programme" (Devanha).	There should be clear split of responsibility: RSLs decide what projects they want to be involved in within Devanha (because they have risk/local accountability). Devanha takes strategic decisions about all projects which are in programme. Devanha has management responsibility for all projects in the programme. To fulfil its role, Devanha needs to have a dedicated team of development staff with line management responsibility through the Devanha MD.
g. Roles and responsibilities: Devanha's agency role allows for delivery of a large programme but there was lack of clarity in the respective roles of Devanha and the RSLs.	Have clear agreement, setting out responsibilities - Devanha should have full control after decision to include project has been taken. Devanha should have service level agreements with each RSL. RSL role should be to decide on participation/non-participation, selection of schemes to include in Devanha programme and to receive completed units. RSLs could continue to have input to development of Devanha specification.
h. Financial risk: Formation Agreement addressed grant redistribution where there was a surplus but not in the event of a shortfall. Lack of adequate cost information compounded the problem because RSLs were unaware of how much risk they were exposed to until late in the programme.	Analysis of risk including positive and negative scenarios should be undertaken at outset and principles for apportioning risk should be explicit. All RSLs should be aware of risk exposure before entering into a collaborative agreement. Development staff working directly for Devanha will help in apportioning professional responsibility. The impact of rent harmonisation on HAG levels should be explored as a prerequisite for further development. SHR should be given further opportunity to comment on treatment of risk in future agreements. The quality of cost data should be sufficient to monitor risk effectively to alert partners to potential shortfalls earlier than was the case with Devanha - buy-in of project managers and contractors is an important element of this.
i. Monitoring performance and managing risk. Overall view that the systems were not adequate and monitoring involved "fixes" rather than the systematic monitoring system which had been sought.	Establishing a performance monitoring system should be a key task for the MD and her/his team at the outset of any programme. This needs the buy-in of the project managers and of the contractors. The development of web based technology to assist in project monitoring is appropriate for any large collaborative programme. However, it is essential that the collaborative partners including project managers and surveyors are involved in the specification and that the technology is fitted to the needs of the client. Commissioning a live version of any tool should be part of the IT project specification. The collaborative organisation should appoint a named project manager for the IT project, in Devanha's case this should be the MD.

Evaluation of Strategic Relationships: Local Authority Engagement

13.18 Devanha's performance in developing external relationships is strongly influenced by its governance structure. However, this issue is considered separately because of its importance, particularly in relation to any future collaborative development programme. Table 13.8 focuses on transferable lessons from Devanha's relationships with local authority partners.

Table 13.8 Transferable Lessons: Strategic Relationships with Local Authorities

Devanha Experience	Lessons
a. Devanha largely delivered its programme in three LA areas in approximately the proportions set out in its Offer of Grant. In the early part of the programme there was discontent over an underspend in one area and overall a feeling that the programme was more focused on some areas than others.	A collaborative approach can deliver a large programme over a number of local authority areas. However, the relationship and information flows with each individual LA need to be given attention. LAs need to be kept informed of decisions which affect their area and the collaborative organisation should try to respond to local authority concerns.
b. Devanha's external image was poor. It was difficult to communicate with and local authorities viewed Devanha as a 'badge' for RSL activity.	Appointment of a managing director with a dedicated team of development staff working for Devanha would overcome this problem. A collaborative organisation should have clear branding and engage with external partners on its own behalf.
c. Devanha was seen as a complicated organisation with which to deal.	Separation of responsibility as discussed in relation to governance, so that programme and project management were Devanha's responsibility, would address this issue.
d. The strategic relationship with LAs was poorly developed. LAs were not involved in determining Devanha's objectives.	The local authority is a key strategic partner and should be involved in setting objectives.
e. Communication with LAs over the implementation of the programme was weak, for example, when where there were changes impacting on numbers or tenure.	The collaborative organisation should be proactive in communication with the LAs.
f. Devanha was a useful organisation in bringing together larger sites or developing difficult sites. LAs were supportive of Devanha in this respect. LAs saw benefits in dealing with a collaborative organisation with a spread of housing management infrastructure in place.	A large programme allows a more strategic consideration of sites. Given the increased strategic role of LAs there are particular advantages in a group of RSLs coming together where they collectively have a wide spread of housing management resources and existing units across a local authority area.
g. LAs are concerned over a lack of capacity to build amongst organisations in their areas. They see a continuing role for a collective approach to development.	A collaborative organisation could have more collective capacity to build a reasonable programme in an area than individual RSLs. Due to its existing housing management capacity (mentioned in f.) it is in some respects at an advantage over a larger non-local RSL developer.
h. Devanha was seen to deliver a quality housing product.	Devanha's good reputation means there could be potential for offering development services to LAs provided that it was viewed as a viable partner.
i. In the medium term LAs are looking to a variety of providers to deliver affordable housing. All options including LA, RSL and private sector developers may have a role. However, Devanha members are not currently taking forward any proposals and other models are likely to become the norm in the area.	Having delivered a large programme of good quality housing, Devanha should be in a position to compete to deliver affordable housing in LAs in the north East. It would have to demonstrate improvements in governance and corporate identity to show that it was an easy and efficient partner.

Evaluation of Performance on Quality & Standards

13.19 Testing out opportunities for bulk procurement and promoting housing quality were key elements of the Devanha initiative. Within the constraints of costs, Devanha performed well in fostering and sharing good practice in achieving housing quality. On the other hand, it was largely unable to achieve benefits from bulk procurement. Table 13.9 summarises lessons in relation to quality and standards.

Table 13.9 Transferable Lessons: Quality and Standards

Devanha Experience	Lessons
a. Bulk procurement: Devanha was largely unable to gain benefits from bulk procurement, in part because schemes had been developed using traditional methods before being brought into Devanha and also because RSLs hung onto elements of individual specification. Relatively small efficiencies were gained in relation to components such as windows and, more significantly, timber kits in later schemes.	Schemes brought into a collaborative programme after being designed traditionally will have limited scope for economies from bulk procurement. A collaborative programme should strive to have a standard specification in place before considering schemes for inclusion. RSLs should submit schemes to a collaborative programme as early as possible and with as little pre-design as possible. The standard specification should balance local variation with standardisation. Member RSLs should have input into the continuing development of the standard specification so that it can be improved with reference to experience and needs of RSLs without unnecessary variation.
b. Environmental performance and sustainability: The programme met targets for SAP rating and eco-homes standards. A small number of houses in non-framework schemes failed to meet the SAP target. Devanha focused on sustainable building and output included local sourcing, timber frame, air source heat pumps and super-insulated houses.	Setting energy efficiency targets provided a useful focus for the programme. More generally, setting key themes for quality improvement can act as a driver of quality in a collaborative programme. Consideration should be given to how standards can be extended to non-framework schemes, as section 75 projects increase in importance in the national affordable housing programme.( In some areas, particularly energy and accessibility, improved building standards are resulting in more convergence between RSL and private developer standards, so this will become easier to achieve.)
c. Promoting Higher Quality Standards - The Performance Improvement Group explored quality but timing was too late to influence some schemes.	Early adoption of standard layouts, specifications, and components would enable a number of benefits, including economies of scale in procurement, and repetition leading to improved quality.
d. Innovation - The Performance Improvement Group explored the potential for innovation and use of modern methods of construction, e.g. off site construction including use of 'pods'.	Early investigation and adoption of innovative approaches may deliver benefit and provide a step-change in the way housing is delivered. The collaborative organisation should have a standard specification in place and update this through an equivalent of the Performance Improvement Group. The potential for sharing knowledge would be increased and risk reduced if all RSLs were involved, along with consultants, contractors and, where appropriate, suppliers. Solutions could be discussed and implemented jointly and address performance, sustainability, economies of scale and buildability.

Evaluation of Training & Employment

The Devanha Offer of Grant set targets in relation to training and employment. As one of a range of targets, training and employment was given relatively low priority. Training targets were knocked off course by the recession in the building industry although there were some achievements. Table 13.10 highlights lessons from Devanha in relation to training and employment.

Table 13.10 Transferable Lessons: Training and Employment

Devanha Experience	Lessons
a. Devanha set up objectives for training & employment with a key target of creating 150 modern apprenticeships. These targets were not achieved but selection criteria for framework contractors included their attitude to training and the contractors did take on apprentices during the recession.	Include a positive approach to training in selection criteria for framework contractors. Even in the deep recession experienced in the industry, committed contractors were able to retain apprenticeships with the help of a large works programme.
b. Devanha had targets for recruitment of people with difficulties which made it hard for them to find employment. Devanha did not retain the services of specialist consultants beyond the first year and had very little success in this area.	Achieving wider benefits in employment and training requires commitment and allocation of resources. To be successful, dedicated resources have to be allocated to provide the necessary support to help people into work.
c. Devanha set up targets for recruitment and training which could be monitored. These were not implemented by the non-framework contractors.	It is valuable to monitor recruitment and training activity. Given the increasing role of section 75 agreements in delivering affordable housing, efforts should be made to encourage and monitor employment and training by private developers.
d. There was very limited impact on subcontractor recruitment and training.	There was no mechanism for engaging with subcontractors in employment and recruitment and no incentive for them to address the issue. Extending best practice in training and recruitment may be explored as part of extending partnering within the supply chain.

Summary Conclusion

13.20 Devanha Phase 2 was designed as a bulk procurement initiative which aimed to produce cost and grant efficiencies while delivering high quality development. Through the use of a framework agreement and NEC form of contract, Devanha would demonstrate a change of style, culture and process.

13.21 Devanha did not meet its works cost targets as set out in the Offer of Grant. It did, however, meet the validated and adjusted cost targets taking into account agreed inflation and compensation events.

13.22 Devanha was hampered by tensions between the roles of the Board and individual RSLs. A Devanha-like collaborative organisation could be more successful with clear demarcation of responsibility; the collaborative organisation would require complete responsibility for strategic programme level issues and operational project level issues.

13.23 Devanha or a successor organisation should create a strong leadership position such as a managing director. Development staff should also work directly for the collaborative organisation.

13.24 Devanha and its partners did not fully implement the collaborative approach and only partially fulfilled the requirements of the NEC contract; cultural change was not fully embedded and some of the advantages of NEC were not fully realised. Addressing these issues would be a key role for a managing director.

13.25 Devanha was successful in building to a high quality but did not achieve significant bulk procurement savings. Further standardisation of specification, sharing of experience amongst partners and application of this early in the design process would provide opportunities to improve quality and explore bulk procurement efficiencies.

13.26 Devanha did not fully address issues of risk to individual RSLs. A more complete analysis of risk should be a prerequisite for collaboration. It would determine whether RSLs should become involved as partners. Risk analysis should be a major feature of choice of contract and procurement route.

13.27 With appropriate changes to its governance and leadership, Devanha could form a model for collaborating with local authorities to deliver affordable housing programmes in future.