This chapter presents a review of recent studies relating to the performance of PFI/PPP projects in their operational phase in the United Kingdom. The vast majority of projects reported on are from mainland Great Britain simply because there were very few projects in Northern Ireland at the time these reports were written. This is one of the purposes for this research i.e. to look at the performance of PFI/PPP projects in NI, now that there are a number of schemes in their operational phase in this part of the UK. The studies and reports reviewed in this chapter have been commissioned by both government and other bodies and go back as far as 2003 up to the present. This review thus addresses the second key objective of this research which is to identify and analyse the performance of operational PFI/PPP contracts in GB.
REVIEW OF STUDIES
Get your grade
or your money back
using our Essay Writing Service!
This section presents a review of the summary of the findings of recent studies relating to the performance of PFI/PPP projects in their operational phase in the United Kingdom.
A subsequent section gives analysis and discussion in the overall context of the review.
4.2.1 National Audit Office (2003), The Operational Performance of PFI/PPP Prisons
This report was ordered by the House of Commons and carried out by the National Audit Office in 2003. The study looked at 21 prisons in England and Wales, 7 of which were operational PFI prisons, two privately managed and twelve were public sector prisons. The evidence for the report was gathered by visiting the prisons, surveying prison officers and prisoners and conducting semi-structured interviews.
The operational performance of PFI prisons compared to other contract prisons has been mixed.
Most PFI prisons have had problems when they first opened but have, in the main, improved.
Amending PFI contracts to accommodate changing priorities is difficult.
The private sector has brought benefits to the prison service.
Competition has been important for improving management and conditions for prisoners.
Innovation has been introduced by using PFI i.e. recruitment and deployment of staff and new technology.
The level of payment deductions is not necessarily a true reflection of performance.
Prisoners in PFI prisons feel they are shown greater respect and treated better than in public prisons.
This report is completely concerned with the operational performance of PFI prisons compared to other prisons in terms of recruitment and deployment of staff and the use of new technology.
A rigorous approach has been taken to select a valid sample across the range of prisons as well as various research instruments used to gather the research data. While undoubtedly the report is of considerable value to the commissioning party and those within the prison sector it is of limited value in terms of this research project because of its scope and terms of reference.
Having said that, however, the report does identify main issues which influence the performance of operational PFI projects and these can be taken forward and used to help formulated research design in this study.
4.2.2 4ps (2005), Review of Operational PFI/PPP and PPP projects
This report was produced by 4ps on behalf of Local Authorities in England. The team reviewed 30 operational local government PFI/PPP schemes and interviewed key stakeholders involved in their management and covered all sectors of local government.
Partnerships - although there were some natural tensions between parties in the delivery of service, the majority of managers described their partnership relationship as very good.
The Output Specification - 92% of those interviewed claimed a good working knowledge of the output specification with 82% that all documentation was complete and available for the project.
Service delivery - only 7% of those interviewed said that services were being delivered in accordance with the output specification all of the time, 71% most of the time and 22% some of the time.
Service changes - 58% of projects had their output specification reviewed of changed since award of the contract, while 66% anticipated changes in the future.
Response times - local authorities were generally satisfied with the way service failure had been responded to with 73% reporting that they had been met all of the time.
Always on Time
Marked to Standard
The Payment Mechanism - 80% of those interviewed had read and understood the payment mechanism of the contract. 77% reported that deductions are being made in accordance with the contract arrangements, 82% felt that it was working, however 37% thought it seen as a means of punishing the contractor. 79% believed the level of payment made in each period reflected the benefit delivered.
Performance Monitoring - 85% of interviewees reported that monitoring reports are prepared by the service provider and assessed by the local authority for each payment period, 84% agree that the reports received are in accordance with the contract and provide sufficient analysis for payment to be made.
The Help Desk - 89% of respondents were aware of the availability of a help desk.
The Contract - 79% of those interviewed had personally read and understood the full contract documentation, with 61% having to refer to the contract on an issue. Only 32% reported that their organisation had negotiated changes since the financial close of the contract and 64% thought that the contract provided sufficient flexibility in the provision of services.
This report studied 30 out of 158 operational local authority PFI schemes in England which were delivering services across a wide range of sectors. Key stakeholders were involved in the data collection process and a structured and well defined process used to give a consistent approach.
The findings of this objective and structured review give a clear indication of what is being achieved using PFI/PPP but also identifies possible issues still to be addressed in certain sectors. From this point of view this report gives a very useful snapshot which could be used to build on and again the main issues described to evaluate the performance of these projects can be used and taken forward in this study.
4.2.3 Ministry of Defence Private Finance Unit (2005), Review of MoD PFI/PPP Projects in Construction and Operation
The Ministry of Defence initiated this report in 2005 to assess how PFI/PPP had performed in practice in the early years of operation in the MoD. This review included 29 PFI/PPP operational projects and excluded those with a capital value less than £20M. Information was gathered from contract management/monitoring teams by use of surveys and structured interviews.
Performance - 97% of the project teams surveyed reported that the performance of their PFI/PPP project was satisfactory or better. 73% of project teams rated the performance of their PFI/PPP project as good or very good.
Flexibility - the review identified that 85% of projects reported that their contracts were suitably flexible to accommodate change and had effective change management mechanisms. Some feedback indicated that changes to the contract required extensive effort and were costly.
Deductions - 41% reported imposing no performance deductions and of the remainder, 76% of these cases amounted to no more than 2% of the unitary charge.
Payment Mechanism - 90% of contract managers felt that the performance mechanism supports the effective performance of the contract, although some thought it was unnecessarily complicated and the effort required was disproportionate to the contractor's incentive.
Relationships - 90% of those surveyed considered that relationship between parties as good or very good.
Managers thought that practices could be improved if the right resources were available, business continuity was improved and if knowledge was shared from one project to another.
This report is narrow in its scope in that it is restricted to MOD projects. Limited analysis can be made as there is a lack of detail in relation to the projects reviewed. This is understandable given the probable sensitive nature of the projects involved.
However, as with other reports it identifies main areas of interest which the researcher can take forward to help formulate research questions.
4.2.4 Cambridge Economic Policy Associates (2005), Public Private Partnerships in Scotland: Evaluation of Performance
This report was commissioned by the Scottish Executive to further its knowledge and understanding of the performance of Public Private Partnerships in Scotland. CEPA received survey questionnaires from 41 operational PPP projects and interviewed staff involved with 25 operational PPP projects in Scotland.
Performance - most authorities were happy with the standard of service received. 91% rated the service either good or very good with the same figure reporting that the service meets or exceeds expectations. Hard facilities management service standards (i.e. building repairs) were considered to be performing less well with 58% rating the service good or very good.
This Essay is
a Student's Work
This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.Examples of our work
Monitoring - 61% of authorities thought that performance monitoring mechanisms were working well or better, 36% satisfactory and 3% poor. Only 45% of authorities thought that the performance indicators were wholly appropriate for measuring contractor performance
Flexibility - contracts were seen by the public sector managers generally as less flexible than non-PPP contracts (34% less flexible, 49% no difference and 6% more flexible). 42% of private sector respondents considered the contract to be more flexible.
Payment mechanism - 88% of projects had made deductions from the payment mechanism. 41% of authorities said that there had been occasions where they could have imposed penalties under the contract but because they were of little effect, amounts were small they decided not to in the interest of partnership.
Change mechanism - all contracts had a facility within them to deal with changes and this had been used in 50% of projects. Informal changes had been made in 28% of contracts. Comments were made on the slowness of making any changes and concerns raised about the value for money of those changes.
Relationships - 85% of authorities and 92% of contractors said their relationship was good or very good.
Although this report is restricted to schemes in Scotland it covers a wider range of subject areas than the other reports reviewed including procurement, design and construction and value for money. These areas are not the subject of this dissertation therefore its usefulness to the current research is limited however one section of the report deals with operational performance.
There is a structured and rigorous approach to the research with the use of key questions and evidence to support the findings made. However, the high level nature of the study does not provide definitive answers to all of the research questions posed.
4.2.5 PartnershipsUK (2006), Operational PFI/PPP Projects
This report was commissioned by HM Treasury in 2005 to give a review of the performance of PFI/PPP projects during their operational phase from a wide range of sectors in the UK.
The methodology was firstly to review a large number of reports and reviews associated with this subject and to identify any common themes or factors that contributed to the performance of projects. Secondly, the report gathered primary data from 105 operational projects by way of questionnaire. Thirdly, a selection of private and public sector managers were interviewed to give their perceptions on the performance of the PFI/PPP projects for which they were responsible. Taken together this gives a good overview of the perceptions of operational managers.
Overall Performance - The main finding in terms of overall performance was that 96% of public sector respondents rated the performance of the service provider as satisfactory or more (26% very good, 40% good, 30% satisfactory, 2% poor and 2% very poor).
Relationships - around 97% of public sector contract managers reported that relationships between them and the private sector service provider as either very good, good or satisfactory (40% very good, 32% good, 25% satisfactory and 3% poor).
User satisfaction - 79% of users were either always or almost always satisfied with the service they were receiving (14% always, 65%almost always, 20% about half the time and 1% almost never).
Dispute resolution - 82% of respondents reported that operational problems in projects were resolved within the time allowed under the contract either always or almost always.
Payment mechanism - 78% of contract managers agree that the payment mechanism supports the effective contract management of their project. 70% of projects have incurred performance deductions with no loss of service delivery or partnering relationship.
Management - 73% of projects have a dedicated public sector management team and evidence suggests this helps to improve service levels.
Change mechanism - Although most of the contracts reviewed have been changed since they became operational 83% of contract managers confirmed that the contract specification accurately reflects the service required.
Benchmarking - this was considered a significant issue although nearly half of all projects reviewed did not have benchmarking arrangements.
Support - a large number of those questioned thought that less support was given to the project when it became operational than it received during the procurement phase.
Training - only a limited number of managers had received formal contract management training or specific PFI/PPP training.
This report is wide ranging in its scope and comprehensive in the issues reviewed. The research strategy for the report was hypothesis driven, which were developed by various parties and experienced personnel.
The extent and high number of respondents adds value to the findings of the research however most of the projects considered were located in GB.
This report has contributed to knowledge in an important way in that it has helped shaped policy, added to the development of good practice and other research done since then has used it as a starting point.
The PUK report will help form the basis of this dissertation in terms of providing a benchmark for comparative analysis in relation to the performance of PFI/PPP projects in Northern Ireland.
4.2.6 KPMG (2007), Effectiveness of Operational Contracts in PFI/PPP
This survey was jointly sponsored by independent consultants KPMG and the Business Services Association. The survey is based on the responses of 93 contract managers in the private sector, across a range of PFI services and used a semi-structured on-line questionnaire.
Relationships - 24% of respondents consider day-to-day working relationships to be good, 56% very good and 5% poor. Regular communication between parties was identified as the key to success.
Operational performance - 59% reported overall performance as very good, 36% good and no one thought it was less than satisfactory.
Deductions - 39% of respondents had received deductions between 1 and 5 times, 34% of contracts have not been subject to any performance of availability deductions and in 53% of cases the public sector has chosen to waive the right to impose deductions.
Innovation - 60% of respondents reported that new ways of working had led to contract performance improvements. Of these, 11% saw a decrease in service delivery cost, 70% saw no change in the charge for the services they provided and in 19% of cases it lead to increases in service delivery costs.
Benchmarking - 87% of respondents indicated that they had not undertaken a benchmarking exercise for their PFI contract. Of those that carried out a benchmarking exercise, more prices rose than fell. In 75% of cases the overall price of the services rose by up to 10%.
Payment mechanism - around 50% of respondents agreed that the payment mechanism supported effective management of the contract, although 25% found it difficult to use and 5% very difficult. 90% of payment mechanism did not provide incentives for over performance.
Dispute mechanism - in 80% of cases this has not been used at all. Of these, 64% reported a very good relationship and 67% a very good overall performance. 15% of contracts have used the mechanism between one and three times.
This report focused on private sector providers of PFI services. It covered a wide range of question areas and sectors within the industry. It adds to knowledge somewhat in that it acts as a set reference data for the industry in terms of private sector perceptions where other reports tended to focus on public sector projects.
The detail of how the research methodology was arrived at and the approach taken by the authors to reach their findings is not immediately obvious from the report however again the main issues around the performance of operational PFI/PPP projects, as outlined above, are included. As noted in the report itself the findings may be useful to build on and add to year-on-year to explore trends in the industry.
4.2.7 Ipsos MORI (2008), Investigating the performance of operational PFI/PPP contracts
PUK, on behalf of HM Treasury, commissioned Ipsos MORI to conduct this research in 2008. An online survey of PFI contract managers across England was conducted with a total of 151 surveys received.
Performance - 73% of contract managers rated this as good or very good and 96% of managers rate performance as satisfactory or better.
Service Delivery - reported as always or almost always achieved by 94% of contract managers.
User satisfaction - of those that carried out a user satisfaction assessment, 92% report that services were being delivered to an acceptable standard.
Resources - 74% of contract managers believed they had sufficient resources to manage the delivery of the contract.
Relationships - 83% of managers rated this as good or very good, 98% satisfactory or better.
Formal handover - 54% of managers reported a formal handover took place from procurement to operational teams, and 41% of these believed that it was effective in helping them to prepare for contract management.
Performance deductions - 70% of respondents say the public sector has never chosen to waive its right to impose performance deductions.
Value testing - 26% reported that value-testing never has to take place under the contract and 64% that the benchmarking or market-testing processes are clearly explained in the contract.
Payment mechanism - 92% of contract managers understand the payment mechanism for the project and 42% find it difficult to use.
Support and advice - 46% of managers would benefit from further support, advice or guidance.
This report is similar in content and format to the earlier PUK (2006) report mentioned above. It is, in fact, an update of that previous research and is as comprehensive and rigorous as the previous research. It adds value in the fact that knowledge is updated in terms of the performance of schemes in general although not necessarily revisiting the same schemes. In fact all of the IT projects included in the previous research are omitted in this report as PFI is no longer used for IT projects. Also because PFI is a devolved matter projects previously included in Scotland, Wales and Northern Ireland are left out. Therefore, caution should be used in interpreting trends between the two reports however the contribution to knowledge is that there has been a marked improvement since the previous report in a number of respects.
This Ipsos MORI (2008) report will help form the basis of this dissertation in terms of providing a benchmark for comparative analysis in relation to the performance of PFI/PPP projects in Northern Ireland.
4.3 ANALYSIS AND DISCUSSION
The reports reviewed in the previous section covered a wide range of projects, sectors and geographical areas associated with the performance of operational PFI contracts. This section seeks to bring the content of these reports together, summarising the key points and giving an analysis and discussion of the findings in the context of their broad headings.
It is evident from the above reports that the findings in this category were overall very high and ranged from 97% reporting satisfactory or better (MoDPFU (2005)) to 91%
rating the service either good or very good (CEPA (2005)).
There may be a number of reasons for such a generally high figure. These responses are from public sector contract managers in the main. They are perceptions of how they perceive the contract to be operating and may be biased from the point of view that they may not wish to criticise their own project for which they are responsible. The effect of this is difficult to quantify, however, if the same premise is used as is proposed in this research then it may be possible to have a direct comparison.
Also, a number of the reports were carried out by the contract managers 'employers' rather than an independent body, therefore suggesting the possibility of biased findings (MoDPFU, (2005)).
It may also be the case that the measurement of performance differs slightly between reports so that strictly speaking a number of them cannot be compared directly. Although the various contracts may follow the same generic format broadly speaking, it is safe to say that each contract is individual in terms of its performance measurement mechanism in relation to their output specifications; however the data gained from these surveys indicate the various contract managers' perception of performance for their own schemes.
4.3.2 The Output Specification
A well-drafted output specification is seen as fundamental to the successful delivery of long-term services (4ps (2005)), in terms of clarity, understanding, flexibility, etc. Although a high percentage of 92% recorded that they had read and understood the output specification only 82% suggested that the documents were complete and available for use. It is difficult to understand how this disparity can exist however there may be an issue on the completeness and availability of documents to the contract manager because of ongoing reviews, changes being negotiated etc. since the initial document completion.
The 4ps (2005) report indicated that only 7% of respondents considered the services being delivered were in accordance with the output specification all of the time, whereas 19% considered that services were being delivered to expectations all of the time. This would suggest that either the output specification is more stringent in its requirements or the expectations of contract managers have diminished over time. Either way this is a very low figure and may suggest that the output specification has been set to a very high standard. This would be supported by the fact that 58% of those surveyed suggested that the output specification has been reviewed and a higher number 66% expect further changes to be made in the future (4ps (2005)).
4.3.3 The Payment Mechanism
As with the output specification, this is also seen as a very important aspect of the PFI arrangement as it should be linked to the outcomes and outputs for the project set out in the output specification. The majority of those surveyed had read and understood this document (ranging from 80-92%). This suggests that a significant number of contract users had not read and understood the payment mechanism and may be explained in a similar way to the above, however suggestion was made in some of the reports that this document was seen as a legal and financial document therefore inaccessible to users, who were included in the response rates.
There is a wide range of responses in relation to deductions being made i.e. 39% (KPMG (2007)) to 88% (CEPA (2005)). Although the majority of reports recorded that the payment mechanism was working well with corresponding deductions being made there is evidence in the more recent KPMG (2007) report that this is not quite the case. There is evidence to suggest that public sector managers were choosing to waive their right to impose deductions to gain other changes without going through the costly and time consuming change mechanism.
On the other hand, it is suggested that service providers are over delivering to minimise the impact of deductions in an effort to maintain the long-term partnership. However, this low deduction rate could suggest either high quality delivery, questionable contract management or a poorly designed payment mechanism.
4.3.4. Performance Monitoring
There is a variety of formal and informal ways used and tools implemented to monitor the performance of schemes such as, monthly management reports and meetings, face to face discussions, KPI's, audits, etc. In some cases (85% of those surveyed by 4ps (2005)) performance monitoring reports are prepared by the service provider and assessed by the public sector for payment.
PUK (2006) report that for the most part performance measurement systems are used to incentivise the service providers to deliver the standard requirements set out in the contracts, although some evidence suggests that some see it as a way of punishing the contractor (4ps (2005)).
As already mentioned in paragraph 4.3.1. there are generally high levels of satisfaction reported in terms of perceived performance; however there does not appear to be a uniform way of monitoring or measuring that performance. This may be due to the fact that the reports reviewed cover a wide range of sectors and types of PFI/PPP project as well as the fact that some of schemes had not been in their operational phase long enough to establish what perceived good performance was. In general terms, performance of the scheme i.e. the service being delivered, was measured against the requirements of the output specification with a reported high level of satisfaction.
Due to the long-term partnership ethos of PFI/PPP schemes it is important that good relationships are in place between the main parties. In the main, reported perceptions of relationships among those questioned was very good with scores ranging between 80% (KPMG (2007)) to 98% (Ipsos MORI (2008)) considering them good or very good. Such a high level would suggest that because the parties were in the partnership together they made an effort to ensure that the relationship worked. There is evidence to suggest that difficulties can arise in relation to interpretation of service delivery, KPI's etc. but in general terms these are sorted out within the good partnering ethos that exists. The fact that 83% of contract managers said that the formal dispute mechanism has never been used since the project became operational demonstrates this point (Ipsos MORI (2008)).
Also, the research showed that better relationships were reported where staff were in post for a longer period of time and that the frequent turnover of staff tended to disrupt relationships (PUK (2006)). This implies that the maintenance of good relations over time is beneficial to the project, however, it could be argued that if a relationship was deteriorating it may be beneficial to move staff out of the situation where required. It is important that the correct staff are in the positions that influence the long-term relationship for the overall good of the scheme.
Only two of the more recent reports mentioned user satisfaction and gave a measurement of it, namely PartnershipsUK (2006) and IPSOS Mori (2008). On the face of it there is generally quite a high level of user satisfaction, of those questioned. However, looking a little deeper into the detail only around 40-50% of those questioned used formal methods of measuring user satisfaction. There may be a number of reasons for this such as, there is such a wide variety of methods of assessment, inconsistency of measurement between various types of project and client, difficulty in establishing a common satisfaction standard, etc.
Having said this however, it is important to assess the satisfaction of the end user against what was initially required and agreed by the client before the contract was signed. It may be the case that each individual project will have its own user satisfaction standard, however if this is agreed beforehand and used formally to measure against once the project becomes operational then it is a viable measure of user satisfaction for that project.
This chapter has looked at studies and reports, commissioned by Government and various bodies, relating to the performance of PFI/PPP projects in the UK. All of the issues identified, analysed and discussed in this chapter have reoccurred from the previous chapter relating to journal papers and academic literature. These issues will be taken forward into the next chapter i.e. the research methodology, in the form of research questions which will help to formulate areas and questions to be incorporated into the primary research instrument questionnaire. This chapter thus addresses the second key objective of this research which is to identify and analyse the performance of operational PFI/PPP contracts in GB.