Legislative Analyst's Office, December 15, 1998

State Should Employ "Best Practices" On Information Technology Projects

In 1994, reports from the Legislative Analyst's Office, the Bureau of State Audits, and the Governor's Task Force on Government Technology called for numerous reforms in the way the state acquired and implemented information technology projects. The Legislature responded by creating the Department of Information Technology and giving it broad responsibilities for oversight of information technology. Although improvements have been made in recent years, a number of the recommendations from the three 1994 reports, including recommendations regarding the use of information technology practices that are found in the private sector, have not been fully implemented.

In this report we examine 12 specific business practices frequently used by the private sector to develop, acquire, and implement information technology. The practices fall into four basic categoriesprocurement, project development, oversight, and contract management. We find that if state agencies use these "best practices," the risk of failure of an information technology project would likely be reduced.

We recommend that these best practices be used on state information technology projects, unless a project has unique characteristics that warrant exceptions. Specifically, we recommend that the Legislature:

• Require the Department of Information Technology to develop and issue policies which use best practices for new information technology projects, but allow for exceptions to be made with justification.
• Require the department to review existing information technology projects and require departments to enhance current projects, to the extent possible, by employing the appropriate best practices.
• Hold the department accountable for implementing the use of these 12 best practices through budgetary and legislative oversight hearings.

Introduction

The state's efforts to deploy large computer systems have resulted in a number of well-publicized costly failures which have not always brought about promised efficiencies. In 1994, three separate reportsfrom the Legislative Analyst's Office, the Bureau of State Audits, and the Governor's Task Force on Government Technologyidentified numerous problems with how the state procured and deployed information technology (IT). These reports also made recommendations on how to resolve these problems. Each of the reports identified shortcomings in state IT policies, including insufficient planning, poor procurement practices, weak contract terms, oversized projects, and lack of risk assessment and experienced staff.

As a result, the Legislature and the Governor enacted Chapter 508, Statutes of 1995 (SB 1, Alquist), which created the Department of Information Technology (DOIT). The department is responsible for developing policies and procedures by which information technology projects are to be conceived, evaluated, bid, and deployed.

Despite some improvements in the state's IT projects in recent years, problems remain and a number of the original recommendations contained in the three 1994 reports have yet to be implemented or have been implemented inconsistently. All three reports recommend changes in policies that incorporated a number of best practices used in industry in several areas relative to procurement and deployment of IT. Although DOIT has developed several policies based on these best practices, use and enforcement of these policies has been sporadic.

The Concept of Best Practices In Information Technology

"Best practices" is a term used to describe generally agreed upon processes and policies that should be undertaken when purchasing and deploying IT projects in order to decrease operational and financial risk. They are strategies derived from experienced industry experts who have, through trial and error, discovered methods for design, development, and operation of computer systems which increase the chances of success and decrease risk. Although a relatively new concept to the state, best practices have been used in industry for some time. A number of these best practices were identified in the three 1994 reports.

The most frequent reason given by state agencies for ignoring a particular best practice is that time will not permit its use. However, the state's experience has shown, especially with recent failures, that ignoring best practices contributes to the failure of projects. In order to reduce the risk of project failure, we believe that the state should consistently use best practices when implementing IT projects unless a strong case can be made for an exception.

Although most departments with a role in the approval process for IT projects acknowledge the need to improve their processes, there is little coordination, and continued bifurcation of responsibility and authority. In fact, policies which have been adopted as statewide guidelines are sometimes not followed by departments, resulting in increased risk to projects.

The state has responsibility for multiple separately maintained communications networks, employs hundreds of technical staff, possesses tens of thousands of personal computers, begins about 200 new IT projects each year, and spends approximately $2 billion annually on IT. The state's investment in IT will only continue to increase, thus the state should employ best practices to achieve a greater return on the taxpayers' investment.

In this report we outline 12 best practices that we believe should be evaluated for implementation on state IT projects, as shown in Figure 1. Each is described in more detail below. These best practices, which fall into four basic categories, are not the only ones employed by industry, but probably represent those which can provide a greater chance of project success if implemented.

Generally, we believe that these best practices should be employed on all large projects, unless a case can be made for an exception. The DOIT, as the state's primary oversight agency for IT, should develop and enforce policies that incorporate these 12 best practices, except in unique cases where they would not be appropriate. Current law provides a statutory framework to implement these practices and no changes in law are required.

The Procurement Process

The procurement process is often cited as the source of many difficulties in implementing IT projects. In many cases, departments have cited the length and the difficulty in successfully completing this process in a timely fashion as the reason for not employing the two best practices we discuss below. Although we believe that the current process is burdensome, current law allows departments to implement two industry best practices which may actually shorten the cycle and increase the chances for a successful procurement process.

Base Procurement on Best Value, Not Lowest Cost

Historically, the state has asked vendors to submit proposals for major IT procurements based on the costs for a particular technological solution prescribed by the individual department. As a result, vendors wage a bidding war for the project, with lowest cost usually winning.

Acceptance of the lowest cost bid is a generally accepted principal in most government procurement processes. The low cost bid process was established to reduce the likelihood that bids were being awarded based on favoritism or connections. Furthermore, it made it easy to determine who had the winning bidit is the one with the lowest cost.

However, there are significant drawbacks to this approach. First, the low cost bid process requires departments to propose technologically prescriptive solutions for a business problem the department is trying to resolve, so that all vendors' bids can be evaluated using the same criteria. Because a department specifies the particular technology, vendors are forced to provide a price on a particular solution which may not be technically feasible or may not be the best solution.

Second, the process has also resulted in vendors "low-balling" their bids but then coming back to the state with requests for additional money to cover costs not included in the bid. If the state does not approve the additional funds, the project may not perform as envisioned; if the funds are approved, the project turns out to be more expensive than anticipated.

Finally, what low cost bids fail to take into account is the best value of the procurement. "Best value" procurements enable a department to evaluate a bid based not solely on costs but also on other important considerations. These could include a vendor's technological solution, experience in a particular program area, the financial strength of a company, the experience of the vendor's staff, and other project components not previously included when bids were being considered. In such an approach, every vendor is made aware through the procurement documents of how the bids will be evaluated; the vendors are evaluated not only on the basis of cost but other important dimensions.

The Department of General Services (DGS)the state department responsible for authorizing procurementshas begun to embrace the concept of best value acquisitions, although state procurement law has not been changed to require these bidding procedures on all IT projects. Currently, only a small portion of IT procurements use this process because departments are not availing themselves of the alternative procurement process. We believe that the best value procurement methodology should be applied as the rule, not the exception.

Outline Business Problem Then
Allow Vendor to Propose Solutions

As indicated above, the state traditionally has prescribed the technical solution for an IT project, required vendors to provide a cost estimate to meet that solution, and accepted the lowest cost bid. Not only does the emphasis on lowest cost have important downsides, but so too does the focus on a prescribed technical solution.

Specifically, the more technologically prescriptive the department's procurement document, then the more confined the vendor's proposal, resulting in less opportunity for alternative solutions to be bid. In addition, the state generally must accept full responsibility if the project subsequently fails because it prescribed the technological solution. Finally, if the department specifies a technological solution that ultimately does not meet its needs a vendor can still fulfill the terms of the contract without the state necessarily obtaining a product that addresses the problem it was attempting to solve.

By contrast, if the state requires the vendor to propose the solution to a stated business problem, the risk related to offering the appropriate solution is predominantly shifted from the state to the vendor. When a vendor proposes its own solution, it is stating that a particular technology will solve the business need. As such, the state can require the vendor to take more responsibility for proposing its technical solution, should it fail.

Project Development

Develop Smaller Projects With Milestones

Historically, the state has attempted to deploy multiyear (and in some cases, multi-decade) long projects. Such lengthy projects pose significant dangers. For instance, it takes a significantly larger amount of money to put such a project back on track if a problem occurs due to the larger initial investment of time and money. Also, it tends to take longer to acknowledge fatal problems on a lengthy project because it is difficult to walk away from an investment of years and "sunk costs" of potentially millions of dollars. In some cases, it is better to simply cancel a troubled project rather than try to fix it.

On the other hand, smaller projects with predetermined milestoneswhere decisions are required to be made at each milestonemake difficult decisions a little easier. For example, it is easier for IT staff to tell executives after three months of problem solving on a year long project to modify or abandon the project than it would be after investing three years.

Smaller projects with established milestones reduce the risk of financial loss. The Department of Motor Vehicles (DMV) spent over $51 million over seven years on its database redevelopment project and the Health and Welfare Agency spent $111 million over seven years on the Statewide Automated Child Support System (SACSS) before these projects were canceled. Neither had the preestablished milestones and each was planned to take many years to complete. In contrast, the California Department of Corrections (CDC) spent $18 million over five years on the Correctional Management Information System (CMIS) project at the time it decided to cancel the project.

Additionally, long project life cycles make it difficult to respond to new business needs or technological changes. For instance, a decade-long project like the welfare automation system may need significant changes to reflect changes in policy and shifts in technology. A smaller project with more discrete components can incorporate such changes easier.

Prioritize Project Elements Up Front

Every project has three major components: (1) the budget, (2) the schedule, and (3) what the system will do, known as the "functionality." During the project life cycle, it is not unusual for problems to occur. In order to know how to solve a problem, the project manager needs to know which of the three components is the highest priority of the department, which is secondary, and which can be the most flexible.

For example, assume that a problem occurs which could result in a delay in deploying the project and the project manager knows that it is of paramount importance that the project be done by a certain date. The manager knows he or she must either increase the budget thereby dedicating more resources to solving the problem or decrease the functionality in order to meet the schedule. Alternatively, if the budget is the number one priority, the manager can delay the project or reduce its functionality.

Determining the priorities between the budget, schedule, and functionality must be made at the beginning of the project to guide the project manager throughout the project's life cycle. Without establishing these priorities, the project loses definition and may no longer be on schedule, on budget, or able to perform as it was intended.

Establish Measurable Objectives for the Project

Automation can bring efficiencies. However, a department has to establish measurable objectives for the project to avoid automating for the sake of automation. Quantifiable goals, such as establishing a target for reducing the amount of time or cost to administer the program, should be established. Without quantifiable goals and baseline data to use in assessing whether the goals have been obtained, progress and success cannot be measured. Broad goals such as "program efficiencies" must be quantified in order to measure progress. Lack of performance standards to gauge and monitor progress makes it virtually impossible to determine whether the project has accomplished its objective. Even if a project is initiated as a result of a federal government requirement, the state should establish quantifiable goals.

Some projects have measurable objectives that provide that the vendor is only paid based on meeting these goals. For example, the Franchise Tax Board entered into a contract to pay the vendor a percentage of the increased collections attributable to the automation project. Such arrangements make establishing the existing capabilities and future goals especially important in order to be able to determine whether the project increased collections or productivity.

Measurable objectives, combined with strong project management, will enable the evaluation of progress and increase the chances of success. Without every participant knowing what the quantifiable goals are, communication and ultimate success become more difficult.

Avoid Decisions Based Primarily on Funding Availability

Frequently, the federal government provides additional funds above its normal share for federally mandated programs (predominantly social services programs). These funds are often provided as an incentive for a state to implement a system or to achieve a federally determined goal, such as deploying a system by a certain date. However, a state's attempt to maximize federal funding can lead to conflicting priorities for a project. In fact, the rush to obtain the federal funding may contribute to ultimate failure if the funding priority conflicts with previously established project priorities or use of best practices. Although it is tempting to maximize the federal funding of information systems, decisions must be made based on project management guidelines to ultimately produce the most efficient and effective system.

As an example, in the early 1990s the state prematurely selected a welfare automation pilot project to be deployed as the statewide welfare system in order to receive enhanced funding. The state chose to deploy a particular system statewide since it was one of two welfare automation pilot projects being piloted at the time and was further along than the other. Little testing had been done to determine whether the system could meet the needs of all the counties. In fact, the majority of counties had chosen the other pilot project as being more closely aligned with their business practices, but since it was not completed at the time the decision needed to be made, it was not chosen for statewide implementation.

In 1995, the Bureau of State Audits evaluated the welfare automation project and determined that the chosen system could not meet the needs of many of the counties. As a result, statewide deployment was halted and planning had to begin anew.

Thus the decision to maximize federal funding resulted in the selection of a system that could not, in fact, meet the state's needs. Ultimately, this resulted in significant and costly delays in automating California's welfare program. Thus, what appeared initially as prudent fiscal management ended up lessening the success of the project. The lesson to be learned from this experience is not that the state should look askance at enhanced federal funds, but that it should not let the prospect of such funding drive its decisions.

Project Oversight

Require the Use of Project Management Methodology

A project management methodology is a blueprint of how the project will be administered. It includes many components which enable a project manager to administer and track progress on a projectessentially a collection of processes which have been tested and are employed to decrease the risk of operational failure and increased costs. Although some would consider the collection of these processes as common sense, many departments view them as a distraction from making progress on a project. Unfortunately, without rigorous project management, it is difficult to track expenditures and success of a project.

A project management methodology should include, among other things, development of a strategic plan, use of a cost accounting system, preparation of a valid cost-benefit analysis, consideration of viable alternatives, determination of how the proposed technology benefits would meet the department's business needs, establishment of a dispute resolution process, hiring of a project manager with project management experience, and employing a process to implement proposed changes. Without establishing a strong and effective management structure, the state risks losing control of the project.

The Health and Welfare Agency Data Center acknowledges the need for project management methodology and is employing one on the newest child support enforcement project. Following such a methodology will enable the data center to account for the total time spent on any task over 40 hours, as well as most expenses for the project.

Require Letter of Credit From Vendors on Larger Projects

Should a project fail, it is beneficial for the state to have a financial instrument from which it can recover some of its losses. Currently, vendors must provide a financial instrument to the state on larger projects. Historically, the state has requested a performance bond. In order to collect on a performance bond, the state must make a case to the issuer of the bond that the vendor did not meet the terms of the contract. The issuer then pays the bond, or hires another vendor to finish the work. Thus, the issuer of the bond may have a conflict in paying the bond to the state since the issuer loses this money.

A letter of credit can also be issued to protect the state's financial interests. A letter of credit is typically easier to collect than a performance bond because it does not require the state to go to a third party which has a vested interest in not releasing the money. The state collected $10 million through a letter of credit for the CMIS project at CDC when the state canceled the contract asserting that the vendor did not perform as required by the contact. The state was able to collect the money by simply withdrawing it from a bank.

The letter of credit may add cost to the project since it requires the vendor to make more capital available than does a performance bond. However, on larger projects, the state's risk is larger and a letter of credit increases its ability to recover potential losses.

Heed Advice of Quality Assurance Contractor or Explain Why Not

The DOIT has encouraged state departments to hire quality assurance contractors to help the departments identify and assess the significance of problems that occur as projects are implemented. These contractors also propose solutions to the identified problems. This secondary vendor assists the department in assessing the prime contractor's performance, thereby minimizing risk by identifying potential problems early in the project's life.

These quality assurance vendors are sometimes known as independent verification and validation (IV&V) vendors. These vendors use a prescribed process to assess the primary contractor's performance by reviewing planning documents, assessing the quality of the system design, evaluating the code being written, and a variety of other tasks. The IV&V vendor makes recommendations to the department on how to obtain a better quality product from the prime contractor.

In several cases, the department which hired the quality assurance contractor ignored the advice of this contractor and proceeded with the project against the vendor's advice. In the case of the SACSS project, ignoring the advice proved to be a contributing factor to the failure of the system. On the other hand, CDC followed the advice of its IV&V vendor on the CMIS project and canceled the contract when it became apparent that it would not succeed, thereby resulting in a relatively minimal loss of investment.

Departments which hire a quality assurance contractor should follow the advice of the contractor or document why the advice is not being followed. If the IV&V's recommendations are not going to be followed, the return on the investment of hiring the quality assurance contractor is negated. This in turn can mean that the project faces unnecessary risk.

Pay Vendor Only Upon Acceptance of Tested Project Deliverables

Historically, the state has paid vendors based on a contractually agreed upon schedule, which did not necessarily coincide with the delivery of a completed component of the project. The result is that vendors received payment whether or not progress had been made on the project. Thus, the state accepted all the financial risk by paying the vendor whether the vendor performed or not.

In order to protect the state's investment, vendors should be paid only upon acceptance of a deliverable, which the state verifies meets the terms of the contract.

Contract Management

Write Stronger Contracts to Better Protect the State

When writing IT contracts, the state has historically borrowed language from state contracts for commodities or services acquisitions. These contracts understandably did not contain provisional language addressing traditional IT processes such as invocation of liquidated damages or spelling out of dispute resolution processes. As a result, the state used inadequate contract terms which resulted in the state being in a compromised position when conflicts arose. The state's IT contract language needs to better set out responsibilities, liability, the dispute resolution process, and terms of payment in order to protect the state's interests.

In 1995, the contract for the child welfare services automation project had to be renegotiated to include functions the state thought it was buying but later determined were not included in the original contract. Additionally, while renegotiating the terms, the state included liquidated damages to better protect itself should the contractor fail to provide a deliverable as required in the schedule. Such terms generally had not previously been included in IT contracts.

The state should seek assistance from outside experts who have experience writing contracts for IT projects and require the use of this language in IT contracts.

Enforce the Terms of Contracts

In the past, departments have not enforced the terms of IT contracts for a variety of reasons. A frequently cited reason is that the department did not want to antagonize the vendor by assessing liquidated damages for failing to meet the agreed upon schedule if there was additional work the contractor was to perform. The result, though, is that the vendor is not held to the terms of the contract, thus rendering the liquidated damages' provision meaningless. For example, the California Student Aid Commission has stated that it failed to require the vendor to provide the contractually agreed upon system documentation on a project because it was fearful that the vendor would remove its staff from operating the system in order to write the documentation.

When a vendor knows it will not be held to the terms of the contract, the contract becomes meaningless. The contractor should not receive payment for services not delivered, or delivered outside the agreed upon terms. The state should send a clearer message to vendors that they will be held responsible to meet the terms of contracts.

Conclusion and Recommendations

There is no single reason why IT projects fail, but we believe that employing the 12 best practices outlined above on each IT project will increase the opportunity for success.

Thus, we recommend that the Legislature:

• Require DOIT to develop and issue policies which use best practices for new IT projects, but allow for exceptions to be made with justification.
• Require DOIT to review existing IT projects and require departments to enhance current projects, to the extent possible, by employing the appropriate best practices.
• Hold DOIT accountable for implementing the use of best practices through budgetary and legislative oversight hearings.

Return to LAO Home Page