Analysis of the 2004-05 Budget Bill Legislative Analyst's Office February 2004

State Information Technology Projects

In March 2003, the Department of Finance began a statewide assessment of 117 state information technology (IT) projects. The results of the assessment indicate that departments are having problems (1) developing their IT projects and (2) adhering to state policies. Our analysis examines the assessment results, identifies some possible deficiencies in the state's IT processes, and makes recommendations on how the Legislature can address the deficiencies.

Every year, the budget includes funding for many different automation projects. For example, this year the Governor's budget requests funding to install computer equipment in a new state building. Another proposal requests funding for the ongoing development of the federally required child support enforcement system. Even though the state funds many IT projects every year, the Legislature has never had statewide information available on how well the state is actually managing its IT projects. For example, information is not reported on how many IT projects are: (1) under development, (2) meeting estimated budgets and schedules, (3) properly managed, and (4) achieving ongoing benefits. The lack of reported information has limited the Legislature's oversight of the state's IT projects.

Department of Finance Conducts First Independent Assessment of All Projects

Prior to March 2003, the Department of Finance's (DOF) Technology Oversight and Security Unit had conducted a number of reviews and assessments of individual IT projects. Most of these reviews were conducted on a project-by-project basis. Individual project reviews provide information on a specific project. These reviews, however, do not provide statewide information on how well the state is actually developing, overseeing, and completing all of its IT projects. In an effort to better understand the state's overall status of project development and oversight, DOF began in March 2003 the first statewide assessment of active IT projects. In June 2003, the results were reported to the state's Chief Information Officer. The primary purpose of the statewide assessment was to establish a baseline of all active projects. Once a baseline is established, the state can measure its performance in project development and oversight over time.

Scope of Assessment. The assessment examined 264 projects initially thought to be in development. Of these projects, DOF found that 147 projects had been completed. The remaining 117 projects were then assessed in two areas: (1) project risk and oversight and (2) adherence to the state's policies on reporting project changes.

Project Risk and Oversight. In February 2003, DOF issued a new state oversight policy, which describes the project management and oversight activities that must occur on state IT projects. The policy requires that each project meeting certain thresholds be assigned a "risk rating" based on a number of quantifiable factors—such as project cost, experience of the project manager, and the type of project. The risk ratings fall into three categories—high, medium, and low. Figure 1 provides examples of typical qualities that projects may have for each of these risk categories.

The risk category determines the level of oversight for the project. High-risk projects have the highest level of oversight. For these projects, departments must report monthly on the status of the project to DOF and their agency (for instance, the Department of Parks and Recreation reports to the Resources Agency). In addition, high-risk projects must have independent contractors monitoring, evaluating, and reporting on the project status. Medium-risk projects require the agency to review and monitor the project, and low-risk projects are overseen by the department. The DOF's policy also requires specific project and risk management activities for each project. The risk rating determines the extent of these activities. High-risk projects have the most activities and low-risk projects the least amount of activities.

Reporting on Changes in Project Budget, Schedule, and Scope. For all projects with a risk rating, state policy requires departments to report on changes of 10 percent or more in the project's budget, schedule, or scope and receive DOF's approval prior to making the change. Project changes must be reported in a Special Project Report that describes the change, the reason for the change, and the impact the change has to the overall project.

Reporting on Completed Projects. According to state policy, departments are required to prepare a post implementation evaluation report (PIER) for each completed state IT project. The PIER must measure the benefits and costs of the implemented IT system and document the projected maintenance and operation costs over the life of the system. The PIER is intended to compare what was expected to be achieved versus what was actually achieved. State policy requires a PIER to be completed within 18 months of project implementation.

LAO's Review of Assessment Information

In our review of DOF's baseline assessment, we found that most current state IT projects are (1) considered medium to low risk and (2) experiencing some type of project change. In addition, our review indicates that many departments have not submitted PIERs for their completed projects. We discuss these findings in more detail below.

Assessment Reports Most State IT Projects Are Low to Medium Risk. As summarized in Figure 2, the assessment found that two-thirds of state IT projects are low to medium risk. This means that most state IT projects (1) do not require oversight by DOF and contractors and (2) do not involve highly complex hardware and software systems.

Project Oversight Lacking in Many Cases. The assessment found that half of the projects did not have adequate oversight. Of the 38 high-risk projects, 15 projects did not have adequate oversight at the time of the assessment. Since the assessment was completed, however, ten of these projects have implemented the required oversight. The five high-risk projects that still do not have the required oversight are:

• Department of Consumer Affairs' Electronic Transmission Project.
• California Department of Education's California School Information Services Program Data Integration Project.
• Department of Fish and Game's Automated License Data System.
• Department of Justice's California Palm Print System.
• Department of Water Resources' Business 2000—Phase 2B Project.

Most IT Projects Were Experiencing Project Changes. As summarized in Figure 2, most state IT projects (about three-quarters), regardless of risk rating, were experiencing project changes of 10 percent or more in budget, schedule, or scope. The assessment does not provide detailed information on the extent of the changes that are occurring on these projects. In other words, the extent to which changes are slightly or significantly over the threshold are unknown. Given the complexities of state IT projects, it is not surprising that most departments are experiencing some kind of changes on their IT projects. The key question is in what cases do these changes raise significant issues that must be immediately addressed. The assessment indicates that almost all high-risk projects are having difficulties in reporting project changes as required by the administration. Of the high-risk projects over $10 million, only four were reporting project changes on a timely basis (see Figure 3).

Many Departments Are Not Reporting on Completed Projects in a Timely Manner. At the time the assessment was conducted, DOF found that 147 projects were completed but many PIERs had not been submitted to DOF and the Legislature. (The DOF states that 85 of the 147 PIERs have now been submitted.)

Assessment Findings Indicate Deficiencies In the State's IT Processes

The DOF's assessment report establishes a good baseline to measure the state's progress in implementing its IT projects. In addition, it appears that the assessment has encouraged some departments to prepare PIERs for their completed projects and implement oversight for their high-risk projects.

The results, however, also indicate state IT projects are encountering serious problems:

• Most state IT projects are experiencing changes in scope, costs, or schedule.
• Departments have not adhered to state policies requiring the reporting of these problems.

These problems seem to occur regardless of whether the projects are small or large. 

The DOF assessment only establishes the current baseline. It does not provide the answers to important questions, such as:

• Are state IT projects being properly managed and overseen by their departments?
• What can the state do to ensure projects stay on schedule and within budget?
• Why are departments experiencing problems adhering to state policies on IT projects?

With the information available, we are not able to answer these questions on a statewide basis. Below, we do, however, discuss several possible factors contributing to the problems.

Departments May Not Be Properly Managing IT Projects. One of the simplest explanations for the widespread changes in project budgets and schedules is that departments may not be properly managing their IT projects. For an IT project to be successful, the project must (1) have strong commitment and leadership by the department's executive team, (2) use good project management practices, (3) have measurable objectives, (4) have strong contract administration and management, and (5) successfully manage change and risk. Since DOF's assessment did not examine how well departments were managing their projects, it is unclear how much of the project changes can be attributed to poor project management. The DOF's ongoing oversight system is designed to better identify mismanaged projects in the future. This oversight system needs to be integrated, however, with a permanent IT governance structure (a proposal for such a governance structure is currently pending before the Legislature in SB 403 [Florez]).

Departments May Prepare Poor Estimates for Project Budgets and Schedules. Another reason for the widespread problems in meeting project budgets and schedules could be that departments prepare unrealistic estimates of the time and money it will take to accomplish the project. With the current tight budget situation, departments may be "low balling" project costs. In other cases, departments may be using poor estimating techniques. Both situations would result in departments preparing estimates not based on the true costs of what it will take to address the identified problem. The state does not require the use of a standard estimating technique for project budgets and schedules. Since there is no state standard for project estimating, initial project budgets and schedules could be inconsistent and inaccurate. Problems in the estimates may become apparent only after the project is underway.

Review Process May Discourage Compliance. The assessment found that most departments are not following the requirements to report project changes and receive DOF's approval before making the project change. In some circumstances, though, DOF`s management of its IT workload may be discouraging departments from complying with state policy. For example, during DOF's busiest budget preparation times, projects that are experiencing changes but do not require a budget action are given lower priority in DOF's reviews. As DOF is not required to complete its reviews within a specific timeframe, departments can wait several months before DOF approves the project change. To comply with the state policy in this example, a department must either stop or slow down the project. When DOF eventually approves the change, the project could again have a budget or schedule problem since it waited for DOF to complete its review.

In other cases, the 10 percent threshold for reporting changes to DOF may discourage compliance. For some projects, a 10 percent level may not represent a significant project change. For example, for a $300,000 project, a $30,000 change could result from purchasing the most current hardware or software for the new system. In such a case, the department may be able to redirect existing funds to pay for the additional hardware and software costs. Yet, state policy requires DOF to review and approve the request before the department is able to make the purchase. While ignoring state policy should not be considered acceptable behavior, departments may be ignoring reporting requirements in some cases for practical and reasonable reasons such as keeping their projects from being further delayed.

Actions the Legislature Can Take To Address Causes of Project Deficiencies

To address the findings of the DOF assessment, we recommend the Legislature take two actions described below.

Require Departments to Report on Assessment Findings. First, we recommend that the Legislature require departments with high-risk projects over $10 million (see Figure 3) to report at budget hearings on the assessment's findings regarding their projects. Departments should be able to explain why delays and other deficiencies went unreported, as well as what actions have been taken to correct identified problems.

Direct DOF to Review State's IT Policies. Some of the problems found in the assessment may be indicative of deficiencies in the state's policies and review procedures. To address these problems, we recommend that DOF reexamine its policies on reporting changes. Now that the state's project risk rating system is in place, DOF should update its reporting policies to reflect those variations in risk. Specifically, DOF should (1) increase the threshold for reporting project changes on low- and medium- risk projects and (2) allow some changes to occur without prior DOF approval. We also recommend that DOF examine the need for statewide standards for estimating project budgets and schedules. Statewide standards could improve the ability of departments to accurately reflect realistic budgets and schedules in their initial proposals.