Legislative Analyst's Office Analysis of the 2001-02 Budget Bill

Department of Information Technology (0505)

The Department of Information Technology (DOIT) is responsible for ensuring that appropriate plans, policies, and procedures are in place to guarantee successful implementation of state information technology (IT) projects. Chapter 508, Statutes of 1995 (SB 1, Alquist) created DOIT to address state problems in successfully implementing IT projects. This legislation included a sunset date of July 1, 2000. The Legislature enacted Chapter 873, Statutes of 1999 (AB 1686, Dutra) to extend DOIT's sunset to July 1, 2002.

The budget proposes $11.8 million ($11.1 million from the General Fund and $750,000 from reimbursements) for support of the department's operations in 2001-02, an increase of $483,000, or 4 percent, above estimated current-year expenditures. The budget proposes 76 personnel-years for the department in the budget year.

Key Program Information Not Currently Available to the Legislature

We recommend that the Legislature not take action on the Department of Information Technology's (DOIT) budget until (1) DOIT submits required supplemental reports and (2) the Legislature has an opportunity to receive and review the outcomes of an audit of DOIT operations performed by the Bureau of State Audits at the request of the Joint Legislative Audit Committee.

The DOIT Has Failed to Meet Legislative Directives in the Past. Over the past four years, the Legislature has directed DOIT, either through budget or supplemental report language, to issue policies in a broad array of areas. These policy directives were to include project management training, procurement alternatives, project sizing, project delegations, maintenance and operations, system and information security, operational recovery, project oversight, feasibility study reviews (FSR), and procurement. With the exception of FSR and operational recovery, DOIT has not issued any new policies as directed by the Legislature.

Figure 1 summarizes the Legislature's requests to DOIT for seven supplemental reports in 2000-01. At the time this analysis was prepared, DOIT had submitted two reports on oversight policy. As regards the report on the oversight of county-based IT projects, we believe that the department has failed to adequately respond to this requirement as discussed below. As regards the remaining five reports, the department indicates that it is in the process of developing three reports that would address (1) policies for a streamlined FSR process and revised state acquisition practices and (2) DMV automation difficulties. The status of two reports (Veterans' Home and "Capability Assessment") is unknown.

The DOIT Has Not Responded to Request for Information on County-Based Projects. One important area where DOIT has failed to respond to the supplemental report requests is in the area of county-based projects. The state has several large county-based automation projects underway. (These are described in detail in our analysis of budget Item 4130 the Health and Human Services Agency Data Center [HHSDC].) Both DOIT and HHSDC provide some level of oversight for these various projects. Last year the Legislature adopted supplemental report language asking the two departments to explain how their oversight roles differ and at what point in a project's development they provide that oversight.

In response to this request, HHSDC developed an oversight plan, with input from the key stakeholders, and provided a report to the Legislature in January 2001. The Legislature also directed DOIT to submit a report which would describe DOIT's oversight role and activities for county-based projects and describe how these activities would be conducted between HHSDC and the counties. The DOIT stated that it met the intent of the supplemental report by providing input into HHSDC's oversight plan.

In our view, the HHSDC oversight plan was not intended, nor does it meet, the intent of the supplemental report for DOIT. It does not describe the role of DOIT, nor does it define the activities that DOIT will conduct. Finally, it does not describe how DOIT's activities will be coordinated between the counties and HHSDC. We recommend that DOIT submit a separate report on county projects prior to budget hearings which addresses the issues identified in last year's supplemental report language.

Bureau of State Audits (BSA) Report Would Assist Legislature's Review of DOIT Budget. In June 2000, the Joint Legislative Audit Committee authorized BSA to undertake a comprehensive review of the state's IT program and projects. The scope of the audit directs BSA to:

• Create a statewide inventory of major IT projects that are in the planning and implementation stages.
• Determine if DOIT has adopted an overall IT strategic plan that guides departmental IT efforts and whether or not departmental strategic plans are aligned with DOIT's plan.
• Select and examine a sample of projects to determine if those projects have been approved by DOIT and designed, managed, and evaluated in accordance with generally accepted project management principles.
• Determine if DOIT has a mechanism for ensuring cooperation among departments on mutually beneficial projects such as E-mail systems.
• Evaluate the sufficiency of DOIT resources.
• Determine if and how DOIT ensures that project risk is shared with vendors.

The BSA advises that the audit will be completed in spring 2001 and that the results will be provided to the Legislature at that time. Because the audit findings may have significant budgetary implications, the Legislature may wish to consider BSA's findings prior to taking any action on DOIT's budget. We, therefore, recommend that the BSA report to the Legislature during budget hearings on the results of this audit and that the Legislature not take action on DOIT's budget until it has had an opportunity to receive and review these audit findings as well as the reports requested by the Legislature.

Information Technology Hardware and Software Replacement Policy Is Needed

The budget contains a number of proposals totaling $15.1 million ($12.7 million General Fund, $2.1 million special fund, and $300,000 in reimbursements) for the replacement or upgrade of departmental personal computer-based office automation systems. We concur with the merits of these various proposals. We recommend, however, that the Department of Information Technology and the Department of Finance update their current policies to ensure that departments adequately plan and budget for the replacement or upgrades of these departmental systems.

Background. Every department has some kind of office automation system which provides word processing, spreadsheets, and file sharing capabilities. These office automation systems are composed of personal computers (PC), printers, file servers, and networks. Departments are able to acquire these systems through two types of budget mechanisms. Specifically, a department can prepare a FSR with an associated budget request or the department can use the state's Workgroup Computing Policy (WCP).

The state developed the WCP in the 1980s in an effort to allow departments to redirect savings in departmental operating equipment and expense funds to purchase office automation systems. The state's WCP requires that each department develop its own policy that ensures PC compatibility, describes planning and management strategies, and ensures proper oversight and risk mitigation.

Current Policy Seems to Work Well for Acquiring Systems. Since the initiation of the state's WCP, office automation systems have become common place in the state and have probably improved departments' overall internal operations in such areas as reducing the need for typing pools and allowing staff to share documents.

Current Policy Does Not Address Periodic Replacement and Upgrades. The current policy, however, does not address the need to plan and budget for periodic replacement or upgrades to these systems. We have seen evidence of this planning and budget deficiency during the reviews of the state's Year 2000 (Y2K) funding requests. Several departments had to request funds to replace outdated PCs so they could use the Y2K compliant software. In this year's budget proposal, there are a number of similar "replacement" requests summarized in Figure 2 to upgrade office automation systems.

Policy Needed Requiring Departments to Budget for Equipment Replacements. We have no concerns with the merits of these particular proposals. We do, however, see a need for the state to update its current policies to require departmental planning and budget adjustments for the periodic replacement and upgrades of their PC-based office automation systems. For this reason, we recommend the adoption of the following supplemental report language for DOIT and Department of Finance (DOF):

The Department of Information Technology and Department of Finance shall by September 1, 2001 issue policies requiring departments to plan and budget for the periodic replacement and upgrades of personal computer-based office automation systems. The policies should include recommended replacement and upgrade schedules and budgeting guidelines for the funding of these replacement and upgrade activities.

Post-Implementation Evaluation Reporting Process Deficient

Our review has found a number of deficiencies and problems with the administration's current process for conducting post-implementation evaluation reports (PIER). We, therefore, recommend that the Legislature (1) fund only those projects with identified measurable benefits; and (2) direct the Department of Information Technology (DOIT) and Department of Finance, through supplemental report language, to issue policies on criteria and funding for independent evaluations. We further recommend the Legislature adopt a three-pronged approach to improve PIER practices during the current year, budget year, and beyond which includes: (1) requiring departments to report at budget hearings on completed Information Technology projects (current year), (2) adopting budget control language requiring DOIT to report on the results of its PIER reviews (budget year), and (3) enacting legislation requiring the administration to provide PIER information (ongoing years).

Background. When the Legislature receives a budget proposal or a FSR for a state IT project, the proposal generally identifies proposed cost savings or improved efficiencies the project is expected to achieve. Figure 3 provides examples of expected benefits from some state IT projects currently under development.

According to the State Administrative Manual (SAM), departments are required to prepare a post-implementation evaluation report (PIER) for each completed state IT project. The assessment must measure the benefits and costs of the implemented IT system and document projected maintenance and operation costs over the life of the system.

The PIER is intended to document what was expected to be achieved and what was actually achieved. Without PIERs, the Legislature is unable to know if an implemented project ever achieved any savings, efficiencies, or other benefits as the administration originally proposed. In addition, without PIERs, the Legislature cannot determine how much an IT system ultimately cost to develop or will cost to operate and maintain on an annual basis.

How the PIER Process Currently Is Administered. According to SAM, most assessments are to be conducted within six months of implementation of the IT project but could occur up to two years after implementation, depending on the nature of the project.

Departments generally conduct their own assessments. In some cases, however, the DOF's Office of State Audits and Evaluation may conduct the assessment and prepare the PIER. After a department has completed the PIER, the PIER is provided to DOIT and the Legislature. The DOF also requires that all PIERs be submitted to it for review. The DOIT evaluates the PIER to determine if the project achieved the anticipated benefits and savings and approves the PIER to finalize the project.

Deficiencies and Problems Exist With Current PIER Process. In our review of the state's PIER process, we found a number of deficiencies and problems which are summarized in Figure 4.

Figure 4

LAO Findings on State's Current PIER Process

No tracking system exists to monitor when PIERs are due or submitted to the Legislature. As a result, the Legislature typically does not receive information about benefit and cost savings resulting from completed IT projects.

The DOIT does not keep records of which IT projects have been completed.

DOIT's PIER review process has lower priority than the review of feasibility study reports and special project reports.

Measurable benefits have not been identified for many IT projects.

Departmental reviews lack independence.

No Tracking System Exists to Monitor PIER Due Dates or Submission to the Legislature. In our review, we found that neither DOIT nor DOF track when PIERs are due on state IT projects. In addition, neither department provides any follow-up when PIERs are overdue. Since there is no tracking mechanism to ensure accountability, it is unclear if departments are completing their PIERs as required by current state policy.

In addition, even though departments are required to provide PIERs to the Legislature, we could find no tracking system within the administration that ensures departmental compliance. It is our understanding that neither DOIT nor DOF is responsible for monitoring whether departments provide PIERs to the Legislature. Therefore, to the extent that the Legislature does not receive PIERs, it is unable to identify the benefits or savings that accrue to the state as a result of IT projects.

The DOIT Does Not Keep Records of Completed IT Projects. The SAM states that DOIT's approval of a PIER "completes" a project. Our review indicates that DOIT does not keep records of PIER reviews. In addition, we found that DOIT does not consistently notify the administration or the Legislature concerning the results of its PIER reviews. According to our records, there currently could be over 400 "open" IT projects (that is, projects for which a PIER has not been submitted to DOIT).

The DOIT Gives PIER Review Process Low Priority. Of the 87 PIERs that DOIT has received from other agencies, DOIT is unable to identify the number it has actually reviewed and approved. According to DOIT, the review of PIERs has lower priority than its other activities such as reviewing FSRs and special project reports. In addition, DOIT states that it did not have enough resources to process this workload until the Legislature increased its budget in the 2000-01 Budget Act.

Measurable Benefits Have Not Been Identified for Many State IT Projects. State policy requires that each state IT project result in measurable program benefits. The PIER is the mechanism used to determine how well the IT system was able to achieve those benefits.

In our review, we found that measurable benefits have not been identified in the FSR for many IT projects, even though such benefits may very well result from the projects. Therefore, it is difficult for departments to evaluate how well the IT project was able to provide measurable benefits.

Departmental Self-Evaluations May Lack Objectivity. Current state practice directs departments to perform their own evaluation of implemented projects. This practice has a potentially significant limitation, particularly for large scale projects: the department that implemented the project is also the same department that evaluates whether or not the project was successful in providing the intended benefits. This practice of using the implementor of a project to also evaluate the results of the project may call into question the validity and objectivity of the final evaluation.

Improvements in PIER Process Needed. We believe that there are a number of steps the Legislature can take to improve the state's PIER process which are summarized below. We recommend that the Legislature use a three-pronged approach with actions to be taken in the current year, the budget year, and future years.

Budget Funds Only for Projects With Identified Measurable Benefits and Require DOIT and DOF to Issue Policies on Project Evaluations. We recommend the Legislature fund only those state IT projects that have specifically identified measurable goals of either reducing government costs, improving service, or increasing state efficiencies. In addition, the Legislature should direct DOIT and DOF, through supplemental report language, to develop and issue policies for use in determining when a post-implementation evaluation review warrants preparation by an independent evaluator. The policies should specify (1) the size and types of projects requiring an independent evaluation and (2) procedures for selecting and funding independent evaluators. The following supplemental report language is consistent with this recommendation.

The Departments of Information Technology and Finance shall issue by January 1, 2002 policies for use by departments in determining when a post-implementation evaluation review warrants preparation by an independent evaluator. The policies shall specify (1) the types of projects requiring an independent evaluation and (2) procedures for selecting and funding independent evaluators.

Require Administration to Report on PIER Reviews. During this year's budget hearings, the Legislature should require that the departments report on all IT projects completed in the last year. (The Legislature has used this approach in past budget hearings for other statewide issues of legislative concern such as implementation of the Dymally-Alatorre Bilingual Services Act of 1973 and departmental compliance with statutory recycling requirements.) Departments should be able to identify what projects have been completed, the benefits or cost savings that were achieved, any cost or schedule deviations that may have occurred, and the annual costs associated with the maintenance and operation of the implemented system.

For the budget year, we recommend that the administration report to the Legislature on the results of PIER evaluations and reviews. Accordingly, we recommend that the Legislature adopt the following budget bill language:

Within 30 days of receiving a Post-Implementation Evaluation Report (PIER), the Department of Information Technology shall provide a copy of the PIER and the results of its PIER evaluation to the chairs of the budget committees in each house and the Chair of the Joint Legislative Budget Committee. The results of the PIER evaluation shall describe the benefits that were achieved as a result of the implemented project, any cost or schedule deviations that may have occurred during the development of the project, and the annual costs associated with the maintenance and operation of the implemented system.

Enact Legislation Requiring the Administration to Provide PIER Information. When the Legislature considers legislation to extend DOIT's sunset date, we recommend adding several provisions to improve the process for conducting post-implementation evaluation reviews in future years. We suggest that language be included to:

• Require the administration to provide copies of the PIER and DOIT's PIER approval letter to the fiscal committees of both houses and the Joint Legislative Budget Committee.
• Require DOIT to report annually on all completed IT projects, overall benefits and savings that were achieved, any cost or schedule deviations that may have occurred, and the annual costs associated with maintenance and operation of the implemented systems.