General Government
The Year 2000 ("Y2K")Computer Problem |
We recommend that the Legislature (1) require departments to provide, during budget hearings, a status of their efforts to modify computers to accommodate the year 2000, (2) delay funding for new projects in those departments which have not modified their computers for year 2000, (3) approve DOIT's budget to provide additional oversight and assistance to departments, and (4) establish a reserve fund from which departments can request monies for yet-to-be identified efforts to address the problems related to the year 2000 conversion.
The budget proposes augmentations totaling $19.6 million by state agencies to fund efforts to modify computers to accommodate the year 2000 (Y2K). This does not include hundreds of millions of dollars in re-directions within departments' budgets to address these efforts. In this analysis, we outline the nature of the problem, what the state is doing about it, the potential costs and risks, and how the Legislature can ensure that the problem is addressed adequately.
Failure to make the Y2K change will for some systems simply produce undetectable erroneous calculations, but some systems will completely fail. The "failure date," as it is known, is not necessarily January 1, 2000. For some systems in California, it has been as early as 1995, because these systems were required to provide dates into the future (for example, licenses that were granted in the mid- to late 1990s that will expire after the year 2000).
The potential risk for the state if computer systems are not converted is significant. For example, the Department of Motor Vehicles reports that the failure to convert its computer systems could result in $3 billion in lost revenue annually. The Department of Consumer Affairs reports a potential annual revenue loss of $241 million.
The DOIT identified almost 3,000 state computer systems; of these 1,100 are either fixed already or do not need to be fixed. Of the remaining 1,900 systems, 650 are mission critical, meaning that they enable the department to carry out its primary responsibilities such as issuing drivers' licenses, collecting taxes, et cetera. The remaining 1,250 systems were identified as essential but not mission critical and will also need to be fixed.
Once identified, there are several ways to fix the code. The code can be changed to accept dates with the entire year included (1973 rather than 73, 1910 or 2010 rather than 10). An alternative method is referred to as "windowing" in which the system is told that any date between 00-10 should be considered in the 2000s and any date from 11 through 99 should be considered in the 1900s. A department can choose one of these or many other solutions, depending upon what type of code it has and how much the existing code has been customized over the years. The more customized the code, the more manual intervention will be necessary.
Some of the methods of remediation, such as windowing, should be considered short-term fixes because dates will once again become a problem as the calendar nears the end of the window. Those departments that use a short-term fix may find that they still have to replace an entire system in the near future.
The DOIT's Y2K Program Guide lists the steps a department should go through to establish a plan for modifying the affected systems. It requires departments to create an inventory of existing systems; identify those systems that are critical to the overall mission of the department (referred to as "mission critical systems"); assess the impact the century change will have on these systems; and develop a plan to fix the systems. Departments had to identify the impact its computer systems have on the department, outside entities with whom it exchanges information, and the public. In addition, DOIT required departments to report this information as well as budget and Y2K conversion schedule in June 1997 with quarterly updates thereafter. Executive Order W-163-97, issued in October 1997, requires all state departments to complete their Y2K remediation efforts by December 31, 1998.
Current-Year Funding. The 1997-98 Budget Act (Item 9899) appropriated $55 million for Y2K conversion costs. This included $25 million from the General Fund, $25 million from special funds, and $5 million from non-governmental cost funds. As of early January 1998, 14 departments have requested and received about $44 million.
Budget-Year Funding. The Governor's budget does not propose to include funding for Y2K efforts in Item 9899 for 1998-99, but rather includes funding for Y2K projects in individual departmental budgets. The budget proposes total augmentations of $19.6 million in 12 departments for 1998-99. This amount does not include funds in the baseline budgets. The Department of Finance has not been able to provide information on how much departments have in their baseline budgets for Y2K efforts.
Embedded-Chip Technology. Nontraditional information technology, known as "embedded chip" technology, also needs to be fixed to accommodate Y2K. This includes elevators, voice mail systems, security systems, heating and air conditioning systems, as well as many other aspects of state government's daily functions. At this time, few vendors have indicated an ability or willingness to perform remediation efforts on the nontraditional systems. Although DOIT's program requires departments to indicate whether they have created plans and embarked on these efforts, the program does not require this to be included in the department's cost estimate, and DOIT does not review these plans. Thus, the state's planning and budgeting for these efforts is in the preliminary stages.
Additionally, each contract needs to be reviewed to determine responsibility for liability for noncompliance. If a major system is not remediated in time, the state runs the risk of being sued by someone who alleges harm as a result of the system's failure. As we indicated above, industry experts predict that legal expenses could add 40 percent to projects costs. Forty percent of the existing $240 million estimate is approximately $100 million for state government. Determining which failures pose the greatest risk to the state should help to prioritize remediation efforts throughout the state and thus reduce potential liability. Review of existing contracts by qualified legal counsel is an important component to the total effort. The DOIT proposes in its 1998-99 budget to provide these services for all state departments that request it.
Testing. The time lines outlined by the departments and reported by DOIT do not provide sufficient time for testing. After the computer code is fixed, the system must be tested. It is generally accepted in the information technology industry that 40 percent of total project time and cost should be devoted to testing. Testing is a critical component of a project because it is at this phase that problems are identified. If adequate time is not set aside for testing, and significant problems are identified when the system is tested, the project may not be completed on time. Once testing has proven successful, the system is put into "production." This is the process in which the system is actually implemented and workers are trained to use it.
Contingency Planning. The time lines do not incorporate time to activate or invoke contingency plans in the event a remediation effort fails to meet project milestones.
Entities With Whom the State Exchanges Data. The state exchanges data with many entities including the federal government, welfare offices, car dealerships, hospitals, schools, and emergency response services. The time lines do not always include enough time for the state department to develop a "bridge" with these partners in the event the partner is unable to modify its systems in time. Without a bridge, even if the state modifies its system on time, it will be unable to exchange data with the confidence that the data are accurate.
As a result of the lack of adequate oversight, the Legislature will not be apprised of the full extent of the problem, efforts being undertaken to fix it, the budget required to complete these efforts, and the fiscal effects if systems are not fixed on time. Since this is the largest information technology project the state has ever undertaken and it has significant ramifications to all citizens, closer monitoring by both the administration and Legislature is warranted.
During budget hearings, the budget subcommittees should require each department to provide a status report on its progress in making its systems Y2K compliant. Some departments which submitted plans to DOIT did not indicate how they were going to fix embedded chip problems; what the impact to the General Fund and special funds would be if its systems failed; potential legal exposure; or what they intended to do should their plans fail. The DOIT does not know whether these departments have a backup plan if initial efforts to fix the computer systems are unsuccessful. If a large computer system which issues billings or tracks collections is not remediated before its failure date, the state could stand to lose hundreds of millions of dollars. Likewise, should a computer system which keeps records for law enforcement not be expected to be remediated on time, the Legislature should know of the potential problems beforehand.
According to DOIT's Y2K project time line, all departments should now be done with developing Y2K solutions and should be in the testing phase. If departments are not in the testing phase, it may be a sign that the state will be at risk of not meeting the deadline. Many department's plans contain a minimum number of days between the time at which the first failure date occurs and the time at which the system will be fixed, known as the "cushion" time. This is a key factor in determining risk. A cushion is needed to allow for resolving unanticipated problems due to the complexity of the Y2K modifications. The less cushion time, the higher the risk to achieving success.
Here are some of the questions that the budget subcommittees should ask of departments during budget hearings:
We recommend that the Legislature require every department, during budget hearings, to provide an update on the status of its efforts to complete remediation efforts. This would allow the Legislature to direct the departments to improve their efforts, develop contingency plans, or take other steps to ensure that the negative impacts of failure to remediate systems are minimized.
In order to encourage departments to focus their resources on Y2K remediation efforts, the Governor issued an executive order (W-163-97), which requires the deferral of new computer systems unless the proposed new system is legislatively mandated or is necessary to help the department become Y2K compliant. A preliminary review of approval letters indicates 28 new projects (totaling over $200 million) were approved in the three months since issuance of the executive order. Of those, only 11 could be identified as being mandated to meet a department's business needs or necessary for a department to become Y2K compliant.
Some of these nonmandated projects are being requested by departments which will not have their systems compliant before they fail or before the Governor's deadline of December 31, 1998.
Due to the nature of the impact of the century change on nearly every aspect of an organization's mission and the immovable deadline, any new project will likely deflect resources from Y2K efforts. For this reason, we recommend that the Legislature deny funding new projects until the department has its mission critical systems working correctly or the system is mandated by law. In addition to mission critical systems, departments still need to make their essential but nonmission critical systems Y2K compliant. The Legislature may consider requiring a department to have all of its systems substantially compliant before approving funding requests for new systems. Failure of the systems that are essential, but not mission critical, may nonetheless significantly affect the ability of an entity to do business effectively.
In our analysis of the DOIT budget (later in this chapter), we recommend approval of the department's request for additional resources for its Y2K office. Currently, DOIT is unable to review many department's plans for, and monitor progress toward, remediation. We believe that close monitoring of departmental Y2K efforts by the state's information technology office is necessary, and recommend that they be provided with the tools to do so. We believe that the funding request is the minimum necessary to accomplish the task.
The state is involved in a multiyear effort to be Y2K compliant. As we indicated earlier, the state's estimate of Y2K remediation costs is probably understated. This is because not all of the plans filed with DOIT include remediation of all systems (especially essential, but not mission critical systems, and embedded chip), implementation of contingency plans, and legal expenses. We believe that the Legislature is likely to receive additional funding requests for these efforts in addition to what is in the Governor's proposed budget.
Departments are refining their cost estimates as they further define the scope of their Y2K remediation efforts. Departments will need to perform the same remediation efforts for nonmission critical systems, potentially invoke contingency plans, review existing contract terms, and defend themselves in court. Many of these expenses have not been identified yet and are not accounted for in the proposed budget.
The Department of Finance is only tracking the augmentation requests made by departments. It
is not tracking how much departments are actually spending on Y2K efforts. Nor does it plan to
track the Y2K efforts in a way to be able to plan for revenue shortfalls in the General Fund or
special funds should these systems not be fixed on time. This will be problematic if a computer
system at a department that is charged with collecting revenue does not function properly.
Since departments will not be able to wait until the 1999-00 Budget Actfor appropriations to resolve these yet-to-be identified efforts, and shortfalls in revenue collected poses a potential problem to the General Fund, the Legislature should consider setting aside funds to resolve these problems. Although some level of redirection of resources by departments is appropriate, we believe that additional resources will be necessary. In the 1997-98 Budget Act, the Legislature appropriated $55 million (all funds) for Y2K remediation efforts. We believe that a reserve fund of at least that magnitude is warranted for 1998-99 above the amounts proposed for the individual departmental budgets.