Legislative Analyst's Office Analysis of the 2001-02 Budget Bill

State Data Centers

Data Center Study Needed

In 1997, the Department of Information Technology (DOIT) completed a study which recommended ways to reduce the number of state-operated data centers. In 1998, DOIT suspended its data center consolidation activities to focus on completing Year 2000 (Y2K) remediation activities. At the time that this analysis was prepared, DOIT had not resumed its data center consolidation efforts.

A number of factors, including technological changes and staff recruitment and retention issues, have rendered DOIT's 1997 Data Center Consolidation Report out-of-date. We therefore recommend that the Legislature direct DOIT to report at budget hearings on the resources and time frames needed to conduct a study which (1) examines data centers' rates for nonmainframe activities, (2) identifies potential opportunities for specialization between the state's primary data centers, and (3) identifies data center functions that can be provided more efficiently by private industry.

The Stephen P. Teale Data Center (TDC) and the Health and Human Services Agency Data Center (HHSDC) are the state's two primary data centers which provide computer and network services to various departments and local jurisdictions. The two data centers are funded entirely through reimbursements from client departments, and their combined expenditure authority is $234.8 million in the budget year.

Data Center Establishment. Chapter 787, Statutes of 1972 (SB 1503, Teale) established the state's current data centers to serve specific program areas, as summarized in Figure 1.

The HHSDC, TDC, and Hawkins Data Center provide the majority of the state's centralized computing services. Neither the Franchise Tax Board (FTB) nor Board of Equalization (BOE) data centers ever developed the larger computing capacity that characterized the other three data centers.

Legislature Directs Administration to Evaluate Consolidation Benefits. Chapter 508, Statutes of 1995 (SB 1, Alquist) made significant changes in the planning, implementation, and oversight of the state's information technology (IT) activities. Specifically, this legislation directed the DOIT to conduct a study evaluating the benefits of consolidating the existing data centers.

  Findings From DOIT's 1997 Data Center Consolidation Study. Figure 2 summarizes the major recommendations from DOIT's 1997 Data Center Consolidation Study.

The study concluded that little savings could be achieved ($7.7 million annually after an initial one-time investment of $12.9 million) by consolidating HHSDC and TDC.

Administration Suspends Implementation of Data Center Consolidation Due to Y2K Efforts. In 1998 the administration notified the Legislature that it would suspend all data center consolidation efforts until completion of Y2K activities with the exception of placing midrange computers at either HHSDC or TDC. At the time this analysis was prepared, DOIT had not resumed data center consolidation efforts. As Figure 2 shows, to date, two of the recommendations have been implemented, several were considered but not implemented, and one recommendation was partially implemented.

New Factors to Consider on Data Center Consolidation

The 1997 study identified several anticipated events, such as staff retirements and changes in contracting costs that might require the state to reevaluate its recommendations at some time in the future. In fact, the state has experienced a number of changes in its IT operations which render the DOIT's earlier report, with its findings and conclusions, out-of-date. These factors are summarized in Figure 3 and discussed in more detail below.

Figure 3

New Factors to Consider in Data Center Study

Technology has shifted from mainframe to mid-range systems; much mainframe consolidation has already been accomplished

Network technology has become more complex

Data center rates for nonmainframe activities are difficult to calculate

State has a staff recruitment and retention problem for information technology classifications

Stephen P. Teale Data Center (TDC) may lose specialized skills as older workforce retires

Health and Human Services Agency Data Center and TDC perform identical functions

More options may be available to contract with private business for specific functions

Technology Has Shifted From Mainframe to Midrange Systems. Technology has dramatically changed since the establishment of the data centers in 1972. At that time, mainframes were the only computers available for processing large volumes of data. Now, midrange systems provide computing power on par with earlier mainframes. The DOIT's 1997 report focused almost exclusively on mainframe systems.

In 1997, DOIT reported to the Legislature that it intended to consolidate the existing IBM-compatible mainframe operations at FTB, BOE, State Controller's Office (SCO), Public Employees' Retirement System (PERS), California State Lottery, and State Treasurer's Office (STO) into TDC. During our review, we found that BOE, STO, and SCO have consolidated all of their mainframe operations into TDC. The administration does not have statutory authority to implement consolidation at PERS and the California State Lottery. At this point it is unlikely that the state would achieve much additional benefit from mainframe consolidation because mainframes are no longer as prevalent as they were in 1997.

The DOIT's 1997 report gave little attention to nonmainframe systems. Since that time, both HHSDC and TDC have experienced tremendous growth (over 400 percent) in midrange systems. As a result of these changes, earlier consolidation recommendations may not address current circumstances and therefore are of limited value.

Network Technology Has Become More Complex. As the state has shifted from mainframe to midrange systems, the data communications networks have also changed. Mainframe communications networks used to allow only one type of data communications signal to transmit through the networks. Now, data networks allow multiple communication signals to transmit through the networks, thereby permitting multiple systems such as personal computers (PC), midrange and mainframes systems to connect to the same network. This makes it more costly and complicated to design, implement, and maintain these networks. Both HHSDC and TDC operate these more complex data networks and have requested budget increases over the past few years to support them. The DOIT's 1997 study did not analyze these functions.

Data Center Charges for Nonmainframe Activities Are Difficult to Calculate. The mainframe and early networks systems were designed for use by multiple organizations. For this reason, they contained mechanisms which allowed data centers to charge based on actual usage rates. The newer midrange and network systems do not have those same mechanisms thereby making it difficult for data centers to base charges on actual usage.

In our review, we found that HHSDC and TDC use different methods to calculate charges for supporting similar systems. In addition, we found that the administration has not examined these rate setting differences to determine which would be most appropriate.

State Has Problems With IT Staff Recruitment and Retention. For the past several years, the state has experienced some problems in recruiting and retaining IT staff. Generally, the state provides lower salaries and incentives than private industry and experiences additional recruitment difficulties due to civil service requirements. The state's current vacancy rate for IT classifications, which ranges between 15 percent to 19 percent, is slightly higher than the overall state vacancy rate of 15 percent. Further examination is needed to assess the state's ability to fill IT positions, particularly at the state's primary data centers which support critical IT systems.

The TDC Has Older Workforce. The DOIT's report found that HHSDC and TDC had an older workforce with highly specialized skills. The report indicated that data centers run the risk of losing these skills due to higher retirement rates among this segment of the workforce.

Our review indicates that TDC has a slightly older workforce than the state's IT workforce as a whole. For example, 17 percent of the state IT workforce is between the ages of 51 and 55 compared to 23 percent of TDC's workforce. This means that TDC can anticipate losing some of its highly skilled workforce in the near future and may be unable to adequately replace these employees because of the state's overall problem of IT staff recruitment.

The HHSDC and TDC Perform Identical Functions. Both HHSDC and TDC perform the same functions including supporting mainframe and midrange systems and communications networks. The exceptions are HHSDC's oversight of large automation projects and TDC's support of Geographical Information Systems. The DOIT's 1997 report did not examine the benefits of specializing existing functions between the two data centers, but instead focused primarily on reducing the number of mainframe data centers. We believe it may be possible to achieve efficiencies—either in cost savings or improved operations—by each data center specializing in particular functions. For example, one data center could provide support to mainframes while the other could support midrange systems.

More Outsourcing Options May Be Available. The DOIT's 1997 report also recommended that the state have private industry perform selected data center functions (a process known as "outsourcing") such as printing, help desk, and network support. Since the report was issued, both HHSDC and TDC have contracted out their printing functions, but not others.

Since technology has shifted from the mainframe to midrange systems, private industry now provides services that were not available in 1997. For example, private data centers provide "web hosting" services which allow customers to be responsible for maintaining web pages while the private data center supports and operates the computer. With anticipated retirements and the state's difficulty in recruiting and retaining IT staff, outsourcing may offer a solution to providing adequate support for some of the state's IT operations.

Data Center Study

The primary purpose of the 1997 study was to (1) reduce the number of mainframe data centers operated by the state, (2) identify areas for contracting with private vendors to provide state data center operations, and (3) identify state functions that could be performed efficiently by private industry. In view of the issues discussed above, DOIT's 1997 report is out-of-date. A new data center study is needed to focus on those activities that were not reviewed in the 1997 study. Those would include (1) examining the rate setting methods for nonmainframe activities, (2) identifying opportunities to specialize existing functions between the state's two primary data centers, and (3) examining potential new outsourcing opportunities that have developed since the 1997 study.

The DOIT Should Report on Resources Needed to Conduct Study. We recommend that the Legislature direct DOIT to report at budget hearings on the resources and time frames it would need to conduct a new data center consolidation study. The DOIT should identify the resources needed to:

• Identify existing data center functions that can be specialized between HHSDC and TDC and which will result in reduced costs or improved efficiencies.
• Examine HHSDC's and TDC's methods for rate-setting for nonmainframe activities and develop a single consistent and appropriate methodology.
• Identify those data center functions private industry can provide at a reduced cost and/or with improved efficiency.