March 26, 2014

The 2014-15 Budget:

Evaluating FI$Cal Project Plan

• Report in PDF

• In This Report
• Background
• FI$Cal Project Status
• Major Components of the Updated FI$Cal Project Plan
• Governor’s Budget Proposal
• LAO Findings
• LAO Recommendations

Background

The FI$Cal Project. Over the last several years, the administration has been engaged in the design, development, and implementation of the FI$Cal Project, which will replace the state’s aging and decentralized IT financial systems. The FI$Cal Project will transform state government processes in the areas of budgeting, accounting, procurement, and cash management by (1) eliminating the need for over 2,500 department–specific applications and (2) enabling the state systems and workforce to function in an integrated financial management system environment. The project is managed by a partnership of control agencies: Department of Finance (DOF), Department of General Services (DGS), State Controller’s Office (SCO), and State Treasurer’s Office (STO). The project intends to deploy FI$Cal in five waves, with more departments and functionality added with each wave.

Evolution of the Project. The planning for the FI$Cal Project began in 2005 when DOF proposed an IT project that would implement an internal financial system for the department. The Budget Information System, as this system would have been called, was envisioned to better meet DOF’s budget development and administrative needs. In 2006, project staff proposed an updated project plan that significantly changed the scope and governance of the project. Rather than building a new system exclusively for DOF, project staff recommended that because a majority of state departments were reliant on aged and inadequate technology, there was a need to modernize and replace the state’s entire financial management infrastructure. The updated project plan proposed increasing the scope of the project to include developing a single integrated financial information system for the state. The project was renamed FI$Cal and would be managed by a partnership of control agencies that would sit on the project’s steering committee. After continued planning and a lengthy procurement that used an innovative procurement approach, a vendor was selected in 2012 to configure existing financial management software applications to address the state’s needs. (Refer to our April 30, 2012 report, The 2012–13 Budget: Evaluating FI$Cal, for a more comprehensive description of the project’s history.)

FI$Cal Project Status

Pre–Wave Deployment Successful

The project deployed Pre–Wave—the first of the five implementation waves—on July 1, 2013. Pre–Wave deployed procurement functionality—including requisitions, purchase orders, and receiving—in seven state entities. The project also implemented a vendor master file for Pre–Wave entities, the first step in developing a single list of all vendors that do business with the state.

The Pre–Wave deployment also triggered the establishment of the FI$Cal Services Center (FSC). The center serves as the project’s point of contact with departments using the system. Ultimately, the FSC will be responsible for maintaining and operating the entire FI$Cal system, including offering supportive services to departments. The FSC will be deployed incrementally through each of the project deployment waves. All components of Pre–Wave were deployed successfully and without incident.

The activities necessary for deploying FI$Cal to departments are largely consistent over the various waves, such as engaging departments, converting data from current technology systems to FI$Cal, training end users, and testing the system. Pre–Wave was designed as a pilot to test these procedures. Although the scope of Pre–Wave was narrow and only a few departments were impacted, Pre–Wave allowed the project to become familiar with (1) the opportunities and challenges associated with the technology; (2) control agency and departmental technical and organizational landscape; and (3) the project methodology that will be used through the entire project lifecycle to analyze, design, build, test, and deploy FI$Cal. As we discuss further below, this pilot allowed the project to gain valuable expertise and draw lessons that can inform decisions moving forward.

Issues in Upcoming Wave 1 Merit Close Attention

Challenges With Wave 1 Budget Functions. The next stage of FI$Cal’s deployment—Wave 1—is scheduled to begin in July 2014 and will deploy financial and budgeting functions to the FI$Cal partner agencies and a limited number of other departments. Specifically, Wave 1 is designed to deploy (1) DOF’s budget control agency functions, establishing FI$Cal as the budget system of record (meaning that FI$Cal will maintain the official record of the state’s budget); and (2) department–level financial functions, including accounting, cash management, and procurement. The project is currently experiencing challenges developing the budget functions associated with Wave 1. Challenges exist in (1) developing the interface between the budget and financial functions, which will be required to share information; and (2) configuring the budget IT solution to accurately reflect the state’s budget operations. These challenges have caused delays in building the budget functions and are subsequently shortening the time available for the testing activities in Wave 1. Although IT projects typically identify some defects upon initial deployment, testing prior to deployment is intended to minimize the scope and severity of system disruptions. The FI$Cal Project could be negatively affected if these challenges are addressed too close to deployment and the project hastens testing in order to meet the scheduled July 2014 deployment.

The state’s Department of Technology and a third–party technical consultant recently identified performance– and capacity–related concerns regarding FI$Cal’s long–term ability to meet the state’s budget functionality needs. These oversight entities are concerned that some of the budget functions may not be ready for implementation in July 2014, but they agree that the project has made significant strides in the last several weeks to address the challenges associated with the budget functions. The oversight entities have not identified significant concerns with the financial functions included in Wave 1.

Project Prioritizing Workload for Wave 1. The project indicates that despite the concerns of the oversight entities, it continues to make progress towards accomplishing the objectives of Wave 1 and remains on schedule for deployment in July 2014. Nevertheless, the project has also indicated that some of the budget functionality may not be ready (or necessary) for deployment on July 1. As a result, certain budget functionalities may be rolled out incrementally after July 1. The project indicates that the delay would have little impact on the budget process since certain budget functionality is not operationally required on July 1 (such as certain functions associated with the preparation of the Governor’s budget, which is released annually in January). The project is determining what activities for core budget functions are priorities for Wave 1. This analysis will inform the development of a contingency plan, if issues with Wave 1 cannot be addressed in time for a July 1 deployment.

High Vacancy Rate and Staff Retention Issues

Project Has Had Difficulties Filling Vacant Positions and Retaining Staff. Staffing difficulties create risk that an IT project will not meet key milestones that affect the project’s schedule and budget and can ultimately lead to a project failing. For example, an independent assessment of the terminated IT Modernization Project at the Department of Motor Vehicles concluded that staff turnover was a significant contributor to that project’s failure. For much of the FI$Cal Project’s history, FI$Cal management has contended with recruitment and retention difficulties.

Many State Departments Have Vacant Positions. The process of filling vacant positions is a normal part of human resources management for employers of any size in both the public and private sectors. In California state government, departments generally have the responsibility to fill vacant positions that have been authorized by the Legislature. The percentage of authorized positions that are vacant is known as the “vacancy rate.” At any one time, a substantial number of authorized positions are vacant in state government. According to data maintained by SCO, the vacancy rate for state positions in February 2014 was about 14 percent. Although vacancy rates vary widely across departments—and can vary throughout the year for a single department—the statewide vacancy rate has consistently been around this level for at least the past decade. (Refer to our write–ups entitled “Vacant Positions” in our Analysis of the 2008–09 Budget Bill and Analysis of the 2003–04 Budget Bill for a more comprehensive discussion of the state’s historical vacancy rate and past efforts to address high vacancy levels.)

Some positions are especially difficult to recruit or retain qualified employees. These difficulties may be due to the location of the job, the type of work performed, low salaries relative to those offered by other employers, or other factors. In some cases, the state provides employees with these difficult–to–fill positions additional pay on top of what is established for the position’s classification in an effort to improve a department’s recruitment and retention efforts. This additional pay is referred to as a “pay differential.” The Department of Human Resources manages the process to establish pay differentials.

Vacancies Cited as Project Risk. Over the past couple years, the State Auditor has flagged the project’s vacancy rate as a risk that the project should mitigate. In its February 2014 report, the State Auditor noted that the project has made substantial progress in reducing its vacancy rate from 22 percent in November 2012 to about 14 percent in December 2013. More recently, however, the project reported its February 2014 vacancy rate to be about 17 percent. The project’s historically high and volatile vacancy rate can be attributed to (1) a relatively limited pool of applicants with necessary skill sets, (2) obstacles inherent in the existing civil service process to hiring qualified staff quickly after positions are authorized by the Legislature, and (3) high turnover rates among project staff (including at executive levels). Even if the project’s overall vacancy rate were low, sustained vacancies or high turnover of key project positions can significantly compromise the project. The project has seen critical positions remain vacant for sustained periods of time as well as turnover in its leadership positions. A pay differential has not previously been provided to FI$Cal Project staff; however, the project has suggested in the past that providing a retention pay differential could increase the project’s ability to attract and retain qualified staff.

Major Components of the Updated FI$Cal Project Plan

In California state government, a feasibility study report documents the initial justification for an IT project and lays out the project plan, while any significant subsequent changes to the project plan—including project scope, schedule, and/or budget—are documented in SPRs. In January 2014, the FI$Cal Project submitted, and the Department of Technology approved, an SPR that updates the project plans. This was the fifth update (SPR 5) to the project plans since FI$Cal began its planning phase in 2005. The project indicates the proposed changes reflect lessons learned since the previous project update (SPR 4) was approved in March 2012. The last two years have been marked with significant accomplishments, most notably the selection of the primary vendor and the successful deployment of Pre–Wave. Drawing on lessons learned over this period, the project reports coming to a better understanding of the magnitude and complexity of FI$Cal, particularly Wave 2 as planned under SPR 4. Specifically, the project determined the risk associated with the project was too large and decided a different approach would be necessary moving forward in order to mitigate the risk of a significant disruption to the project in future years. The project indicates that the revised plan, outlined below, reduces the overall risk associated with the implementation of FI$Cal. See Figure 1 for a summary of functionality and departments included in each wave according to SPR 5.

Expands Project Scope to Completely Include DGS. The project determined during its planning phase that if a department had an operational financial management system—or was in the process of implementing such a system—it would be designated as a “deferred department.” In order that the state’s financial information is centrally housed, deferred departments will interface their current financial management systems with FI$Cal without being direct users of the new system. Over time, a deferred department will transition to FI$Cal as its current system exceeds its useful life. Other departments are statutorily exempt from ever becoming users of FI$Cal. Figure 2 lists departments that are deferred or entirely exempt from FI$Cal under SPR 5. Under SPR 4, DGS’s internal financial management system was deferred, but the department’s control agency procurement functions were included in the FI$Cal Project scope. The DGS’s internal financial management system has reportedly reached its end of life sooner than expected. Consequently, per SPR 5, the project proposes to accommodate a request by DGS to replace its existing internal financial management system with FI$Cal. The new updated project plan expands FI$Cal’s scope to accommodate this.

Revises Deployment Schedule for Final Three Waves. The project proposes to change the system deployment by shifting the bulk of departments to the final wave (Wave 4) so that control agency functionality is deployed before departments come on line. Specifically, the revised project plan makes the following changes to Waves 2, 3, and 4.

• Wave 2. The updated plan decreases the scope of Wave 2 by shifting the SCO and STO control functions previously in Wave 2 to Wave 3. The focus of Wave 2 becomes DGS’s control agency functions, but also includes the replacement of DGS’s internal financial management system and 50 contract fiscal services departments that rely heavily on procurement functionality. Wave 2 will begin three months earlier than previously planned to allow for 15 months of design and development of the procurement functions, however, its deployment date is still planned for July 2015.
• Wave 3. With the updated plan shifting SCO and STO control agency functions from Wave 2 to Wave 3, Wave 3 is expected to (1) begin six months earlier than previously anticipated and (2) end six months later. These changes allow for a 24–month design period and align the deployment with the start of 2016–17.
• Wave 4. Under the new plan, all other departments previously in Wave 2 and Wave 3 are shifted to Wave 4. The final wave of the project will also implement a public transparency website. The plan delays Wave 4’s deployment from July 2016 to July 2017 to allow a full 24 months for design and development.

Schedules Expected FI$Cal Software Upgrade. The previous project plan (SPR 4) indicated a need to upgrade the FI$Cal software but assumed the upgrade would take place after all waves were implemented. (Oracle—the developer of the IT system that FI$Cal will implement—has released a newer version of the software since the project began.) The new plan proposes to schedule the necessary upgrade to the FI$Cal software as part of Wave 3.

Does Not Change Wave 1. The new plan proposes no changes for Wave 1—scheduled to deploy in July 2014.

Requires Longer Period of Heightened Staffing Levels. The SPR 4 provided for a ramp–down of project staff—from the peak level—due to a decrease in workload in later waves. As a result of Wave 4 expanding in scope, SPR 5 requires sustaining the peak level of staff through the duration of the project (through 2017–18). Under the new plan, total staffing will peak in 2014–15 at 294 positions and is sustained at that level through the deployment of Wave 4. Staffing then decreases in 2018–19 when FI$Cal enters its first full year of maintenance and operation (M&O). Although the bulk of departments are shifted to the last wave under SPR 5, reaching a peak level of staff in 2014–15 and sustaining this level for the duration of the project seems reasonable given the more complex nature of control agency functions that will be deployed in Wave 2 and Wave 3.

Affects Project Cost and Schedule Relative to SPR 4. The proposed changes to the project reflected in SPR 5 result in a 12–month schedule extension, as shown in Figure 3, and increase the project cost by $56 million—to a total project cost of $673 million. (The increase in project cost is mostly attributed to the proposed levels of project staff over the additional 12 months.) These additional costs also include an increase in project vendor cost of $14 million—to a total vendor contract cost of $226 million. The actual project costs over the last seven years—from 2005–06 through 2012–13—were $160 million. In 2013–14, budgeted expenditures are $85 million. See Figure 4 for project costs (including planning and other predevelopment expenditures incurred to date) by fiscal year through 2018–19.

Governor’s Budget Proposal

The Governor’s budget proposes an increase of $4.3 million in 2014–15 to expand the project’s scope to include DGS’s internal financial management system as part of FI$Cal. This brings the total 2014–15 request for the FI$Cal Project to $107 million ($94 million General Fund). Beyond the 2014–15 budget proposal, the Governor identifies funding for the project through 2018–19, which includes the design, development, and implementation of the project as well as the first year of M&O as outlined in SPR 5. Approval of the Governor’s proposal allows the project to effectuate the proposed changes in the new project plan. The total cost of the project for 2015–16 through 2018–19 is $321 million ($254 million General Fund). As shown in Figure 4, this brings the total cost of the project (including the first full year of M&O) to $673 million.

LAO Findings

Plan Attempts to Mitigate Project Risk for Wave 2 Through Wave 4. Typically, project staff update plans in response to negative project indicators, such as a project consistently missing established milestones or tracking expenditures above its budget. This is not the case for FI$Cal, as the project is on schedule and within budget, per SPR 4. Instead, the update (SPR 5) is an attempt by project staff to reduce the risk associated with the project and improve the likelihood of FI$Cal’s success. The proposed changes reportedly reflect lessons learned in the nearly two years since the vendor was selected and the development of the system began. For example, the modifications to the project are described as reflecting the following lessons.

• Align Deployments With Start of Fiscal Year. The project reports that it has come to understand that the deployment of waves in July, at the start of the fiscal year, minimizes the disruption to departments associated with a mid–fiscal year deployment.
• Establish Stabilizing Control Agency Functions as a Priority. Minimizing the number of departments in Wave 2 and Wave 3 is intended to focus project resources on control agency functions. These changes may reduce the overall project risk because these functions are more complex than departmental implementations.
• Reduce Repeated Deployments to Departments. Because control agencies affect every other agency and department, there is substantial risk of rework to departments by not having control agency functions implemented and stabilized prior to bringing on the bulk of departments. The updated plan dramatically expands the scope of Wave 4 by shifting to Wave 4 most departments previously planned for implementation in Wave 2 and Wave 3. This transition and the Wave 3 system upgrade may further reduce risk and eliminate the need for retraining departments’ staff and the adverse impacts of repeated deployments.

While the updated project plan is associated with an extension of the project’s schedule and an increase in the project cost relative to SPR 4, these changes could potentially be less costly than moving forward with the prior plan. This is because the previous plan, especially the significant workload in Wave 2, could have resulted in costly rework that likely would have disrupted the state’s financial systems.

Plan Potentially Shifts Some Project Risk to Wave 4. While we think the modifications to FI$Cal in SPR 5 reduce overall project risk and strengthen FI$Cal’s likelihood of success, SPR 5 potentially shifts some of the implementation risk from earlier waves to Wave 4. Because SPR 5 deferred the deployment of FI$Cal for the bulk of departments until after control agency functions are stabilized, the number of departments included in Wave 4 has increased substantially under the new schedule. The large size of the final wave creates greater risk during this wave. The FI$Cal Project staff support departments throughout implementation as the departments participate in readiness activities related to changes in business processes and the technical aspects of using the FI$Cal systems. This time–intensive engagement is necessary to ensure departments are ready for system deployment. Given the breadth of the final wave, we question whether FI$Cal Project staffing will be at adequate levels at that time to actively engage departments as needed for Wave 4 to succeed within the proposed time frame.

Under New Schedule, Staff Turnover a Bigger Risk at Final Wave of Project. Under the revised project schedule, the final wave is more critical because of the number of departments included in the wave. In order to stay on schedule while rolling out the system to the increased number of users in the final wave, the project plans to maintain its peak staffing levels through the end of the project. After the final wave, many of these positions will be eliminated. We expect it will be increasingly difficult for the project to maintain a constant level of staff—let alone at peak levels—as the end of the project approaches because (1) employees will look for other employment opportunities in anticipation of their project job ending and (2) fewer qualified candidates will apply for vacant positions that will soon be eliminated. The likelihood of greater turnover and increasing vacancy rates towards the end of the project creates greater risk that the project will not successfully maintain the proposed schedule and budget in the final wave. Improving the project’s ability to retain qualified employees—especially towards the end of the project—would mitigate this risk.

Providing Retention Benefit Might Improve Retention Efforts. In an effort to mitigate risk resulting from staff turnover, the state has provided pay differentials for state workers employed by one IT project in the past. In an effort to address recruitment and retention issues at SCO’s 21st Century Project (MyCalPays System), the state provided a pay differential to state workers employed by the project. The state has no data to inform decision makers whether or not these pay differentials improved the 21st Century Project’s recruitment and retention efforts. We note that staff turnover is not considered a primary cause of that project’s suspension. (Refer to our March 19, 2014 report, The 2014–15 Budget: 21st Century Project Update, for more information regarding that project’s history.)

LAO Recommendations

Monitor and Reassess Approach for Wave 4 Through Modified Annual Reporting. Currently, the project is required to provide the Legislature an update in February of each year. As part of this update, the project is required to provide a discussion of lessons learned and best practices that will be incorporated into future changes in project management activities. We think it is important for the Legislature to remain apprised of project developments through the annual report. However, we think the timing of the annual report to the Legislature should more closely align with the deployment of FI$Cal waves. As the February annual report is not available to the Legislature until seven months after the project’s July deployments, too much time may have elapsed for the Legislature to address problems arising from the deployment of a wave. We therefore recommend the Legislature revise the due date for the annual report from February 15 to October 15, in better time for the administration and Legislature to make necessary budgetary changes. We also recommend the Legislature direct the project after the completion of Waves 1, 2, and 3 to identify—as part of the annual report—foreseeable challenges with the implementation of future waves, particularly Wave 4.

Consider Retention Pay Program. Although it has not been conclusively proven that pay differentials improve staff recruitment and retention on IT projects, we think the Legislature should consider directing the administration to develop a retention benefit that rewards state workers for staying at the FI$Cal project through its completion. While the state may be limited in what types of retention benefits it could extend to FI$Cal staff due to civil service rules (for example, the state likely could not provide outgoing staff hiring preferences upon the completion of the project), we think the administration could explore various retention benefits. We recommend that any retention benefit for FI$Cal staff include an assessment of the benefit program’s impact on the project’s ability to recruit and retain qualified staff. Measuring the outcomes of a retention benefit would help ensure that the benefit is set at an appropriate level and could be used to establish a best practice for future critical IT projects.

Approve 2014–15 Budget Proposal, While Understanding Inherent Project Risk. The Governor’s budget proposal reflects a reasonable funding plan to implement the updated project plan (SPR 5). We believe that the time and effort that project staff has spent in updating the project plan has reduced overall risk and strengthened FI$Cal’s likelihood of success. Nevertheless, FI$Cal involves the development of the most ambitious and complex IT system in the history of the state and significant risk remains. In its review of the Governor’s proposal and its ongoing oversight of the FI$Cal Project, the Legislature should be aware of and monitor not only the general risk inherent in all IT projects but also the shifting of some risk to the latter end of the project due to the substantial increase in Wave 4 departments. We recommend the Legislature ask the project to identify the steps it is taking to address the risks inherent in the substantial broadening of the scope of Wave 4. Ultimately, we believe that the benefits of proceeding with FI$Cal development outweigh the risk and therefore recommend approval of the Governor’s budget proposal. Should the project make significant changes going forward, a new budget proposal would be submitted for legislative review.