The 2017-18 Budget

California State Payroll System

---

Background

Recent Unsuccessful Attempt to Replace Legacy Payroll Systems. The SCO is responsible for issuing pay to the state workforce, including employees of the state, CSU, and judicial council. In 2004, the SCO proposed the Twenty‑First Century (TFC) project, the information technology (IT) replacement for its existing human resources management and payroll systems. In February 2013, after the project experienced various problems during its pilot stage, the SCO terminated its contract with the project’s primary vendor, SAP Public Services, Inc. (SAP). Since then, the SCO has entered into a settlement agreement with SAP for its lawsuit related to the TFC project, resolved the various payroll errors produced by the system while it was in use during the pilot, and closed out the project. For a full history of the TFC project through the settlement agreement in June 2016, please see the Appendix.

Still May Be Necessary to Update the State’s Payroll System. In the 2016‑17 budget, the SCO received funding for eight positions to restart the replacement of the state’s payroll system (now renamed the California State Payroll System, or CSPS). When the TFC project was first proposed in 2004, the SCO justified the project by noting there were functionality and stability issues that warranted pursuit of an updated payroll system. For example, the SCO believed there would be ongoing challenges with maintenance as the legacy systems are technically challenging and labor‑intensive to update. They also noted that a new system would reduce the likelihood of significant payroll disruptions in future years. However, the department has now suggested that the primary reason to replace the state’s payroll system is to improve functionality, not to address potential future stability issues. For example, functionality improvements in a new system would allow the SCO to respond more quickly to payroll changes, issue reports to other state agencies and stakeholders, and allow employees online access to payroll and tax information.

New State IT Project Approval Process Aims to Improve Success of State IT Projects. Historically, when departments proposed IT projects, the California Department of Technology (CDT) required them to prepare Feasibility Study Reports (FSRs). Various shortcomings with the FSR approval process meant that projects often experienced challenges once they were underway. These challenges were frequently associated with significant cost increases and schedule extensions for IT projects. The CDT has begun implementing a new IT project approval process—known as the Project Approval Lifecycle (PAL)—with the goal of helping to bolster project planning and reduce the likelihood of project challenges or failure.

PAL Divides Approval Process Into Four Stages. The stages are: Stage 1 business analysis, Stage 2 alternatives analysis, Stage 3 procurement analysis, and Stage 4 bid analysis and finalization of project details. Each stage: (1) requires sponsoring departments to conduct specific planning‑related analyses and submit an associated planning document to CDT, and (2) provides CDT with a discrete decision point in its approval process. Departments cannot begin their projects without receiving approval from CDT for each of the four stages. (Please refer to our February 2017 report, The 2017‑18 Budget: The New IT Project Approval and Funding Process, to learn more about the PAL process.)

Governor’s Proposal

Budget Request

Governor’s Budget Proposes $3 Million to Begin Stage 2 of the PAL. The Governor’s 2017‑18 Budget includes nearly $3 million to begin Stage 2 of the PAL process and continue internal efforts to replace the state’s payroll system. (The SCO anticipates it will complete Stage 1 of PAL in April 2017 using currently authorized resources.) This proposal includes $1.1 million in one‑year limited‑term funding to support 11 positions (in addition to the 8 positions authorized in the 2016‑17 budget package). These positions would provide information on existing payroll policies and procedures, provide technical expertise on the legacy payroll systems, and respond to the PAL documentation requirements.

Proposal Includes Funding for Consulting Services. The proposal also includes $1.8 million in 2017‑18 for consulting and professional services. These services include an interagency agreement with the Department of Human Resources (CalHR) to ensure the new system appropriately applies legal and regulatory requirements and supports statewide human resources and labor relations policies and practices. The SCO also includes external consulting services to: (1) help the state rethink its payroll process so they are more efficient (known as business process re‑engineering), (2) improve its project management, and (3) support a procurement support vendor.

Tentative Proposed Timeline for Completion of PAL. With these (and other previously authorized) resources, SCO anticipates it will complete Stage 1 of the PAL process in April 2017. The SCO plans to complete Stage 2 by August 2018, Stage 3 by May 2019, and Stage 4 by December 2019 (with the resources requested in this proposal and additional requests in future years). The SCO anticipates project work would begin in 2020‑21.

Initial Alternatives Under Consideration

SCO Has Identified an Initial List of Alternatives to Consider. The SCO has identified the initial list of alternatives it will evaluate during Stage 2 of the PAL process. Below, we summarize these alternatives:

• Modular Approach Toward Centralized Payroll System. Largely as a means of reducing IT project risk, departments across the state are beginning to consider developing and deploying their IT projects in modules rather than all at once, which has been the traditional approach. The modular approach works by breaking projects down into multiple discrete units of functionality, prioritizing them, and then deploying them over relatively short periods of time. In the case of the CSPS, modules would likely focus on discrete aspects of the state’s payrolling processes, such as benefits administration or deductions. Collectively, the modules would create a centralized payroll system for the state.
• Decentralized Payroll Systems. Payroll practices can vary significantly across state departments, and some departments—such as the Department of Forestry and Fire Protection (CalFire) and the Department of Corrections and Rehabilitation (CDCR)—have particularly complex payroll needs that complicate the state’s ability to integrate these systems. (Employees in these departments, for example, work unusual hours or receive variable compensation based on a variety of conditions.) This alternative would assess updating payroll systems in a decentralized fashion that integrates less complex payroll departments together and considers alternative approaches for modernizing the payroll systems of complex departments.
• Using Commercial Off‑the‑Shelf (COTS) Software. This alternative would assess modifying existing software—known as COTS software—as necessary to address the state’s payroll needs. The SCO would contract with a vendor to make the necessary software modifications. The SCO anticipates evaluating the viability of several COTS applications, including SAP software and the Financial Information System for California’s (FI$Cal’s) People Soft application. The TFC project also relied on SAP COTS software and contracted with SAP to modify the software for use by the state. FI$Cal is a state IT project that, when fully built, would integrate the state’s financial management systems in the areas of budgeting, accounting, cash management, and procurement.
• Internally Custom‑Built System. Rather than rely on existing software and vendors, this alternative would assess building a custom system primarily using SCO employees. Under this approach the state would not have to conform its payroll process to accommodate the particular functions of a COTS software. According to SCO, this would position it to effectively maintain and operate the system without the reliance of a vendor. This could allow the SCO to relatively easily modify the system as payroll practices change, such as when a new memorandum of understanding (MOU) is ratified between the state and bargaining units.
• Implementing a New Front‑End System on Top of Existing Technology. The existing payroll systems rely on mainframe technology. This alternative would assess building a new “front‑end” system (the interface a user sees) using modern technology while continuing to rely on the current mainframe technology in the “back‑end” (the code and software that support the system). Despite SCO’s concerns with the stability of the existing systems when the TFC project was first proposed in 2004, SCO now believes the existing mainframe technology is stable and reliable well into the future. However, SCO has identified some challenges with hiring employees with the technical expertise to maintain and operate mainframe technology.
• Initiating “Software as a Service.” This alternative would assess establishing a subscription for an online application to process the state’s payroll. This option is similar to the COTS alternative. However, a third‑party would own and operate the software, rather than state.

The Legislature should note that these alternatives are not mutually exclusive. For example, the SCO could evaluate using a COTS application to modernize the payroll systems for less complex departments. Meanwhile, to address the needs of departments with complex payroll processes, the SCO could evaluate using a modular approach.

LAO Comments

State IT Projects Are Expensive and Carry a Great Deal of Risk. The state has a history of significant challenges in successfully implementing IT projects. In some cases, projects have experienced significant cost overruns and multiyear delays. In other cases, the IT projects have failed altogether, resulting in either project suspension or termination and receiving significant legislative and media attention. While the PAL should increase the quality of up‑front planning and result in more accurate cost and schedule estimates than the prior FSR process, it does not ultimately guarantee success.

Replacing Payroll System Will Be Challenging . . . The CSPS would be a huge undertaking. The state’s human resource management and payroll systems are very complex and updating those systems (even if it is through a decentralized or modular approach) carries significant risk. As we have noted, some departments, like CalFire and CDCR, have particularly complex payroll needs that complicate the state’s ability to integrate these systems. The state’s payroll needs are also continually evolving as new MOUs institute different compensation schemes. As such, a new payroll system may be met with significant, and unforeseen, challenges. While we do think there is merit to the state pursuing an updated payroll system, we would caution the Legislature that the risks common to IT projects are particularly acute in the case of CSPS. As a result, substantial delays and cost overruns are even more likely than usual.

. . . But Rigorous Planning May Result in a More Favorable Outcome. While unforeseen challenges surely will arise, we are cautiously optimistic about the SCO’s current effort to replace the state’s payroll system. In particular, the new PAL process, while still relatively untested, has the potential to reduce the likelihood of project challenges or failure through a more rigorous planning process. Through the PAL, the SCO is currently collaborating with CDT and, through this year’s proposal, would formally institute its ongoing collaboration with CalHR. The SCO’s initial list of alternatives suggests it is exploring a wide range of possible project types. These developments are encouraging.

State Should Give Alternative Selection Careful Consideration. The alternative selected at the end of Stage 2 by SCO and CDT will have significant consequences for the future of the state’s payroll processes. Once an alternative is selected, it could be years or decades before the state would implement a different system. (We would note the SCO proposed the TFC project in 2004, over 12 years ago.) The appropriate selection of a project alternative will be critical to the success of the future project.

No Concerns With the Governor’s Current Proposal. At this point, we have no concerns with the Governor’s proposal to provide one‑year limited‑term funding to the SCO for the PAL alternatives analysis. We agree it makes sense for the state to assess the potential risks, both in terms of functionality and stability, to the state’s payroll system and explore potential solutions to these problems.

Recommend SCO Report to the Legislature With Analysis and Preferred Alternative. The PAL process presents the Legislature with an early opportunity to weigh in on its own priorities for IT projects. However, the SCO’s anticipated timeline for completing the PAL does not neatly align with the budget cycle. In particular, the SCO anticipates it will complete Stage 2 and move on to Stage 3 (which initiates the procurement process based on the solution selected in Stage 2) in August 2018, just after the adoption of the 2018‑19 state budget. If so, the Legislature may not have an opportunity to evaluate SCO’s alternatives analysis and express its preferences in the selection of a project alternative. We recommend the Legislature require SCO to present its full findings from the Stage 2 alternatives analysis, alongside its preferred alternative, before it receives funding for Stage 3. This would provide the Legislature with a clear opportunity to provide meaningful input and weigh in with its own priorities for the CSPS before a solution is procured.

Appendix

History of the TFC Project

Modernization of State’s Human Resources and Payroll Management Systems. In 2004, the State Controller’s Office (SCO) proposed the Twenty‑First Century (TFC) Project, the information technology (IT) effort to replace the existing statewide human resources management and payroll systems used to pay approximately 260,000 state employees. The new system, also called MyCalPAYS, was intended to allow the state to improve management processes such as payroll, benefits administration, and timekeeping and include self‑service access for employees and managers, among other capabilities. The SCO noted that the existing systems, commonly referred to as “legacy systems,” were developed more than 30 years ago and are inflexible, fragmented, and costly to maintain. In 2005, the Legislature approved the project with an estimated total cost of $130 million with full implementation scheduled for July 2009.

Two‑Phase Procurement. In conjunction with state IT oversight officials, SCO decided to pursue a two‑phase, or “unbundled,” procurement approach. This meant the state sought two vendors and undertook two procurements. The first vendor would supply the software package, and the second vendor (the primary vendor) would integrate the software to the state’s business requirements. In April 2005, SAP Public Services, Inc. (SAP) was selected to supply the software package. The system integrator contract was awarded to BearingPoint in June 2006.

Early Issues Delayed Project Development. During 2006 and 2007, SCO asserted that multiple problems had emerged with the work of BearingPoint. The vendor asserted that issues with the software package and with SCO caused delays. In October 2007, following multiple schedule delays, SCO issued a breach‑of‑contract notice to BearingPoint. The vendor and SCO then reached a plan to address project failures and integration continued. These delays extended the schedule by two years and raised estimated total costs to about $180 million.

Vendor Contract Terminated. After several months, BearingPoint once again fell behind schedule, unable to complete project activities and provide deliverables on time. With the project’s schedule and development in jeopardy, the Department of General Services (DGS) issued a default notice to the vendor on December 3, 2008. The notice stated that the vendor failed to: (1) properly manage the project, (2) complete designs in a timely manner, and (3) make progress toward development. On January 6, 2009, SCO formally terminated the contract, and primary work on the TFC project stopped.

Strategy to Move Project Forward Developed. Following the termination of the primary vendor contract, SCO developed a new strategy. In particular, the project scope was narrowed by excluding California State University (CSU) from the project. The CSU has different payroll requirements from those applicable to state civil service employees. The legacy system would continue to process payroll for CSU until a revised system for CSU employees was completed as a separate project. The SCO also decided to select a new system integrator using a two‑stage procurement approach, discussed below. The new strategy was documented in Special Project Report (SPR) 3.

New Contract Procured. In March 2009, DGS released a request for proposal for a new system integrator. The procurement was conducted as a two‑stage procurement approach. Stage I was the selection of contractors to evaluate the work completed to date and its possible reuse, and to better understand the requirements of the project. Accenture, LLP, and SAP were selected in Stage I. Both companies submitted Stage II proposals, which detailed the approach, cost, and schedule for completing the project. In February 2010, SAP was awarded the contract, at which point the project costs and schedule were revised. The project schedule was extended to October 2012, and estimated total costs rose to $283 million. Implementation was to occur in five phases—known as pilots and waves—where Pilots 1 and 2 would bring a small number of employees into the new system in order to test it prior to Waves 3, 4, and 5, which would fully implement the system in three large and roughly proportional stages. (SPR 4 documented these changes.)

Subsequent challenges occurred when the project began converting data from the legacy system to the new system. Project management issued a cure notice to the primary vendor, which then subcontracted with a data migration vendor, BackOffice Associates, to improve the data conversion process. SPR 5 accounted for these delays, increased the estimated total project costs to $373 million, and extended the final wave of the project, Wave 5, by one year—from October 2012 to September 2013.

Pilot 1 Test. On June 11, 2012, the first major test of the state’s new payroll system took place. The test, known as Pilot 1, produced payroll, benefits, timekeeping, and position management activities for about 1,500 SCO employees. This pilot program tested the new system’s functionality with a small number of employees in an effort to identify and correct potential problems before expanding the number of employees covered by the new system.

Problems Encountered During Pilot 1. Although SCO expected minor discrepancies during Pilot 1, significant errors surfaced during the first payroll cycle and persisted through each of the subsequent seven monthly payroll cycles. In particular, incorrect paycheck deductions were made, payroll and pension wages were erroneously calculated, and medical benefits were denied for some employees and their dependents. In one case, employees that took vacation time during the payroll cycle received compensation in addition to their base salary. Attempts to correct these errors created further problems in the following payroll cycle. In early August, project staff determined that the severity of these issues warranted the delay of Pilot 2, an expansion of the new system to 15,000 employees across numerous departments. Pilot 2 was initially delayed from September 2012 to March 2013. As a result of continued challenges, discussed below, Pilot 2 was not implemented.

Cure Notice Issued to SAP. On October 25, 2012, SCO issued a cure notice to the system integrator, SAP, expressing serious concern regarding SAP’s ability to successfully implement the new system. According to SCO, SAP’s lack of expertise and strategic planning lead to inadequate scheduling, staffing, knowledge transfer, deliverable management, and quality assurance. The SCO also identified concerns regarding design, testing, organizational change management, and training weaknesses. In total, the cure notice cited 13 grievances and prompted SAP to correct these problems by November 30, 2012, so that the project could move forward.

SAP Responds to Notice. The SAP submitted a response to the cure notice on November 30, 2012. In its response, SAP did not assume responsibility for the grievances outlined by SCO and took no action to resolve the issues.

Contract Terminated in Early 2013. On February 8, 2013, SCO terminated its vendor contract with SAP, citing inaction regarding issues listed in the cure notice and a lack of confidence in the vendor to implement the project successfully. At the same time, the Technology Agency—now the California Department of Technology (CDT)—suspended further work on the project until a new plan could be established.

State Payroll Reverted to Legacy Systems. Beginning in March 2013, SCO returned the payroll processing for its Pilot 1 employees to the existing legacy systems. According to project staff, SCO began running parallel payrolls on the legacy systems in order to identify inaccuracies and ensure that no pay or benefits discrepancies were left unresolved. As a result of this precaution, SCO indicated the return to the legacy payroll systems for these employees should not pose a problem.

Lawsuits Filed. In November 2013, the SCO filed a lawsuit against SAP for breach of contract, seeking to recover payments made to SAP for system integration costs prior to project termination. (The primary vendor payments made to SAP totaled $50 million of the $90 million contract.) SAP later filed counter‑claims against the SCO for damages and declaratory relief in the amount of $23 million.

SCO Corrects System Errors Produced During Pilot 1. In 2014, SCO took on a “Do the Math” reconciliation effort to identify and correct the errors produced during Pilot 1. Of the 1,542 SCO employees that participated in Pilot 1, the SCO corrected errors for 267 employees who were underpaid and 541 employees who were overpaid. In addition, the SCO completed reconciliations for the 100 interfaces that connect with other payroll system payees, such as healthcare providers, retirement savings systems, and tax agencies.

Lawsuits Settled. The terms of a June 6, 2016, settlement agreement resulted in SAP paying the SCO $59 million and abandoning its own claims against the SCO. The $59 million settlement was distributed to the state’s General Fund and special funds, using the same proportions established to support the TFC project. Figure A1 shows the timeline of major events from the start of the project in 2004 through the close of the project at the end of 2016.

System/Software Assessment Released. In late 2015, Grant Thornton, a contractor hired on behalf of CDT, released the results of an assessment of the TFC system and software. The assessment detailed which system requirements were satisfied, determined which of the delivered products could be leveraged to complete the project, and estimated the cost for a vendor to complete the system. CDT had planned to complete two additional assessments of the TFC project—a project management assessment and an alternatives assessment. However, these were later rolled into the Project Approval Lifecycle (PAL), the state’s new IT project approval process.

Post‑Implementation Evaluation Report (PIER) Completed. In November 2016, the SCO submitted a PIER to the CDT. The PIER includes a detailed history of the TFC project, compares the project objectives to the actual project outcomes, documents the failures and successes experienced on the project, and describes the corrective actions the SCO will take to improve the chances of future success. The submittal of the PIER represents the official end of the TFC project. For SCO, it also represents the start of a renewed effort to update the state’s human resources management and payroll systems. In December 2016, the SCO named the renewed effort the California State Payroll System.

---