State’s Challenges in Implementing Information Technology (IT) Projects Have Led to Reform Efforts. The state has experienced considerable challenges realizing the potential benefits of modern technology. Recently, there have been various high profile state IT project failures that have received considerable legislative and media attention, where projects have been terminated or suspended by the Department of Technology (CalTech) before they were completed. In other cases, projects have ultimately been completed, but only after significant cost overruns and multiyear delays. Given recent failures, there has been strong interest from the Legislature and administration to reform existing processes to enhance the likelihood of IT project success. While CalTech is implementing efforts to more effectively fulfill its project approval and oversight responsibilities, the focus of this report is on the administration’s plan to give CalTech a much greater role with respect to project management.
The Legislature Required a Plan for a Centralized Project Management Office (PMO). The 2014–15 Budget Act provided funding to plan for the establishment of a centralized PMO in CalTech that would create a team of skilled project management professionals who would provide project management services to departments that are not equipped to handle this responsibility independently. According to CalTech, consolidating much of the state’s project management effort into one centralized location would make information sharing among project managers more feasible, allow IT project–related training to be applied in a more uniform fashion, and allow the staff resources utilized for project management to become experienced professionals who are capable of managing complex projects. The Legislature—while generally supportive of the concept of a centralized PMO—indicated the need for additional details on how the office would be structured and implemented before it could be established, and required that CalTech submit a plan for the office to the Legislature by January 10, 2015.
The Administration’s Plan. The administration plans for the centralized PMO to provide departments sponsoring IT projects three levels of services based on the existing capacity for managing projects within the departments—from advisory services to assuming responsibilities for day–to–day management activities. In addition to these direct services the PMO would provide sponsoring departments, the administration’s plan includes building capacity in departments themselves so that they are better equipped to manage their own projects moving forward. It proposes to do this by updating project management standards and expanding training offerings.
A Centralized PMO Has Its Benefits, but There Are Implementation Issues to Consider. We find that CalTech’s plan for a centralized PMO responds to critical issues facing state IT projects—poor capacity for managing IT projects due to inexperience and/or challenges recruiting and retaining project managers. Addressing these issues through the PMO could help the state more effectively develop and implement IT projects, provided that the PMO is appropriately structured. As regards the latter, we have identified several implementation issues for the Legislature to consider, including vague statutory authority for the PMO and a potential conflict between CalTech’s current oversight responsibilities and its project management responsibilities that would be exercised by the PMO. Additionally, setting up a PMO for success will require that deficiencies in the exercise of CalTech’s oversight role—recently identified by the State Auditor—be addressed.
LAO Recommendations. On balance, we find that sponsoring departments stand to benefit from a well–structured centralized PMO that builds a body of skilled project management professionals to serve IT projects that lack their own management capacity. We offer a series of recommended steps intended to help ensure the PMO is implemented successfully. Specifically, we recommend the Legislature (1) monitor CalTech’s efforts to improve its approval and oversight functions, (2) formally establish the PMO in statute, (3) direct CalTech to develop and report annually on performance measures, (4) direct CalTech to develop a plan for bolstering the PMO’s programmatic expertise, and (5) require CalTech to justify the need for it to retain its Consulting Division while also establishing the advisory service model for the PMO.
Technology Can Improve State Operations and Delivery of Services. Technology has the potential to improve how Californians interact with government—making this interaction more efficient, reliable, and convenient. Strides in private sector technology allow the public to access tools that improve their daily lives—from online portals that facilitate communicating with doctors to mobile applications that schedule restaurant reservations. The public has come to expect a similar level of service from government. The state is currently undertaking numerous IT projects that are intended to increase the quality of services provided to the public and improve the efficiency of state programs.
State Has Had Challenges Implementing IT Projects. The state, however, has experienced considerable challenges realizing its technology objectives effectively. While there have been some project successes, there have also been various high profile state IT project failures resulting in either project suspension or termination that have received considerable legislative and media attention. In other cases, projects have been ultimately completed, but only after significant cost overruns and multiyear delays. The factors that led to these poor project outcomes vary, as will be discussed in more detail later. One such factor is poor project management. Project management includes (1) identifying project requirements based on the sponsoring department’s business needs; (2) addressing the needs and concerns of stakeholders while planning and implementing the project; (3) ensuring parties involved in the project meet deadlines; (4) maintaining active and collaborative communication among parties involved; and (5) balancing competing project parameters and constraints, including scope, quality, schedule, budget, resources, and risk. The project management process begins during the project’s initial planning phase and ends when the project is fully implemented. There is consensus that poor project management has been a significant—although not sole—contributor to the serious problems state IT projects have recently experienced.
State Considers New Approach for Improving IT Project Outcomes. Given recent failures, there is strong administrative and legislative interest in implementing new practices that may enhance the likelihood of IT project success. While the state has considered and implemented various efforts to improve IT projects that address some of the common challenges, the focus of this report is on CalTech’s plan to improve the day–to–day management of IT projects. The 2014–15 Budget Act authorized resources to plan for the development of a centralized PMO, which would consolidate within CalTech a body of experienced project managers who would provide their services to projects throughout the state. (While not viewed as a “silver bullet” solution to address all reasons why projects have failed in the past, the effort to improve project management through a centralized PMO was seen as a significant contributor to improving IT project outcomes.) The required plan was submitted to the Legislature in January 2015. This report provides background regarding CalTech’s IT project responsibilities, discusses common project challenges, describes how projects are currently managed in the state, comments on the administration’s plan to centralize project management, and makes recommendations on how the Legislature should proceed with implementing a centralized PMO in California.
The state currently has over 40 “reportable” IT projects—projects that are approved by and under the oversight of CalTech—in various phases of development. (See the nearby box for a definition of reportable IT projects.) These projects span across departments and are intended to meet various needs of state government and the public. The total cost, should the state complete all of these IT projects as currently envisioned, is estimated to be about $4.6 billion. A small fraction of projects in development comprise a significant portion of the cost estimated to complete all projects. Specifically, the five largest IT projects alone total $2.6 billion, or 57 percent, of the total cost for all approved IT projects. The state’s largest project, the Financial Information System for California, which will replace the state’s aging and decentralized IT financial systems, is estimated to cost $673 million by the time the project is completed in 2017. With so much at stake (fiscally and otherwise) and several recent high–profile failures of state IT projects, the state has looked to CalTech—the state’s central IT organization—to determine what changes are necessary so that IT projects are successfully completed. In the following sections, we discuss CalTech’s roles and responsibilities, highlight challenges projects commonly experience, describe various options for structuring PMOs, and discuss the administration’s plan to establish a centralized statewide PMO as reflected in the January 2015 report to the Legislature. We note that the Governor’s 2015–16 budget proposal submitted in January does not reflect implementation of the plan. The related budget proposal to implement the plan will be submitted later in the spring.
Reportable projects generally face a higher level of scrutiny, as they are approved by and under the oversight of the Department of Technology (CalTech), whereas non–reportable projects are completely within the authority of sponsoring departments to manage. Projects that meet one or more of the following characteristics are reportable:
- Estimated project cost exceeds departmental delegated cost threshold authority assigned by CalTech. The delegated cost thresholds range from $200,000 to $5 million.
- Projects for which the costs are not absorbable by the sponsoring department and therefore require an appropriation.
- Projects that are specifically mandated by the Legislature.
- Projects that meet previously imposed conditions for reportability set by CalTech.
Non–reportable projects either do not meet these criteria or are sponsored by state entities outside of CalTech’s jurisdiction, such as Covered California. Projects remain reportable until the project’s sponsoring department has submitted a Post–Implementation Evaluation Report (PIER)—a report that details whether and how the project objectives were accomplished, documents lessons learned, and provides a final summary of actual versus expected costs—to CalTech. Once the PIER is submitted, the project becomes an IT system (as opposed to an IT project) operated by the state and is no longer subject to reporting.
The state has experienced considerable challenges realizing the potential benefits of modern technology. There have been various high profile state IT project failures recently that have received considerable legislative and media attention, including the State Controller’s Office’s 21st Century Project and Department of Motor Vehicles’ IT Modernization Project, which were suspended and terminated, respectively, by CalTech. (Refer to our March 19, 2014 report, The 2014–15 Budget: 21st Century Project Update, and our April 2, 2013 analysis, Information Technology Modernization (ITM) Project, for a discussion of the challenges these particular projects experienced.) In other cases, projects have ultimately been completed that serve the needs of the IT project’s sponsoring department, but only after significant cost overruns and multiyear delays.
CalTech coordinates with sponsoring departments and collects information on IT efforts across the state (for example, through the annual statewide IT Capital Plan). This provides CalTech with a statewide view of California’s overall technology needs. This statewide perspective enables CalTech to (1) identify common challenges departments experience when implementing IT projects and (2) provide opportunities for sharing best practices among departments. The factors CalTech identified that commonly lead to these poor project outcomes vary, but include such factors as:
- Inadequate definition of project requirements, which means an IT system may not ultimately meet the sponsoring department’s needs.
- Deficiencies in the drafting of contracts with vendors that inadequately protect the interests of the state.
- Ineffective preparation of organizations to transition to and operate new IT systems.
- Inadequate testing so that when the system is implemented, numerous and serious shortcomings are discovered.
These and many other challenges commonly experienced by projects largely connect back to deficiencies in a project management process that often fails to implement best practices to ensure projects remain on track and are implemented successfully.
Given these recent IT project challenges and a number of high–profile project failures, the Legislature and the administration have taken a number of actions and have proposed or are implementing new initiatives to enhance the likelihood of IT project success. Some of these relate to improvements to the project approval and oversight process, which is the responsibility of CalTech for most major state IT projects. These changes are intended to help CalTech fulfill its current responsibilities more effectively. Other responses are intended to help CalTech implement new practices that may further enhance the likelihood of IT project success, such as expanding CalTech’s role in project management through the creation of a centralized PMO, which is the focus of this report.
As noted earlier, the project management process begins during the project’s initial planning phase and ends when the project is fully implemented and operational as an IT system. Project management is typically performed by the department implementing the project. The approval and oversight process—typically carried out by CalTech—largely occurs in parallel with the project management process. These two processes are integrally related—the purpose of the oversight process is broadly to make findings and recommendations to ensure that projects are effectively managed. Project management has a role to follow through with and effectively implement the recommendations from the oversight process.
We now turn to a brief discussion of the revamped project approval and oversight process before focusing our analysis on the administration’s plan for a centralized PMO in CalTech.
CalTech is the result of three Governor’s Reorganization Plans (GRPs)—in 2005, 2009, and 2012—and related legislation that ultimately consolidated statewide IT functions into a single entity. The most recent GRP established CalTech within the newly created Government Operations Agency with a department–level head—the Director of Technology—who reports directly to the Governor on issues relating to IT. A key function of CalTech is its lead responsibilities for approving and then overseeing reportable state IT projects. (Other responsibilities include providing data center and telecommunications services to state, county, federal, and local governments throughout the state; managing IT procurement; and establishing and enforcing state–level IT plans and policies.) In order to address some of the challenges projects have experienced in the past, CalTech is revising the project approval and oversight processes. The new approval process is intended to ensure that projects are well planned and begin with a strong foundation, and its oversight process is intended to ensure that project managers use best practices to achieve successful implementation of projects.
The State Technology Approval Reform (STAR) Project. All reportable technology projects must be approved by CalTech before a department may begin work on an IT project. CalTech is in the process of implementing a new IT project approval process to replace the one that has been in use for decades. Under the former process, projects submitted a Feasibility Study Report (FSR) at the onset to CalTech that included the sponsoring department’s comprehensive approach for the project. The FSR was essentially a cost–benefit analysis of the project that justified the need for the project, evaluated alternatives, selected the preferred approach, and determined the schedule and cost. If approved, the FSR represented the state’s expected outcomes, time line, and costs for realizing the project objectives. Because the FSR was developed early in the project’s life cycle and was based on preliminary information, revisions were often necessary to the project, including schedule extensions and cost increases. When changes were needed to the approved approach, a Special Project Report would be submitted to CalTech that justified changes to the project’s scope, schedule, and/or cost.
The STAR Project intends to change the state’s IT project approval process so that projects have a stronger justification and clearer objectives, reflect appropriate solutions to identified business needs, and report more accurate cost projections and schedules. The STAR Project divides the approval process into stages, separated by “gates.” Each stage consists of a set of prescribed activities to develop deliverables used as the inputs for the next gate. The gates provide a series of “go/no go” decision points that request only the necessary and known information needed to make sound decisions for that particular point in time.
- The first stage, which requires the development of an analysis of a department’s business needs (including identifying the current IT challenges and objectives of an IT project).
- The second stage requires departments to evaluate the merits of various alternatives to meet its objectives, including business process changes and IT system development that will meet their stated business needs.
- Stage three procurement analysis and stage four solution analysis are currently being developed by CalTech. (While stage one is already implemented, the remaining stages will be phased in through December 2015.)
The new decision points (created by each gate of the project approval process) ensure that a “no” or a “go back and re–think” decision is communicated sooner rather than later through the process if the level of detail provided is inadequate or the approach is flawed. Early engagement by CalTech in its oversight role is an essential first step for project success. Improved communications early in the process will allow for course corrections before projects get too far along and problems are harder to correct.
CalTech’s Oversight Responsibilities for IT Projects. Once an IT project is approved, CalTech’s principal responsibility is to provide oversight of projects under development. The guidance CalTech provides through its oversight function is intended to assist departments’ successful implementation of IT systems. CalTech generally uses two types of independent oversight:
- Independent Verification and Validation (IV&V)—used to ensure that a system is being developed in a way that accomplishes its intended purpose. CalTech relies on consultants to provide IV&V oversight.
- Independent Project Oversight (IPO)—used to ensure that effective project management practices are in place and in use. This oversight is generally provided directly by CalTech using its own staff resources.
CalTech determines whether one or both types of oversight are appropriate and how often these oversight entities should report on the status of the project based on the complexity of the project and CalTech’s assessment of how well the sponsoring department—the department implementing the IT project—will handle the project. The most complex projects receive monthly status updates by both forms of independent oversight, while less complex projects may receive oversight less frequently.
As part of its IPO responsibility, CalTech evaluates the management of IT projects and makes associated recommendations. For example, CalTech would note when a project is ineffectively identifying risks or not adequately tracking the project schedule. Sponsoring departments act on the recommendations of IV&V and IPO to varying degrees. Sponsoring departments that do not act on the oversight entities’ recommendations may do so for various reasons. In some cases, the department may disagree with the oversight entities on the best path forward, while in other cases the project many cite constraints to acting on recommendations—such as inadequate staffing resources or insufficient time.
In a recent report on CalTech’s oversight function, the State Auditor was critical of CalTech for not utilizing a more aggressive model of oversight—a model whereby CalTech requires sponsoring departments to take remediation steps when it identifies serious and persistent issues with the project. While CalTech is taking steps to address these identified deficiencies in the project approval and oversight processes, the focus of this report is on CalTech’s plan to improve the management of IT projects through the establishment of a centralized PMO. Given this, we will now turn to a discussion of the current state IT project management structure.
Historically, the state has generally relied on individual departments to manage their own IT projects—using state staff, contractors, or a combination of the two. Under such a decentralized approach, project managers are expected to develop and implement project management processes and practices, and create a team to support the project implementation process. Because smaller departments generally undertake IT projects infrequently, these departments typically acquire staff resources for project management on an as–needed, case–by–case basis. One of the primary challenges of utilizing a decentralized project management approach can be the lack of experienced project management staff at the departmental level. Many project managers are starting from scratch—they lack experienced team members and an established governance structure to guide their working relationships. (This structure sets the responsibilities and decision–making authority among the parties involved in the IT project.) They also may have a limited understanding of project management processes and practices as they may be unable to draw from lessons learned in previous projects.
There are, however, some exceptions to this lack of experienced project management staff at the departmental level. First, some larger departments, such as the Board of Equalization and Franchise Tax Board, which often simultaneously manage multiple IT projects of high complexity, have their own PMOs. These established resources provide a body of project management expertise for use throughout their respective departments. Second, the most robust model for project management in the state is the California Health and Human Services Agency’s (CHHSA’s) Office of Systems Integration (OSI), which was established in 2005 to provide project management on health and human services–related IT projects. Some of the projects managed by OSI include the state’s automated welfare system, a case management payroll system, an electronic benefits transfer project, and a child welfare services case management system. (CalTech indicates it used OSI as a model while developing its plan for the PMO.)
While PMOs have become popular in recent years in both the private and public sectors, these offices first began to appear in the mid–1990s. Generally, PMOs are organized bodies assigned with various responsibilities related to the management of IT projects under their jurisdiction. However, PMOs vary in their structure and in the responsibilities assumed by them. In some cases, the PMO provides only project management support functions—such as developing and enforcing the use of project management processes, tools, and techniques—while in other cases, the PMO is responsible for directly managing one or multiple projects on a day–to–day basis. This fluid concept allows an organization (which could be as broad as state government itself) to structure a PMO in a way that best meets its needs and capacity.
Establishing a PMO typically involves some degree of centralization of project management responsibilities within an organization—whether it is a business or a governmental entity. Such centralization has its potential benefits and drawbacks. As to potential benefits, a PMO can result in more effectively managed projects that meet the sponsoring department’s objectives while remaining on budget and within schedule. This is because the centralized model facilitates the sharing of lessons learned and best practices, thereby allowing for the consolidation and expansion of project management expertise. As to potential drawbacks, any centralization of project management can move this function further away from the activities of the organization for which the IT system is being developed. The potential consequences of this—at least for the near term after a PMO is first established—is that project managers may lack the program area expertise that allows them to fully understand the needs of the sponsoring department. Over time, however, project managers develop program expertise. In concept, we think that the potential benefits of a centralized PMO—if properly structured—outweigh its drawbacks.
Governor Proposed to Establish Centralized PMO Within CalTech. As part of his 2014–15 budget, the Governor proposed to establish a centralized PMO within CalTech. The proposal sought two permanent positions and $280,000 (General Fund) in 2014–15 to establish the initial elements of the office, and $304,000 (General Fund) for 2015–16 to continue implementation activities, including establishing standards, tools, and guidelines, and identifying projects that require project management assistance. The proposed PMO would have brought together IT project managers into a central office within CalTech and deploy them to departments in order to manage IT projects. The proposal indicated that by centralizing project management resources, the PMO would be able to share lessons learned and avoid common project management pitfalls, thereby increasing the overall likelihood for success of state IT projects. The proposal anticipated that by 2016–17, the costs associated with the PMO would transition off the General Fund, and instead be fully supported through fees charged to departments for project management services provided through the office.
Legislature Expressed Concerns With Governor’s Proposal, Modified Proposal to Allow for Additional Planning. The Legislature—while generally supportive of the concept of a centralized PMO—indicated the need for additional details regarding several implementation issues before the office could be established. Some of the issues and concerns identified by the Legislature included: (1) CalTech’s current vague statutory authority to manage IT projects (the Governor was not proposing to amend this authority), (2) the possible conflict between CalTech’s oversight and project management responsibilities, (3) uncertainty regarding which IT projects the PMO would manage, (4) the lack of a long–term plan for OSI, and (5) concerns as to the readiness of CalTech to assume this new responsibility.
Accordingly, the 2014–15 Budget Act provided $208,000 General Fund to plan for the establishment of a centralized PMO within CalTech, modeled after OSI, that would create a centralized team of skilled project management professionals who would manage IT projects throughout the state. Two, two–year, limited–term positions were authorized and tasked with planning and establishing the framework for the centralized PMO. The Legislature also directed CalTech to report to the Legislature by January 10, 2015, regarding the implementation of the office, including information on the office’s resource requirements and a time line and transition plan for the office’s creation. Specifically, the supplemental reporting language (SRL) requested CalTech to submit a report to the Legislature that includes, but is not limited to, (1) a staffing plan, (2) a high–level schedule for when the department will initiate its project management activities, (3) the selection criteria for projects the department will recommend be serviced by the statewide PMO, and (4) a transition plan for how the department will take over a project already in development or for a newly approved project. The Legislature took this “planning” action to ensure implementation issues were adequately thought through and addressed before the office was formally established and CalTech given the resources to manage projects. We now turn to our review of the plan developed per the SRL requirement and submitted to the Legislature on January 9, 2015.
CalTech proposed moving from a decentralized to a more centralized approach for project management through the creation of a PMO within CalTech. Broadly speaking, this office would provide project management services to departments that are not equipped to handle this responsibility independently. According to CalTech, consolidating much of the state’s project management effort into one centralized location would make information sharing among project managers more feasible, allow IT project–related training to be applied in a more uniform fashion, and allow the staff resources utilized for project management to become experienced professionals who are capable of managing complex projects.
As described by CalTech, the PMO’s mission is to provide “centralized project management of IT projects so that strategic benefits are realized through standardized frameworks, education, training, and tools and techniques based on proven best practices and lessons learned.” Specific responsibilities envisioned for the PMO in the administration’s plan include: (1) managing projects for the departments that lack the capacity, (2) developing a project management framework that embraces best practices in the field of project management, and (3) developing the project management tools and training resources that can equip departments to manage their own projects successfully.
CalTech plans to establish three distinct service models for supporting IT projects, each reflecting a different level of engagement of the PMO depending on the project management capacity and needs of the sponsoring department. In all of these models, sponsoring departments retain control over and ultimate ownership of the project. We discuss each of the three models in turn.
Advisory Project Management Assistance. Under this model, a CalTech PMO project manager would work with a sponsoring department’s IT project staff on a full–time advisory basis to provide assistance on a variety of project management issues. The PMO’s responsibilities in this model would include: (1) reviewing project plans and ensuring plan completeness, correctness, and soundness; (2) identifying major risks and issues and providing advice on how to resolve them; (3) identifying areas of need and suggesting the addition of experts, advisors, or consultants; (4) ensuring that the project is following industry best practices; (5) helping to communicate between various stakeholders; and (6) providing general project management expertise. That is, the project manager from the sponsoring department would remain the project manager and maintain his/her day–to–day management responsibilities, while the CalTech project manager assumes a supporting role. This model provides the lowest level of services to departments and provides CalTech with little authority and the departments the greatest amount of discretion.
Targeted Project Management Assistance. Under this model, a CalTech project manager or associated support staff would be absorbed into the sponsoring department’s project management team. Rather than allocating a complete project management team (which includes a project manager and associated support staff, such as schedule, testing, and quality assurance managers), the PMO would allocate targeted resources based on the specific needs of the project. Ultimately, the project management team would include a combination of PMO and sponsoring department staff resources. In the case where CalTech assigns a project manager, that manager would have complete authority over all project staff resources—whether from the PMO or the department. The project manager would follow the standard processes to modify any of the project parameters, such as scope, cost, and schedule. The sponsoring department would maintain final authority over the decisions that affect the project, as prescribed in the project’s governance structure.
Full Service Project Management Assistance. Under this model, the PMO would provide a project manager and associated support staff to assume the day–to–day management of the project in lieu of the sponsoring department’s staff resources. (Sponsoring department project staff would not support project management.) The project budget and contract executing authority would be transferred to the Deputy Director of the PMO. This model provides CalTech with the greatest degree of authority and reflects a department’s inadequate project management capability. However, as with the targeted assistance model, the sponsoring department (through its project governance structure that it would set up and participate in) would maintain final authority over the decisions that affect the project.
Under the administration’s plan, departments sponsoring IT projects would have three pathways for engaging the PMO.
- Department Requests PMO Services. At any time, a sponsoring department could request the PMO’s services. Reasons for requesting services could include the lack of available personnel or project management expertise. In either case, the PMO would assign the appropriate level of services to adequately manage the IT project.
- CalTech Requires PMO Engagement as Condition of Project Approval. If the project approval process reveals that the sponsoring department lacks the capacity to manage the project, CalTech could approve the project with the condition that the department engages the services of the PMO. Once that decision has been made, a comprehensive transition plan would be developed in collaboration with the sponsoring department. The transition plan would address the shifting of responsibilities, staff, facility infrastructure, budgetary authority, reporting structure, and contracts, as necessary.
- CalTech Engages At–Risk Project. While exercising its oversight responsibilities, CalTech may determine that a project currently under development and managed by a sponsoring department is at risk and requires its intervention. In such cases, CalTech could become involved by (1) providing independent coaching and assistance to the project through its Consulting Division to help bring the project back on track or (2) engaging the PMO through one of its service models. Once the decision to engage the PMO on an at–risk project is made, a comprehensive transition plan would be developed in collaboration with the sponsoring department.
Using its authority to establish and enforce statewide IT policies and processes, CalTech has worked toward standardizing state practices around project management, IT acquisitions, and data center services, among other areas. As discussed earlier on the approval process, CalTech is now revising the standard approval practices through the STAR Project. On the project management side, the administration’s plan includes efforts to update project management standards and tools, such as how to identify staff resource needs and assess project risks, to reflect current best practices while also providing the flexibility to meet the needs of state IT projects that vary in size and complexity. (The current standards are characterized by CalTech as excessively rigid, thereby making it difficult for sponsoring departments to adapt the standards according to the needs of their own projects.) The administration also plans for the PMO to more adequately train departmental staff on how to use the new standards, thereby developing a project management competency throughout the state and making it easier for project managers familiar with the state’s management processes and practices to more easily take on management responsibility for any project across state departments.
As described previously, CalTech has oversight responsibilities of IT projects as part of its control agency functions—chiefly to evaluate the management of IT projects and make associated recommendations. Establishing the centralized PMO means CalTech would in some cases be evaluating the effectiveness of its own project management, thereby creating a potential conflict between CalTech’s current oversight responsibilities and its proposed project management responsibilities.
The department is currently organized along two main areas of responsibility: the Operations Branch, which delivers IT services (such as data center and professional development services), and the Policy Branch, which focuses on policy and oversight. Each branch is led by a Chief Deputy that reports to the Director of Technology. The plan envisions organizing the PMO within the Operations Branch of the organization. By aligning the PMO within the Operations Branch, as opposed to the Policy Branch where its IPO responsibilities lie, CalTech hopes to prevent potential conflicts.
Funding. The CalTech budget is structured so that departments pay for services they receive directly from CalTech (such as data center services) and the General Fund pays for “statewide” activities (such as CalTech administrative functions, including its legislative unit). As CalTech primarily provides services, most of the department is funded through the Technology Services Revolving Fund, which collects fees paid by customer departments for services provided by CalTech. The model would continue should the PMO be implemented as envisioned. The PMO would charge departments for project management services, while seeking General Fund support for statewide activities, such as training for CalTech staff. The Legislature would receive two budget requests for projects CalTech manages. The first budget proposal would be from the department sponsoring the IT project (as the Legislature traditionally receives), while CalTech would submit a second proposal requesting expenditure authority for the cost of providing the project management services.
Implementation. CalTech is currently finalizing the project management framework. The administration’s plan indicates that it would recruit staff resources and begin providing project management services to IT projects in 2015–16. A new budget request would be necessary to authorize the requested positions and funding in 2015–16 to begin managing projects during this period. A budget proposal for this purpose is expected later in the spring.
While we find that there are strong policy merits for moving forward with the administration’s plan for a centralized PMO in concept, we have identified several implementation issues for the Legislature to consider. We follow with our recommendations on how to respond to the administration’s plan.
Policy Merits of Proposal in Concept. CalTech asserts, and we agree, that a lack of experience and expertise of sponsoring departments in managing larger IT projects creates serious risk and can undermine a project’s success. Establishing a centralized PMO could create a body of skilled project management professionals who would concentrate project management functions in a central organization that serves IT projects throughout the state. A PMO may be of particular benefit for smaller departments or those that undertake IT projects infrequently. In such cases, institutional project management knowledge gained though previous projects is generally lost between projects. Departments undertaking IT projects of a greater complexity than they previously managed may also stand to benefit from the services of a centralized body of experienced project managers. The PMO could be better situated than departments to retain lessons from prior IT projects and enhance project management skills over time as expertise is further developed. To the extent that a centralized PMO helps to manage a project effectively and ensure best practices and standards are followed from the start, the added cost, time, and complexity of turning around a troubled project in the midst of development is potentially avoided. In addition, the administration’s plan to establish more robust project management standards and train department–based project managers to follow those standards would address the wide variability in the level of project management expertise across departments and help cultivate capacity within departments to manage their own IT projects.
CalTech’s Statutory Authority to Manage IT Projects Is Vague. CalTech asserts statutory authority for a PMO already exists, citing statute that gives the Director of Technology the duty to perform “enterprise information technology functions and services, including . . . project management activities in partnership with the owning agency or department.” Although statute broadly lays out the Director of Technology’s authority to perform various functions, including project management activities, statute does not explicitly lay out the role of a PMO or what the Director’s project management role encompasses, nor does statute explicitly provide a project management role for CalTech as a whole. As with the administration’s 2014–15 budget proposal for a PMO, the administration’s plan does not propose any statutory changes to clarify and refine the role for a centralized PMO in CalTech.
Plan Addresses Potential Conflict Between CalTech’s Oversight and Project Management Roles. The plan includes a strategy for preventing potential conflicts by creating a “firewall” between CalTech’s oversight and project management responsibilities. It attempts to do so by placing these two functions in two different branches of the department, each with its own chief deputy that reports to the Director of Technology. An alternative is to entrust project management responsibilities to another entity within state government so that these responsibilities are completely separate from CalTech’s oversight function. This approach, however, would dilute previous efforts by the Legislature to concentrate technology authority within a single agency.
In a recent report, the State Auditor identified a potential conflict between CalTech’s oversight role and the coaching its oversight analysts provide sponsoring departments. According to the Auditor, oversight analysts must be independent to ensure that they can remain objective when conducting oversight; actively coaching projects may jeopardize their objectivity. Given the potential for harm noted by the Auditor—when mixing oversight and coaching—or in this case project management—safeguards are necessary so that the new project management role is not blurred with the oversight responsibilities. On balance, we find that the proposed approach—creating a firewall between the oversight and project management responsibilities—would allow CalTech to perform its oversight functions while preserving CalTech’s consolidated technology authority.
CalTech’s Oversight Role Will Affect its PMO Role. If the PMO is to be successful, CalTech’s deficiencies in effectively fulfilling its oversight responsibilities must be addressed. According to a recent report by the State Auditor, (1) CalTech’s oversight analysts are unclear when to recommend corrective actions or project suspension/termination to their managers; (2) CalTech does not formally set expectations regarding sponsoring departments’ responses to CalTech’s exercise of its approval and oversight functions; (3) there is a potential conflict between oversight analysts’ role to oversee IT projects and their role to provide advice to agencies; (4) high turnover, insufficient state job classifications, constrained resources, and inconsistent training of staff impact CalTech’s ability to effectively oversee state IT projects; and (5) CalTech was aware of significant problems concerning particular projects but did not intervene to require the sponsoring departments to correct the problems. How well CalTech exercises its oversight functions directly affects its planned new project management role. If CalTech were able to effectively and objectively fulfill its oversight responsibilities—actively identifying project challenges and ensuring that projects take actions to course correct when necessary—projects should perform better and fewer projects would require hands–on management by the PMO. (Over time, a measure of success of the PMO would be it creating an environment where there are fewer and fewer troubled projects requiring its intensive intervention.)
Recruiting and Retaining Qualified Project Managers Will Be Key to PMO’s Success. CalTech’s ability to recruit and retain project managers and associated support staff—so that expertise is retained within the centralized office—will be critical to the ultimate success of the PMO. While the state likely will continue to have problems competing with the private sector in securing IT staff, CalTech may be better positioned than departments to recruit and retain staff resources through the PMO because of the permanent nature of the positions and a career pathway the office could offer.
We note, however, that the relative appeal of a PMO position may create additional challenges for remaining departments that need to recruit and retain their own project management staff resources. To the extent that CalTech negatively affects departments’ ability to hire qualified project managers, CalTech may need to provide project management services to more departments over time, as they are unable to recruit and retain qualified staff resources.
Pathway for Developing Program Expertise Lacking. In addition to having the technical project management expertise to successfully complete a project, the most effective project managers understand the needs of the program served by the IT project. A drawback of the centralized PMO is that its staff resources are removed from the sponsoring department so that they lack familiarity with the program area that an IT project addresses. OSI indicates that over time it has developed extensive expertise regarding health and human services programs and policy issues. This allows OSI to bridge the divide between the vendor and department staff—that is, translating between a program’s needs and the technical opportunities and limitations of the project. The PMO would have a much broader jurisdiction (program–wise) than OSI currently has—admittedly making it harder to develop extensive program expertise with respect to any particular program. The administration’s plan for the PMO does not address at all the development of program expertise. In the long run, projects could benefit from the PMO developing at least some programmatic expertise relating to the projects it manages.
Sponsoring Departments Remain Ultimately Responsible and Accountable for IT Projects. As noted, IT projects each have their own governance structure that sets responsibilities and authority for key parties involved in the project. While project managers (whether from the PMO or sponsoring department) have some degree of authority in day–to–day operations, more complex decisions are relegated to higher levels of the governance structure. Typically, the project sponsor, either an individual (department head) or a body of voting individuals (board or commission members), has ultimate responsibility and authority for critical project decisions, such as whether the system is ready to “go–live” or if the project scope should change. Projects managed by the PMO would have to operate within the governance structure of the given project. (CalTech’s oversight responsibilities include an assessment of the reasonableness of the governance structure and an ongoing evaluation of whether project decisions are made in accordance with the governance structure.) As a result, the sponsoring department would continue to have ultimate responsibility—and accountability—for IT projects managed by CalTech. Having clear lines of accountability assists the Legislature in the exercise of its oversight function.
Uncertain How Current Consulting Unit and Proposed Advisory Engagement Differ. In response to project needs, CalTech established a consulting unit that can deploy resources as needed to support projects. CalTech indicates that its Consulting Division is geared towards short–term assistance on the specific areas challenging projects, such as the development of requirements or testing of the system. The PMO’s advisory service model would provide consulting support through the duration of the project. In either case, CalTech provides assistance rather than hands–on management to the project’s sponsoring department. The services the Consulting Division provides seem similar to the services the PMO anticipates providing through its advisory service model. We are unclear as to the differences between these two sets of services so as to justify the continuation of the Consulting Division in addition to the advisory engagement model within the centralized PMO. Differences in the duration of support do not appear in and of themselves sufficient to justify the existence of two separate units, each with similar consulting roles.
Long–Term Plan for OSI Lacking. As described previously, OSI was established in 2005 to provide project management and support services to a portfolio of large, complex, and high criticality health and human services IT projects. Since its inception, OSI has developed a track record of successfully managing and deploying IT systems that support health and human services programs at the state, federal, and local levels. Informal conversations with CalTech indicate that its intent is for OSI to continue to manage health and human services IT projects, but the plan does not explicitly discuss OSI’s continued role in light of a PMO. Specifically, the plan does not indicate whether (1) CHHSA and its subordinate departments would be excluded from the PMO’s jurisdiction (and continued to be served by OSI) or (2) OSI would be eliminated, with project management services for health and human services projects offered by the PMO.
Plan Does Not Provide How Success of PMO Will Be Measured. The plan to establish the centralized PMO fails to include a strategy for measuring the performance of the office. Performance measures would allow the Legislature, sponsoring departments, and other stakeholders to determine if the PMO is meeting its objectives and could inform the potential need for refining the operations of the PMO so that it more strongly benefitted state IT projects. Without robust and rigorously tracked performance measures, the Legislature would be hindered in its ability to hold CalTech accountable for the PMO’s performance. CalTech indicates that the most important metric of success—broadly speaking—would be the delivery of completed projects that meet the sponsoring departments’ needs and expectations. While we agree that this is an important measure, assessing other aspects of the PMO’s activities would be important for a comprehensive assessment of the office’s performance.
While there are concerns regarding CalTech’s current ability to effectively oversee IT projects and some may question CalTech’s ability to take on the new project management responsibility in light of the oversight deficiencies, on balance we find that sponsoring departments stand to benefit from a well–structured centralized PMO that builds a body of skilled project management professionals to serves IT projects throughout the state. Our recommendations below provide steps to help ensure that the PMO is implemented effectively within CalTech.
Recommend the Legislature Monitor CalTech’s Efforts to Improve Approval and Oversight Functions. Given the connection between approval, effective oversight, and project management, we recommend the Legislature monitor CalTech’s progress in implementing the State Auditor’s recommendations intended to address the deficiencies in CalTech’s oversight of IT projects and modifications to the project approval process. CalTech could update the Legislature on its progress during budget or policy hearings.
Recommend the Legislature Establish, and Provide Policy Direction for, the PMO in Statute. We recommend the Legislature formally establish the centralized PMO in statute, given the vague (and unexercised) current authority in statute for CalTech to manage projects. The statute would provide the Legislature an opportunity to indicate its priorities and guide the operation of the office. We recommend that the statute codify:
- The objectives of the office. A potential objective of the PMO is to manage IT projects for departments that lack capacity. A parallel objective could be to build project management expertise among departments so that they are better equipped to manage their own IT projects.
- The circumstances under which CalTech should manage projects. To the extent the Legislature agrees with the administration’s plan, the Legislature could codify the pathways to engagement and service models (various levels of engagement) described in the plan.
- The PMO’s jurisdiction. Given OSI’s track record for successfully managing IT projects, we recommend it continue to manage health and human services projects, while CalTech’s PMO provides project management services in all other areas of state government, where necessary. We also recommend a revision to OSI’s authorizing statute distinguishing its jurisdiction from the PMO’s jurisdiction.
- The need for a strong firewall between CalTech’s oversight and project management responsibilities. We recommend codification of the proposed strategy to establish this firewall by placing these two responsibilities in two different branches of CalTech.
Require Development and Annual Reporting on Performance Measures. We recommend the Legislature require CalTech to develop performance measures that enable an assessment of how well the office is meeting its objectives. The measures should include, but not be limited to:
- The extent to which completed projects met all of the original project objectives, noting the level of engagement, if any, of the PMO for each project.
- The variance in project cost and schedule from that planned when projects were first proposed, noting the level of engagement, if any, of the PMO for each project.
- Evaluations that capture sponsoring departments’ assessments of CalTech’s performance.
We recommend that CalTech report annually on its performance during budget hearings (using performance measures to be developed) and assess at that time if changes are necessary so that the office more effectively supports departments’ project management needs and delivers IT projects successfully. In its annual report, CalTech should provide a summary, for the prior year, of (1) the demand for PMO services and the PMO’s capacity to respond to the demand, (2) the PMO’s level of engagement (that is, services provided) for each state IT project, and (3) what triggered the PMO’s engagement (if any) with respect to each state IT project.
Require Development of Strategy for Bolstering Program Expertise Over Time. We recommend the Legislature direct CalTech to develop a strategy to bolster program area expertise of the PMO so that the PMO may more effectively support IT projects. In the short term, CalTech’s strategy may largely be to rely on the sponsoring departments offering pertinent program briefings to PMO–based project managers. Over time, as the PMO’s capacity expands, the office might specialize in the development of its own program expertise, similar to OSI.
Require Justification for Existence of Both the Consulting Division and Advisory Service Model. The services offered by CalTech’s Consulting Division and the advisory service model for the PMO seem duplicative. We recommend the Legislature require CalTech to justify the need for both forms of assistance. If CalTech provides insufficient justification, we recommend the Legislature direct CalTech to consolidate the services offered by the Consulting Division and the advisory service model into a single entity.
Overall, we find that CalTech’s plan for a centralized PMO responds to a critical issues facing state IT projects—poor capacity for managing IT projects due to inexperience and/or challenges recruiting and retaining project managers. Addressing these issues through the PMO could help the state more effectively develop and implement IT projects, provided that the PMO is appropriately structured to address implementation issues that we have raised. We offer a series of recommendations intended to build safeguards to help ensure that the PMO is implemented effectively within CalTech and better positioned to succeed.