Improving Workforce Education and Training Data in California

---

Introduction

Workforce education and training programs provide job–specific training, basic skills education, and related support services to help individuals participate in civic life and the labor market. We estimate that over three million people access services funded by the state’s almost 30 workforce education and training programs annually. Though the state’s workforce system is extensive, policy makers currently cannot answer basic questions about it. For example, even getting an exact count of the people who use the state’s workforce programs is difficult. With programs administered by so many agencies and providers, data housed in so many places, data requirements varying by program, and key data elements linked only in limited situations, policy makers routinely struggle to assess whether individual workforce programs and the system overall is effective. Though the state’s workforce system has had these shortcomings for many years, recent changes in federal law and some recent state actions are requiring workforce programs to collect more standardized data in a more coordinated way. These changes signal an opportunity for the state to reconsider its approach to workforce data.

We begin this report by providing background about the state’s workforce programs and their data reporting requirements. We then provide information about standardizing performance measures and linking data across entities. Next, we evaluate how well current data collection practices provide information about programs. We conclude by making recommendations regarding the next steps the Legislature could take to improve the usefulness of workforce data in California.

Back to the Top

California’s Workforce Programs

Eight State Agencies Administer Workforce Programs. The California Community Colleges (CCC) Chancellor’s Office and the California Department of Education (CDE) are the main state–level administrators of workforce education and training programs. The California Department of Social Services (DSS), the California Employment Development Department (EDD), and the California Department of Rehabilitation (DOR) also administer large workforce programs. Three other state agencies—the California Department of Corrections and Rehabilitation (CDCR), the California Conservation Corps, and the California Prison Industry Authority—administer relatively smaller workforce programs targeted to more select populations.

California Spends More Than $6 Billion Annually on Almost 30 Workforce Programs. Figure 1 lists 29 publicly funded workforce programs currently operating in California. The Appendix contains descriptions of each of these programs. The funding amounts shown understate spending on workforce education and training because they do not include any portion of the unrestricted funding that high schools receive through the Local Control Funding Formula (LCFF). Under this formula, high schools receive a per–pupil funding rate intended to recognize the higher costs of career technical education (CTE). The state’s accounting system, however, currently does not have the capability to track LCFF spending specifically on CTE. Statewide, districts likely spend between hundreds of millions of dollars and billions of dollars on CTE. The large range is due to varying definitions of what to count as CTE instruction and support.

State and Federal Governments Provide Significant Workforce Funding. Of the $6.5 billion identified in Figure 1, 62 percent ($4 billion) is state funding. State funding primarily supports workforce education and training programs provided through CCC and CDE. The remainder of the funding is mostly federal, with a small amount from state special funds (such as the Clean Energy Job Creation Fund). Federal funding primarily supports workforce programs provided through DSS, EDD, and DOR. The largest federal workforce programs are the California Work Opportunity and Responsibility to Kids (CalWORKs) employment and training services program, supported with Temporary Assistance to Needy Families funding, and the various programs funded by the Workforce Innovation and Opportunity Act (WIOA). Some workforce programs, including CalWORKs, Offender Services, and Vocational Rehabilitation, receive funding from both the state and federal governments.

State Workforce Plan Provides Overarching Vision. The WIOA has particular importance for state workforce policy because it requires each state to prepare an overarching strategic workforce plan every four years. Federal law requires that the plan lay out a vision and goals for preparing a skilled workforce and meeting employers’ needs. The state also is required to set specific growth targets for WIOA–funded programs related to WIOA’s performance accountability measures (discussed further below). The state negotiates these targets with the federal government. Upon its development, the plan is intended to serve as California’s framework for all workforce–related policy and program decisions as well as federal and state spending decisions.

California Workforce Development Board (CWDB) Tasked With Developing the Plan. The CWDB is the state’s designated coordinating body for workforce issues and consists of 53 members appointed by the Governor. The board consists of a broad group of stakeholders, including representatives of the Legislature, business, labor, education, and corrections. The U.S. Department of Labor required states to submit their workforce plans for the 2016 through 2020 period by April 1, 2016. States were to begin transitioning to the new plans beginning July 1, 2016 and are required to fully implement the plans by July 1, 2017.

State Plan Outlines Primary Goals and Identifies Policy Strategies to Meet Goals. California’s workforce plan revolves around a few key goals. Most notably, the plan sets a goal of producing a million “middle–skill,” industry–recognized, postsecondary credentials in areas with labor market value by 2027. (The plan defines these credentials as associate degrees, certificates, and professional certifications and licenses that do not require a baccalaureate degree.) The state plan also sets a goal of doubling the number of people enrolled in apprenticeship programs. To meet these goals, the plan identifies various policy strategies, such as increasing support services to participants in workforce programs, increasing the number of “earn and learn” opportunities (including apprenticeships), and building data capacity across agencies administering workforce programs.

Back to the Top

Workforce Data

Historical State and Federal Data Reporting Requirements

State and Federal Programs Require Data on Participants and Outcomes. Most workforce education and training providers are required to submit information about the number of participants in their programs, typically including some associated demographic data, such as the participants’ age, gender, and ethnicity. In some cases, providers must collect data on whether participants have financial need and qualify for grant or loan assistance. In addition, programs often must report certain outcomes. Both state and federal programs typically require providers to report on participants’ near–term outcomes, such as program completion, test scores, and certificates or diplomas awarded. Increasingly, data requirements also include participants’ longer–term outcomes, such as subsequent employment, earnings, and enrollment in further education or training. Federal workforce programs require data on participants’ longer–term outcomes more commonly than state programs.

State Often Asks for Different Sets of Data Across State Programs. Some workforce programs with very similar goals require providers to collect somewhat different performance data. For example, CCC and CDE jointly administer the CTE Pathways Program and the Career Pathways Trust program. (Career Pathways Trust grants were awarded in 2013–14 and 2014–15 and schools can spend the funding through 2016–17.) Both programs provide funding to schools and colleges to create career pathways from high school into postsecondary education and the workforce, yet state law requires that each program report different information. State law requires CTE Pathways Program grantees to report on the wages of program graduates, whereas Career Pathways Trust grantees must report on students’ transitions related to employment, apprenticeships, and job training. Similarly, CDE administers both the California Partnership Academies (CPA) and the Specialized Secondary Programs (SSP). Each of these programs provides high schools with funding to create programs that focus on a career theme. For the CPA, the state requires grantees to annually report attendance, credits earned, test scores, grade point averages, graduation rates, and postsecondary plans. For the SSP, grantees have more flexibility in what data they annually report to the state. For example, grant recipients may choose to report either improvement in participants’ grade point averages or “other appropriate standards of achievement.”

Even for Same Types of Outcomes, Measures Can Differ. Even when programs share the same goals and require providers to track the same types of outcomes, specific outcome measures can vary. For example, programs often are required to report on their participants’ subsequent employment and wages. These measures, however, differ among programs in timing, with some measures focusing on outcomes within three or six months of program completion and others looking out three or five years. In addition, some measures consider only employment in a job related to the training a participant received, while others include any employment or wage gains. Such differences frustrate efforts to aggregate data across different programs, compare outcomes among programs, and assess the overall system’s performance. Such differences also increase administrative burdens for agencies and providers, which may need to track outcomes differently for various funders and programs.

Recent Moves Toward Common Measures

In recent years, efforts at various levels of government have sought to address the problem of collecting different outcomes for similar workforce programs. We describe these efforts below.

Federal Government Expanded the Use of Common Measures for Its Workforce Programs. Although some federal workforce programs used common data measures in the past, WIOA updated and expanded the use of common measures to all workforce programs funded under its provisions. As shown in Figure 2, these measures include skill gains, employment, and earnings.

State Working Toward Common Measures. The CWDB has committed in the state workforce plan to assist state programs in aligning their outcome measures with the WIOA measures. Separately, Chapter 13 of 2015 (AB 104, Committee on Budget) requires CDE and the CCC Chancellor’s Office to develop common measures of effectiveness for adult education programs. In a November 2015 report, the two agencies identified a minimum set of enrollment and outcome information aligned with WIOA requirements that all adult education providers must report and recommended that the state create a centralized clearinghouse to track student outcomes.

Community Colleges Using Common Measures for Workforce Programs. The Chancellor’s Office developed its own set of common measures for most CCC workforce education and training programs and implemented them beginning in 2014–15. The Chancellor’s Office recently aligned several of these measures with the WIOA common performance measures and is in the process of modifying more of its measures to align with WIOA measures.

California’s Workforce Data Collection and Reporting Systems

Each Agency Typically Has One Major Data System. State agencies that operate workforce programs each have one primary data system to which providers report information about program participants. For example, at CDE, the majority of student data is collected in the California Longitudinal Pupil Achievement Data System (CALPADS). Similarly, CCC’s primary data system is the Chancellor’s Office Management Information System (COMIS), DOR’s primary data system is Accessible Web–based Activity Reporting Environment (AWARE), and CDCR’s primary data system is the Strategic Offender Management System (SOMS). Each of these systems compiles data from numerous local service providers into a statewide database.

In Addition, Ad Hoc Data Systems Proliferate. Rather than adding new data elements to their primary systems, which can be costly and time consuming, agencies sometimes create ad hoc data systems to fulfill new reporting requirements as they implement new programs. This practice is especially common for programs with unique data requirements, limited–term programs, and programs with tight reporting deadlines. For example, for the 12 workforce education and training programs CDE administers either alone or jointly, it uses at least six separate internal data systems to collect and store performance data. Moreover, in 2015–16, the first year of the Adult Education Block Grant, the CCC and CDE used a temporary data system to collect performance information from adult education providers in order to comply with their annual reporting requirement to the Legislature and administration.

Some Agencies Display Selected Data Publicly. Agencies sometime provide aggregate information about their program participants to the public. For example, the CCC Chancellor’s Office makes publicly available some aggregate information on students, course taking, and student outcomes through its online Data Mart, Student Success Scorecard, Salary Surfer, and Wage Tracker tools. Likewise, CDE makes publicly available some aggregate information about students, course taking, and short–term outcomes on its online DataQuest and Ed–data.org tools.

Back to the Top

Data Linking

The Value of Linked Data

Data Linking Enables Providers and Policy Makers to Track Participation Across Programs. Data linking refers to matching participant records from two or more programs or agencies and using those linked records to track participation and outcomes. In the workforce area, this process is essential for illuminating which programs are working well. For example, linked data can show if students who complete high school–equivalency programs at adult schools successfully transition to community colleges or other training programs. Linked data also can show if students who begin a program in one agency complete a similar program in another agency (perhaps due to relocating). Data linking also can identify gaps and duplication of services.

Data Linking Provides Information About Longer–Term Impacts of Programs. In addition to facilitating tracking across programs, data linking enables policy makers to track participant outcomes over time. As noted earlier, state and federal programs increasingly require information about program participants’ eventual employment, wage gains, and enrollment in further education. Linking data across state agencies over certain periods of time can provide this information. For example, to learn how many students continued their education and training and/or went to work in a particular industry, a high school CTE program could link its student information with CCC and EDD data for periods ranging from one semester to several years after high school graduation.

Federal Workforce Act Requires Data Linking Across Some State Agencies. As part of WIOA implementation, the federal government is requiring states to establish an integrated data system to which WIOA–funded programs would contribute participant and outcome data. The purpose of the requirement is to improve state agencies’ ability to (1) coordinate service delivery across programs and (2) record outcomes across programs in a single report.

Examples of Data Linking

Various entities already are linking data across workforce education and training providers to track program participation and outcomes.

Some National, Regional, and State–Level Data Linking. At the national level, the U.S. Department of Education has developed the College Scorecard, which links data from colleges and universities with federal student financial aid and tax data. For a given college, the Scorecard provides information about admissions, student demographics, completions, costs (by family income), student aid (including student loan repayment rates), and the average earnings of students after attending the college. At the regional level, the Western Interstate Commission for Higher Education is piloting the Multistate Longitudinal Data Exchange, which links several western states’ higher education and employment data to provide information about student outcomes across state lines. At the state level, several states routinely link K–12 education, postsecondary education, and employment and wage data through longitudinal systems to provide information about students’ progression through the educational system and post–graduation outcomes. Florida, Texas, and Virginia have among the most comprehensive of these systems.

Linked Data in California. Although WIOA requires states to develop a comprehensive approach to linking data, California does not have a linked education or workforce data system. Various entities, however, have made efforts to link information to track program participation and outcomes over time. These efforts include Cal–PASS Plus, created and funded through the CCC Chancellor’s Office, which links data for some K–12 and higher education agencies with each other and with workforce outcome data. In addition, the state tasked the CWDB in 2014 to produce a tool known as the Workforce Metrics Dashboard, which shows wage and employment outcomes for certain workforce programs. (The Dashboard likely will be available in late 2016.) Some state agencies also are linking data. For example, the CCC Chancellor’s Office has developed the state–level CCC Salary Surfer and the college–level College Wage Tracker, both of which use linked CCC and EDD data to make information on median wages by program of study publicly available. At the provider level, the Bay Area Community College Consortium, consisting of 28 community colleges in the region, is exploring the development of a data system that would link information about students enrolled in the region’s community colleges and adult education programs.

Three Models for Linking Data

Entities typically link participant data to measure longer–term outcomes using one of three models. Figure 3 shows these models.

---

---

Agency–to–Agency Model. Under this model, two agencies enter into a memorandum of understanding to share selected data for specified purposes on a one–time or recurring basis. Participating agencies determine the terms and conditions, as well as the mechanics, for linking their data. Additionally, the memorandum of understanding specifies who can access the data and for what purposes it will be used. In California, data linking for workforce education and training programs occurs only through agency–to–agency agreements. As the top of Figure 3 shows, CCC, CDCR, DOR, and DSS each have entered into separate agency–to–agency data sharing agreements with EDD to collect information about their participants’ wage gains. Notably, CDE has not linked its data with EDD records. In California, many other agency–to–agency agreements relate to the sharing of many other elements of workforce data.

Central Repository Model. Under a central repository model, agencies send their data to a central agency, such as a higher education coordinating board, which maintains a data repository or “warehouse.” As the middle section of Figure 3 shows, once an agency submits its data, the central agency links it to other data in the warehouse. The central agency (and sometimes other authorized agencies) can then query and use the data. In 1999, the Legislature directed the California Postsecondary Education Commission to develop a central repository that linked data across state education institutions. This repository, which included some data going back to the late 1980s, operated until 2011. Many other states—including Florida, Texas, and Washington—currently use this model to link their education and workforce data.

Federated Model. Under a federated model, sometimes called an “integrated data system,” agencies maintain their data in–house and permit other agencies access to some of that data through a shared interface (see the bottom section of Figure 3). Using automated processes, the shared interface can link data from multiple agencies upon the request of one or more agencies and with the approval of all affected agencies. While this model is newer and (to date) less common than the central repository model, several states, including Virginia and Illinois, use it to link their data.

Other Considerations When Linking Data

Regardless of the data–linking model entities use, issues arise about how to identify participants and protect their privacy. We describe these considerations in more detail below.

Common Identifiers Facilitate Data Linking. A key step in linking data is identifying individuals who appear in two or more data sets. This requires a common individual identifier that agencies can use to match the records. For example, CCC and EDD both document Social Security numbers in their records and can use these numbers to identify individuals who both (1) attended a community college program and (2) earned reportable income following completion of the program. Use of Social Security numbers is not universal. State law does not allow schools to collect these numbers for K–12 pupils, complicating CDE’s ability to link records with CCC and EDD. Similar issues arise for programs that serve undocumented immigrants.

Workarounds Used When Common Identifiers Are Not Available. Agencies have developed two workarounds for reporting employment outcomes in the absence of a common identifier. One method is to survey former participants about their longer–term outcomes. This survey data tends to yield information that is less reliable than official wage data and typically is used by agencies that do not link data through any of the three models described above. The other workaround is to use “fuzzy matching.” Fuzzy matching uses several personal characteristics, such as name, date of birth, gender, and address to link information about an individual across separate data systems. As the name implies, fuzzy matching is not as precise as a common identifier, but it commonly is understood to be the next best option, providing a reasonably valid and reliable way for linking data. Agencies that use any of the data–linking models described above often use this method to identify participants when common identifiers are not available.

Agencies Must Address Privacy and Security Concerns When Sharing Data. State and federal laws allow sharing of education data across agencies for legitimate educational and research purposes. These laws also require agencies to safeguard participant privacy. To comply with legal requirements, agencies sharing data with each other typically develop formal data sharing agreements to establish (1) rules about how and to whom data can be disclosed and (2) steps agencies will take to prevent unauthorized access to the data. We discuss these issues in more detail in the nearby box.

Back to the Top

Assessment

Despite having made some progress, the state still lacks data measures that are applied systemwide and it still does not have a coordinated data–linking approach. These two key shortcomings hinder policy makers’ ability to examine performance across California’s workforce education and training system. We discuss these issues in more detail below.

Recent Progress Toward Common Measures, but Work Remains

Some Programs Use Common Measures . . . When fully implemented, the federal requirement that common measures be applied to all WIOA–funded workforce programs will improve the usefulness of the state’s workforce data, allowing policy makers to examine participation and outcome data consistently across those programs. Efforts to establish common measures for certain state–funded programs will further enhance the usefulness of California’s workforce data. Similarly, recent efforts to use common adult education outcome measures across all providers is resulting in more useful data.

. . . But State Still Does Not Apply Common Measures to All Programs. The state’s WIOA–funded programs, CalWORKs employment and training program, CCC workforce programs, and various other state programs (such as the CTE Pathways Program and the Career Pathways Trust) still do not all share a set of common measures.

Current Method of Linking Data Falls Short

Current Efforts to Link Data Inefficient and Burdensome. The state’s current linking model, the agency–to–agency model, is a fragmented and time–consuming method to link data across agencies. It also is not comprehensive, as not all relevant agencies link with all other agencies serving the same participants. Moreover, the model is inefficient both administratively and technically. It requires that each agency’s legal counsel negotiate individual agreements with every other participating agency. In addition, the model often requires an agency to develop a unique technical approach for working with every other participating agency because data are stored in different formats. Similarly, the surveys some agencies have developed to report outcomes in the absence of linked data are relatively burdensome and unreliable. In some cases, programs do not report any outcome data, despite reporting requirements. The nearby box provides an example of a longstanding program that has been unable to report longer–term outcomes due to lack of linked data.

Systematically Linked Data Could Improve Decision Making. Policy makers often must rely on anecdotal evidence to make decisions because programs are limited in the outcomes they can reliably report without linked data. By contrast, the availability of systematically linked data could provide information about the types and volume of workforce education services currently provided across the state; the number of distinct individuals receiving these services; patterns of movement for participants across these services; short–term and long–term outcomes of programs, including participants’ employment and wage gains at various intervals following service delivery; and alignment of workforce education services with regional economic needs. Common and linked data also could permit service providers to compare participants’ outcomes across programs and examine factors that contribute to program success.

Other Linking Models Offer Notable Advantages Over Current Method

Both Central Repository and Federated Models Better Than Agency–to–Agency Model. Both the central repository and federated models would be more efficient than the agency–to–agency model because data sharing would be governed by a comprehensive master agreement rather than the numerous separate agreements that each agency otherwise would have to negotiate. Moreover, both the central repository and federated models would provide more accurate and complete information about participant outcomes because data linking would be much more systematic than under the agency–to–agency model.

Central Repository Model Has Certain Advantages Over Federated Model . . . On the one hand, the central repository model has been used by many states for many years and uses proven technology. It also may yield more reliable data than the federated model because the central agency typically reviews and validates the data submitted by participating agencies to the repository. In addition, data queries under this model tend to be conducted more quickly, as responding to them does not involve pulling new data from several agencies.

. . . But Federated Model Might Be More Suitable Today, and in California Context. On the other hand, the federated model has become more common in recent years due to certain advances in technology and information science. Largely as a result of these advances, the federated model improves data security by avoiding creation of a separate, large, linked, continuously maintained data repository for which security breaches are particularly serious. In the states currently using the federated model, the model also appears to have cultivated greater acceptance by participating agencies due to avoiding conflicts regarding control and stewardship of data. This is because data are not stored centrally or long term under the federated model. In California, an additional advantage of this model could be greater acceptance by policy makers who might be reluctant to create a state agency to oversee a central, longitudinal education data repository.

Back to the Top

Recommendations

To help policy makers, providers, current and prospective participants, and the public gain access to linked, longitudinal data that would help them make better decisions regarding workforce education and training programs, we recommend (1) finalizing and applying common performance measures for all of the state’s workforce education programs; (2) establishing a statewide structure for linking data across agencies to determine programs’ longer–term outcomes; and (3) improving how data are used by communicating results, incorporating information into policy and budget processes, and facilitating policy research. We discuss each of these recommendations in more detail below.

Develop and Use Common Measures

Convene Group to Establish Common Performance Measures. We recommend the Legislature direct the CWDB, which includes representation from all major workforce providers, to lead a task force that would resolve remaining inconsistencies among performance measures for the state’s workforce programs. We anticipate that the resulting measures would include the new federal WIOA measures plus a small number of additional common measures that reflect other state priorities. The task force’s work should include the development of detailed data definitions, reporting schedules, and other specifications needed to ensure that agencies collect and report data to state and federal authorities in a consistent and efficient way. To ensure broad stakeholder input and buy–in, we recommend the Legislature direct the group to consult with workforce and economic development officials, employers, and other agencies that administer workforce programs as well as provide a public comment period.

Require Programs to Use the Measures. Following identification of statewide, common performance measures for workforce education programs, we recommend the Legislature amend state law, as needed, to align data reporting requirements with the common measures. Legislation also could clarify to which programs the common measures will apply by offering a clear definition of workforce education and training programs.

Streamline Data Linking Among Agencies

Replace Agency–to–Agency Agreements With a Systemwide Model. We recommend the Legislature direct the CWDB to study and report on potential solutions to developing a statewide, streamlined data–linking approach for all workforce programs. The CWDB already has begun working with various agencies to explore such a system, so we believe the task force could report back to the Legislature by July 2017. At that point, the Legislature could weigh the trade–offs of potential solutions. Regardless of the preferred solution, the system could be developed through collaboration among the participating agencies. We recommend the Legislature require workforce education and training programs, as a condition of receiving state funding, to participate in the new system, as applicable, to provide specified performance data.

Use Data to Inform State Budget and Policy Decisions

Once the state implements common measures and a streamlined data–linking model, the Legislature can start using the resulting information to inform its workforce funding and policy decisions.

Require Transparent, Periodic Performance Reporting. Regardless of the model the state adopts to link workforce data, we recommend tasking the CWDB with creating a small number of standardized reports that will communicate results clearly to policy makers, providers, participants, and the general public. Several states, for example, have implemented online performance dashboards to help users find and explore performance data. Beyond state policy makers, local providers could use the data to better guide program development, and interested individuals could use the reports to choose workforce education and training programs that are suitable for them.

Incorporate Information Into Budget and Policy Process. We recommend the Legislature annually review statewide workforce education performance data as part of its budget and policy processes. Specifically, the relevant legislative committees could ask state agencies administering workforce programs to discuss their performance results in hearings. This would give committees the opportunity to ask agencies questions about their programs’ effectiveness and describe their plans for improvement.

Require Access to Data for Policy Research. We recommend directing workforce education agencies to develop and adopt a uniform protocol by which researchers could gain access to data. Such a protocol should contain appropriate safeguards for participant privacy and confidentiality. By conducting third–party analysis of workforce data, researchers could help inform state policy and fiscal decisions, thereby increasing the value of the data collected.

Back to the Top

Conclusion

The state’s current hodgepodge of workforce data requirements is cumbersome for agencies to manage and not very useful for informing state policy. Our recommendations to implement common measures and a statewide approach to data linking would reduce the administrative burden for state agencies that often struggle to meet disparate reporting requirements for their programs, negotiate cumbersome data–sharing agreements, and use labor–intensive surveys and other data collection strategies to compensate for lack of outcome data that could be available from other agencies. Taken together, we believe that our recommendations for improving the alignment, integration, and accessibility of workforce data would provide more useful information to help guide state policy and funding decisions and ultimately result in better workforce education outcomes for participants and the state.

---

Appendix