Legislative Analyst's Office Analysis of the 2001-02 Budget Bill

K-12 Databases

The Governor's budget proposes funding for three projects to collect and analyze student-level data. The budget provides $16.5 million to continue a multiyear project to develop an integrated student level record system at each district in the state, and $25 million for two new student-data proposals. The new proposals are:

• $20 million for Academic Performance Index (API) data analysis.
• $5 million for the API Student Tracking System.

The policy goals of the two Governor's initiative are (1) to improve the use of student achievement data in state and local decision making, and (2) to support improvements to the API. To date little information has been provided by the administration on either of these proposals.

California School Information Services

We recommend the Legislature augment the Fiscal Crisis Management Assistance Team (FCMAT) item (6110-107-0001) by $800,000 (Proposition 98 General Fund) to continue implementation the California School Information Services (CSIS) project. We further recommend augmenting the FCMAT budget (Item 6110-101-0349) by up to $11.5 million (Proposition 98 General Fund) to provide additional CSIS local implementation grants to school districts.

The California School Information Services (CSIS) is a multiyear project to develop, implement, and manage a statewide student-level database and information-transfer network. The program is administered by the Fiscal Crisis and Management Assistance Team (FCMAT), a part of the Kern County Office of Education. The FCMAT, under a contract with the state, provides fiscal advice, management assistance, and training to school districts. Although FCMAT administers CSIS, the State Department of Education (SDE) administers local assistance grants to FCMAT and is integrally involved in advising on the student-level database. The SDE also has a direct role in CSIS implementation by supporting the transition of state data reporting to electronic submission.

The CSIS has three main program goals:

• Build local capacity to maintain and use student information systems to inform educational decision making.
• Enable school districts and school sites to electronically transfer student records (such as transcripts, test scores, and health records) to any other district or school in the state. Transfer student transcripts to institutions of higher education.
• Simplify and increase the accuracy of district data reported to the state.

Meeting these goals would provide the state with several benefits, as summarized in Figure 1.

The Proposed Budget for CSIS

The Governor's budget proposes $12 million from one-time, prior-year Proposition 98 monies for CSIS implementation grants to school districts, and $4.5 million for support of the FCMAT operation costs. Figure 2 provides a budget summary for CSIS. Based upon concerns about inadequate state oversight of CSIS and potential risks inherent in large database projects, the Legislature provided funding to SDE for ongoing independent project oversight in the current year. Funding for the independent evaluator is continued in the Governor's budget.

Figure 2

CSIS Budget Summary

2000-01 Budget Act

Funding for FCMAT a

$8.6 million audit recovery funds provided for implementation grants to Local Education Agencies. To the extent that audit recovery funding does not materialize, the $8.6 million is backfilled with Proposition 98 reversion funds. $4.2 million one-time Proposition 98 reversion funds to FCMAT to support California School Information Service (CSIS) operation costs. $250,000 Proposition 98 for project management services.

Funding for Department of Education (SDE)

$1.2 million to address workload in supporting CSIS. Department of Finance (DOF) and Department of Information Technology (DOIT) must approve feasibility study report prior to the release of funding. $150,000 to continue to contract for independent project oversight, with quarterly reports to Legislature and administration. Requires DOF and DOIT to approve scope of work.

2001-02 Budget Proposal

Funding for FCMAT

$12 million audit recovery funds provided for implementation grants to LEAs. To the extent that audit recovery funding does not materialize, the $12 million is backfilled with Proposition 98 Reversion Funds. $4.5 million Proposition 98 to FCMAT to support CSIS operation costs and project management services.

Funding for SDE

$500,000 provided for SDE workload in support of CSIS. $150,000 to continue to contract for independent project oversight, with quarterly reports to Legislature and administration.

a Fiscal Crisis and Management Assistance Team.

Current Status of CSIS Implementation

Currently CSIS is in the second year of a multiyear implementation plan. Over the last year, FCMAT has (1) developed a long-term implementation and budget plan for CSIS, (2) continued to establish the infrastructure of CSIS to administer the transfer of data among school districts and to the state, and (3) increased the number of local education agencies (LEAs)—school districts and county offices of education—participating in CSIS from 70 LEAs to 161 LEAs. Instead of mandating participation in CSIS, FCMAT is attempting to entice school districts to participate with the benefits of a student-level database (outlined above), and implementation incentive grants.

As of 2000-01, FCMAT has provided incentive grants (or is in final negotiations) to nine consortia of districts representing over 1.5 million students. The nine consortia are listed in Figure 3. Each consortium is overseen by either a school district or a county office of education, and is working with a specific software vendor to develop its local system to be able to transfer student records to CSIS. The CSIS will not maintain or store the student records for districts, but will facilitate either the movement of records from school district to school district, or the aggregation of school district data required for state reporting.

Each consortium plans to expand to other school districts using their specific software. For example, NCS Pearson, the vendor for the largest consortium, owns a software product called Schools Administrative Student Information Software (SASI). The SASI software is used in over 280 school districts in California containing approximately 2.7 million students. Each year the consortium overseen by the Alameda County Office of Education, plans to increase the number of SASI users participating in CSIS.

The CSIS Long-Run Budget Requirements

The FCMAT has provided a detailed estimate of the costs of fully implementing CSIS statewide. Based upon the funding needs for school districts in the first two phases of the CSIS implementation, FCMAT has developed a funding formula to fund the remainder of the state's school districts. School districts would receive implementation funding of $8.51 per student plus $2,500 per school site. The FCMAT has also developed a small school district formula. Based on these various formulas, FCMAT has developed the expenditure plan shown in Figure 4. Over the eight years of the project, CSIS would have total costs of approximately $120 million, including $11 million for contingencies (such as potential enrollment increases, charter schools, and other uncertainties).

Governor's Funding Level Will Slow Implementation. The Governor's proposed funding level for local implementation grants is approximately one-half of the funding request made by FCMAT. The Governor's budget provides $1.1 million less for CSIS operation costs. Based upon the budget level, the CSIS project completion date will likely need to be delayed at least one fiscal year. The CSIS implementation plan is in part based upon the expansion plans provided by the nine consortia. In 2001-02, the CSIS proposed plan would add 258 LEAs to the project with combined enrollment of 1.85 million students, which would double the size of the program. With a reduced appropriation, the number of districts in the CSIS expansion plan will be revised downward.

Los Angeles Unified School District (LAUSD) Seeks New Student Information System. Currently, LAUSD is reviewing proposals to develop a state-of-the art integrated student information system. The LAUSD will be selecting the vendor to design and implement its new system in the spring of 2001. The participation of LAUSD in CSIS is essential to the successful completion of the CSIS mission of implementing a statewide student-level data system. Also, LAUSD participation would significantly enhance the short-term benefits of records transfer among school districts in southern California.

According to staff at LAUSD, one of their design goals is to become a CSIS participant at the same time that they develop this new database system. Based upon the CSIS funding formula, the costs of the local implementation grant for LAUSD could be around $7.5 million. The LAUSD is seeking proposals to implement the new data system over the next three years. It is uncertain whether FCMAT would provide the entire CSIS funding in one year or spread the funding over several years for LAUSD or other large districts. In any case, if LAUSD receives an implementation grant in the budget year, there will be little funding left for other school districts.

We believe that FCMAT is prepared to expand the number of districts that are participating in CSIS in line with its spending plan. As shown in Figure 1, CSIS benefits the state by supporting the API and evaluation, and school districts by improving student tracking, providing more accurate and timely student records transfer, and tracking student drop-outs. The CSIS implementation plan provides adequate detail on how the project would be implemented and the related long-term costs. Accordingly, we recommend augmenting the FCMAT support item by $800,000 and the local implementation grant funding by up to $11.5 million (both Proposition 98 funds). This funding level is consistent with the CSIS spending plan for 2001-02.

Potential CSIS Funding Expansion

We recommend that the Fiscal Crisis Management Assistance Team report at budget hearings on the potential values and costs of offering certain school districts a California School Information Services readiness assessment.

Many school districts are not prepared to participate in CSIS. Applying for a CSIS implementation grant requires a significant amount of preparation by the district, and many districts do not have the time or the resources to study what needs to be accomplished to prepare for CSIS. The approach used to incorporate a school district into the system is to adapt the student information system that a school district already has. The worse the state of the current student information system, the more the school district will have to do prior to participation in CSIS. Some school districts have a long way to go to be ready for such participation. In addition, many have no local capacity to assess what changes need to be made prior to applying for participation in CSIS. Helping these less prepared school districts assess their data information needs and develop a multiyear plan to become CSIS ready would help ensure the CSIS system is delivered on time. School districts could benefit from external assistance in becoming CSIS ready.

The FCMAT has developed professional standards to assess and develop recovery plans for struggling districts throughout the state. We believe that FCMAT has the capacity to develop a set of professional standards for assessing the readiness of school districts to adequately maintain and use student data, and eventually participate in CSIS. One way this could be accomplished is by providing funding to FCMAT to offer CSIS readiness assessments for school districts and develop multiyear implementation plans for those districts. We recommend that FCMAT report at budget hearings on the potential values and costs of offering school districts a CSIS readiness assessment.

Will Apportionment Rewrite Be Compatible With CSIS Design?

We recommend the State Department of Education report at budget hearings on efforts it is taking to ensure the compatibility of the Principal Apportionments System Rewrite to receive California School Information Services data.

The Governor's budget provides SDE with $1.7 million to continue the Principal Apportionments System Rewrite (PASR). The PASR is the state's distribution process through which SDE allocates $7 billion in major categorical funding to school districts based upon student enrollment, student average daily attendance (ADA), and other factors. The method used to distribute categorical funds historically was through a set of independent data processes (including outdated programs maintained at the Teale Data Center), and 27 newer categorical programs developed on stand-alone personal computers. The PASR is an ongoing project to integrate these programs into one central process. Two of the major inputs into the apportionment system are enrollment and ADA.

In the future, CSIS will be the main provider of data on enrollment and ADA to the SDE. Based on our review it is uncertain if the rewrite will be able to accept data from CSIS. Since both projects represent major technology investments by the state, the Legislature will want to ensure that the systems will be compatible in the future. We recommend the SDE report at budget hearings on efforts they are taking to ensure the compatibility of PASR to receive CSIS data.

Department of Finance and Department of Information Technology Delay External Reports

We recommend the Legislature increase non-Proposition 98 General Fund from $150,000 up to $400,000 for the State Department of Education to contract for external oversight of the California School Information Services (CSIS) project (Item 6110-001-0001). We recommend the Department of Information Technology and the Department of Finance report at budget hearings on their delay in approving the scope of work for external oversight of CSIS.

The Governor's budget proposes $150,000 in funding to SDE to contract with an independent project consultant to oversee the FCMAT implementation of CSIS. Generally, state departments are subject to Department of Information Technology (DOIT) and Department of Finance Technology and Investment Review Unit (DOF-TIRU) oversight in efforts to develop large data systems. The DOIT and DOF-TIRU have no oversight role in CSIS because CSIS is administered by FCMAT, housed at a county office of education.

To address this apparent lack of oversight, the Legislature appropriated funding in the 1999-00 Budget Act to SDE to contract for quarterly reporting from an independent project oversight consultant. In the 2000-01 Budget Act, the administration proposed—and the Legislature adopted—budget bill language to continue the quarterly reporting and to require SDE to have the scope of work of the external oversight contractor approved by DOIT and DOF-TIRU. The SDE delivered the scope of work on August 1, 2000 to DOIT and DOF-TIRU for review. The DOF-TIRU approved the scope of work in December 2000, and DOIT approved it in February 2001. The first two quarterly reports required by the 2000-01 Budget Act were not submitted, and given the time requirements of contracting, it is unlikely that the third quarterly report will be delivered by the March 1, 2001 deadline. Hopefully, at least one report can be delivered in the fiscal year.

We recommend that DOIT and DOF report at budget hearings on why they have delayed approval of the scope of work for the external oversight contract.

As the program grows, so will the amount of oversight that an external consultant will have to review. The Governor's budget provides a modest increase in funding for CSIS, but provides no expansion of the funding for oversight. As the CSIS program grows, the size and number of projects the oversight consultant must oversee grows as well. We recommend increasing funding for the external oversight to provide more complete information to the Legislature upon which it can ensure that the project will be successful. The increase is especially needed if the Legislature adopts our earlier recommendation to increase the CSIS funding level above the Governor's amount. If the Governor's funding level of $16.5 million is maintained, we recommend increasing the funding level provided to SDE to contract for oversight from $150,000 to $250,000 to keep the amount of oversight in line with the size of the project and to ensure that the Legislature has high quality information upon which to measure the progress of the project. If the Legislature adopts our prior recommendation to increase CSIS funding up to $29 million, we recommend the Legislature provide $400,000 for external oversight.

Increase Funding for the SDE to Adequately Support the CSIS Program

We recommend that the Legislature provide $1 million to the State Department of Education (SDE) for staff and services that will allow the SDE to complete tasks that are necessary for it to support the California School Information Services (CSIS) program. These tasks would allow local education agencies to transmit data to the SDE through the CSIS program, thereby reducing school districts' state data reporting workload. (Increase Item 6110-001-0001 by $500,000.)

As discussed above, CSIS has three program goals (1) build school district's data capacity, (2) support student records transfer, and (3) facilitate state and federal reporting. Meeting the last of these stated goals requires the CSIS program to work with the SDE to design a process whereby data collected from LEAs by the CSIS program is transmitted to the SDE in lieu of the SDE's regular reporting requirements.

Background. The SDE brings expertise and knowledge about the data that LEAs must report. The department completes the following tasks to support the CSIS program for each set of data collected by the system:

• Transition Data Collections. This requires the SDE to specify how data should be organized from CSIS so that it can be merged with and replace traditional SDE data collections.
• Validation Rules. The SDE must develop criteria for evaluating the quality of data that is transmitted from CSIS.
• Test Data Comparability. Once data is submitted to SDE from CSIS, SDE must have a process in place to check that the data meets specific quality standards.
• Certification of Data. If the data meets or exceeds quality standards, then SDE must certify that CSIS submission is a suitable replacement for traditional data collection methods. Once an LEA is certified to submit data to SDE through CSIS, the LEA no longer needs to submit data through the traditional process.

Working with the CSIS Advisory Group, LEAs, and other stakeholders, SDE selected 40 data collections that it considered suitable for reporting by the CSIS program. The 2000-01 Budget Act provided the SDE with $1.2 million, conditional on the approval of a feasibility study report (FSR) by the DOIT and DOF-TIRU, to support tasks to begin planning and implementing a process to accept data from CSIS during the current year for five of the 40 identified data collections.

As discussed earlier, SDE submitted an FSR to DOIT and DOF-TIRU for review in August 2000. The SDE has revised the FSR to address concerns expressed by DOIT and DOF-TIRU, and anticipates that around $500,000 of the $1.2 million will be released to support work to transition the collection of data from traditional data collection methods to CSIS data. This lower level of funding will support SDE's efforts to prepare five data collections for CSIS data submission in the current year, but will delay planning efforts for additional data collections.

The SDE Needs Additional Resources to Facilitate State Reporting. Increasing the number of SDE data collections that can accept data from CSIS participating LEAs is critical to the long-term success of CSIS. Many of the current LEAs participating in the CSIS program have joined the program anticipating a reduction in their SDE reporting workload. In addition, prospective CSIS participating districts may elect to forego participation in the system if SDE is unprepared to accept data from the CSIS program.

The Governor's budget proposes $500,000 for SDE to support the CSIS program. If the SDE is to expand the number of state data collections transmitted by CSIS, additional funding is required. We recommend that the Legislature increase the amount of funding provided to SDE to $1 million, an augmentation of $500,000. This funding would allow SDE to expand its data transition and collection preparation work from five data collections to up to 20 additional data collections. During budget hearings, SDE should provide a proposal for how these funds could be allocated. In addition, SDE should also provide information about how such funding could affect its ability to support the CSIS program.

The API Student-Level Database May Have Great Potential

We withhold recommendation on the $5 million Academic Performance Index (API) database proposal because of the significant number of unanswered questions about how the API database will operate. To the extent that the Legislature appropriates funds for this project, we recommend the funding be provided to the State Department of Education (SDE) and not the Office of the Secretary for Education because SDE is the administrative agency in charge of student assessments and accountability. (Withhold recommendation on Item 0650-115-0001.)

The Governor's budget provides $5 million from the General Fund (Proposition 98) to the Office of the Secretary for Education (OSE) to create the API database. The API database would be a student-level database consisting of data provided by the state's major assessments—starting with the data from the Standardized Testing and Reporting (STAR) program. The API database would not be necessary if the CSIS database were fully operational for all school districts in the state. However, until CSIS is fully operational, the API database would provide some of the benefits that CSIS would provide, but on an accelerated time schedule. The API database would be created to eventually be folded into CSIS, and would use unique CSIS student identifiers to track students over time.

There are four potential benefits of this proposed API database:

• Allow for Calculation of Value-Added Measures of Student Achievement. A value-added measure of achievement determines the improvement that each individual student makes from one year to the next. Education experts suggest using a value-added measure of achievement for accountability purposes is preferable to using schoolwide averages. California's current API is calculated using an averaging approach.
• Provide Schools Valuable Longitudinal Information. Many school districts are not able to track student achievement from one year to the next. In the fall, a teacher often has no student assessment information from prior years. Even in school districts with well integrated information systems, the district may receive no assessment information for students coming from another district. The API database could potentially provide such assessment information.
• Allow for Higher-Quality Evaluation of State and Local Programs. Having a longitudinal database would help researchers evaluate the impact on student achievement of various state and local programs.
• Help School Districts Transition to CSIS. Providing school districts with accurate information on students could show school districts the value of a quality student information system. If the proposal provides school districts with a subset of the CSIS data elements, the burden of CSIS start-up could be reduced.

What Is Known About the Proposal. At the time this analysis was prepared, the following information was available about the proposal:

• Information in the "STAR Header." The data set that would be collected for each student would contain the information collected from data provided on the top of the STAR answer keys commonly referred to as the STAR header. The information includes mainly demographic information—including race/ethnicity, gender, language status, length of time in school district, economic disadvantaged status, and special education status. In addition, the school would be required to include a unique CSIS identifier in the STAR header. This data would be linked with the test results to provide the data for the database.
• The CSIS Identifier Would Link Records Over Time. The CSIS identifier is a number assigned by CSIS to each individual student in the state. Once the CSIS identifier was provided on the student's STAR header, CSIS would be able to track a student's results based upon the unique identifier. Each year, the same CSIS identifier would be used for each individual student, and would link the information from one year to the next.
• Nonpersonally Identifiable Data Would Be Provided to the SDE. Similar to the current program, the SDE would receive records of student performance on state assessments in a nonpersonally identifiable format.

Unanswered Questions. Beyond the basic framework of what student information would be collected and how it would be linked over time, there are numerous unanswered questions.

• Where would the data be stored and maintained—the test publisher, CSIS, SDE, another entity?
• Who would have access to the data—SDE, home school district, researchers, policymakers?
• How would security of individual student information be guaranteed?
• How would schools or CSIS track the unique CSIS identifiers from year to year for students that move from one school district to another?
• Given that CSIS would eventually provide all the functionality of the API database, are the benefits of having this longitudinal tracking data available a few years earlier worth the costs?
• What would the role of the test publisher be, and how would those duties transition if the publisher changes when STAR is reauthorized (see discussion of STAR reauthorization in "Assessments" section of this chapter)?

As discussed earlier, the API database has a lot of potential benefits. However, given the number of unanswered questions, we withhold recommendation, pending receipt and review of a more detailed implementation plan.

The SDE Is the Appropriate Agency to Oversee Project. The SDE will have to be fundamentally involved in this proposal to create an API database because it:

• Administers the STAR test that will provide the data for this database.
• Administers the contracts with the test publisher that will need to be amended to have the publisher provide information to a database.
• Is the end user of the database in the calculation of the API.
• Will likely be involved in correcting data irregularities.

In contrast, OSE's main state role is as a policy advisor to the Governor on education issues. Given the need to have SDE intimately involved with this proposal, we recommend that any funding that the Legislature provides for the API database be provided to SDE, not OSE.

The STAR Data Analysis

We recommend redirecting $20 million Proposition 98 funds provided for the Standardized Testing and Reporting data analysis incentive grants to other legislative priorities. (Reduce Item 6110-113-0001 by $20 million.)

The Governor's budget provides $20 million to school districts to purchase or develop software to aid in analyzing STAR test data. According to the Governor's budget, such software would help schools determine how students are performing academically, thereby allowing schools to most effectively allocate resources to assist students in mastering the academic content standards. The administration has not been able to provide additional back-up on the proposal.

The lack of information raises several questions about how the funding will be used:

• What information would these software packages supply that is not already available to schools?
• How would the funding be distributed and would it cover the costs of software programs currently available?
• Do these software packages require that the district have a functional student information system?
• Would these programs concentrate on the Stanford-9 or the Standards-Aligned STAR, or both? What about other assessments?
• Since the Standards-Aligned STAR is so new, are there software products available to analyze that portion of the test?
• Would this funding be one-time or ongoing? Do these programs have ongoing licencing costs?

Proposal Is Poorly Timed. Even if the administration is able to answer these questions, there would still be reasons to delay the implementation of this proposal.

• The API Database Should Come First. As discussed above, the Governor has proposed the creation of an API database to collect longitudinal student assessment data. According to the OSE, the earliest that the API database would be functional would be summer 2002. Presumably, whatever software a school district would purchase or develop would benefit from that database being operational. The school districts would want to ensure that any software product they purchased or developed could take advantage of this new data system.
• The STAR Reauthorization Could Affect Use of Software. As discussed in the "Assessments" section of this chapter, the Legislature will be considering the future of the state's assessment system this year since the current system will sunset January 1, 2002. Now that Standards-Aligned STAR is operational, the Legislature may want to rethink the use of the nationally norm referenced part of the STAR program. Until the Legislature makes a decision about reauthorization, providing incentives for school districts to purchase or develop software to analyze Stanford-9 test data may be an ineffective use of funding.
• Dependence Upon a Functioning Student Information System. Many school districts do not have student information systems. Additionally, many of the school districts that do have systems continue to struggle to maintain those systems. To the extent that data analysis software is dependent upon having a functional student information system, this proposal may benefit from more school districts having first implemented CSIS.

Because of all of the unanswered questions about this proposal, and the potential timing conflicts with the creation of the API database and the reauthorization of STAR, we recommend the Legislature redirect funding provided for the STAR data analysis to other legislative priorities.