Legislative Analyst's Office Analysis of the 2001-02 Budget Bill

School Accountability

The Public Schools Accountability Act (PSAA) (Chapter 3x , Statutes of 1999, [SB 1x, Alpert]) created a statewide school-level accountability system which (1) rewards schools for academic improvement, (2) provides external assistance for lower-performing schools, and (3) potentially sanctions schools which continue to fail after receiving external assistance. The Academic Performance Index (API) is the cornerstone of the PSAA. Its purpose is to measure a school's academic performance and the growth in performance over time.

Recently, the second annual set of API scores was released, allowing the first API growth scores to be calculated. Based on the API growth data, the State Department of Education (SDE) will be distributing $677 million in rewards to schools, teachers, and other school personnel in the current year. Figure 1 shows the various rewards programs tied to API results.

As shown in Figure 2, the budget also proposes significant resources for various programs targeted at low-performing schools, based upon the API. As Figures 1 and 2 illustrate, the state is allocating a significant amount of money—almost $1.7 billion in 1999-00 through 2001-02—for programs using this index. The box nearby provides some background information on how the API is calculated and how the measure is used.

The API Is a Work in Progress

The state's accountability system is still a work in progress. The API is currently based on only one measure, namely the Stanford-9 test results. The Stanford-9 is an "off the shelf" student assessment produced by Harcourt Brace and administered to California students in grades 2 through 11. The PSAA requires that the API include at least five additional measures: graduation rates, student attendance rate, teacher attendance rate, Standards-Aligned Standardized Testing and Recording (STAR), and the High School Exit Exam (HSEE). The Legislature included this broad set of outcome measures to ensure that increases in the API were based upon real gains in achievement and not temporary fluctuations in a single measure.

The SDE has not included graduation rates, student attendance nor teacher attendance in the API because it is currently not able to collect accurate school-level data on these outcome measures. The SDE has not included Standards-Aligned STAR and HSEE results because the assessments are not yet fully operational. While California's current system is a long way from the full implementation, many efforts are under way to improve the quality of the information used in calculating a school's API (see shaded box on Improving the API).

 Problems With the Stanford-9

The Stanford-9 provides California schools with a score reflecting how students in California perform relative to a national norm created from the results of administering the same test to a nationally "representative" group of students. We see three major problems with the Stanford-9:

• The Stanford-9 is not aligned to California's academic content standards.
• Experts question whether Stanford-9 test scores for many of the state's English Language Learners (ELLs) are meaningful.
• The test questions do not change from one year to the next.

We discuss each of these problems in more detail below.

Stanford-9 Not Aligned to Standards. The current version of the API is solely dependent on the Stanford-9, an assessment tool that is not aligned to the state's academic content standards. The California Academic Content Standards adopted by the State Board of Education (SBE) define what the state wants students to learn in each grade and subject (see shaded box for actions related to the academic content standards). Since the Stanford-9 tests a different set of information than the content standards, the state is sending conflicting messages about what schools should be teaching. Should teachers and administrators focus instruction on areas covered by the Stanford-9? The state is offering bonuses that can exceed $25,000 per individual for success measured by that test. Or should they teach what is expected under the academic content standards for which the state has invested $1 billion for new textbooks, the Governor is proposing $335 million for staff development, and upon which the HSEE is based?

  How Meaningful Are Stanford-9 Scores for ELLs? The second problem with the Stanford-9 is the validity of Stanford-9 scores for many ELL students. The advisory committee of experts appointed by the SBE for development of the API advised the board to exclude Stanford-9 test scores for ELL students from the API calculation. Their recommendation was based upon analysis of Stanford-9 test results for ELL students. A school may be doing an excellent job of helping a student transition to English fluency, but that student's Stanford-9 score may not reflect the improvements the student has made. The SBE did not follow the recommendation. Although not implemented by the SBE, the advisory committee's recommendation puts into question the comparability of API scores between schools with many ELL students and schools with few or no ELL students.

Test Questions Don't Change From Year to Year. The final problem with the Stanford-9 is that the set of actual questions on the test does not change from year to year. Particularly with a high stakes test, it is important to vary test questions from year to year in order to minimize possibilities for literal "teaching to the test" and outright cheating. The SDE did not stipulate the rotation of questions from year to year in its contract with the publisher of the Stanford-9. So for the five-year duration of the contract—which expires after the administration of the spring 2002 test—the test given in each grade is exactly the same in each of its administrations. Teachers know the exact questions that their students will be asked and upon which their school will be evaluated. Providing $677 million in rewards based solely upon this test creates incentives for schools to misuse their knowledge of Stanford-9 test questions.

Reduce Funding for Governor's Performance Awards

We recommend the Legislature reduce the funding proposed for the Governor's Performance Awards by $219 million until the Academic Performance Index is based upon a broader set of reliable performance indicators. (Reduce Item 6110-123-0001 by $219 million.)

Funding Too High for Untested Measure. As discussed above, the API is a work in progress and will take several years to develop fully. The administration is making progress on increasing the number of outcome measures used to calculate the API, and is planning to switch to a value-added approach, which will improve the use of the existing achievement measure. Until then, however, there are numerous problems with the Stanford-9 being used as the sole measure for the API as discussed above.

Consequently, we believe it is not only premature to attach such high stakes (and such large funding amounts) to the API but it may be counterproductive to improving learning outcomes for California's public school students by redirecting the focus of teaching from the state's broad content standards to the set of material tested on the Stanford-9. Once the Governor and SDE have been able to develop the data collection systems and assessment tools needed to support a quality accountability system, the Legislature then should determine the appropriate level of funding needed for school incentives. Accordingly, we recommend reducing funding for the Governor's Performance Awards to $131 million, the same level appropriated in the 2000-01 Budget Act. This recommendation would create an ongoing $219 million of Proposition 98 savings which the Legislature could direct more effectively to other K-12 education needs.
(Reduce Item 6110-123-0001 by $219 million.)

Inadequate Funding to Evaluate PSAA

We recommend the Legislature (1) increase funding to the State Department of Education (SDE) to contract for an evaluation of the Public Schools Accountability Act from $250,000 to $500,000, and (2) reappropriate $250,000 included in Item 6110-001-0001 of the 2000-01 Budget Act for the same purpose. We further recommend SDE report at budget hearings on the possibility of using federal funds for this evaluation.

The PSAA requires SDE to contract for an external evaluation of the PSAA—including the Governor's Performance Awards program, Immediate Intervention/Underperforming Schools Program (II/USP), and the impact of these programs on student achievement. The 2000-01 Budget Act provided SDE with $250,000 for the evaluation. In the summer of 2000, the SDE released a Request for Proposal for the evaluation but did not receive a single qualified bid. As a result, the state has not begun to evaluate this major program. The Governor proposes an additional $250,000 from the General Fund (non-Proposition 98) for the PSAA evaluation this year.

We believe that the $500,000 that would be appropriated between the two budgets is still inadequate given the complexity of issues that the evaluation must address. For example, under this evaluation, the SDE is required to contract for a comprehensive evaluation of the implementation, impact, costs, and benefits of II/USP and Governor's Performance Awards program. A preliminary report is due March 31, 2002. The state may distribute as much as $1.7 billion related to these programs in 1999-00 through 2001-02. Potentially more important, as early as fall 2002, schools and employees of schools participating in the II/USP could face sanctions from the state under the terms of the PSAA. Sanctions could include the Superintendent of Public Instruction reassigning principals, reorganizing schools, reassigning certificated staff, contracting out the management of schools or even closing schools. The state needs to make sure that the measure upon which such high-stakes decisions are made truly tracks changes in student achievement. Given the importance of this program and the extensive list of legislative requirements for the evaluation, we recommend that the Legislature (1) increase the appropriation for the evaluation from $250,000 to $500,000 in the budget year, and (2) reappropriate $250,000 included in the 2000-01 Budget Act for this purpose. (This would make $750,000 available to the study.)

As discussed below, the II/USP is partially funded with federal funds. The federal grant which provides funding for II/USP allows the state to use up to 5 percent of the grant amount to pay the costs of administering the program. We believe that evaluation of the program is an essential part of the program's administration. To the extent feasible, we believe it is preferable to use federal funds instead of the General Fund to pay for at least part of the PSAA evaluation. Federal funds could not be used for all of the evaluation costs because the PSAA evaluation must evaluate programs beyond II/USP. We recommend SDE report at budget hearings on the feasibility of using federal funds to pay for at least part of the PSAA evaluation.

Certificated Staff Performance Incentives Overfunded

We recommend that the Legislature reduce funding for the Certificated Staff Performance Incentive Awards by $50 million on a one-time basis to take advantage of carryover funding available for this program. (Reduce Item 6110-133-0001 by $50 million.)

Chapter 52, Statutes of 1999 (AB 1114, Steinberg), created the Certificated Staff Performance Incentive Act to provide $5,000 to $25,000 rewards to certificated staff at schools meeting two criteria:

• The school is in the bottom half of the API.
• The school "significantly" exceeds its annual API growth target.

Figure 3 shows the appropriations and expenditures that have been made or are proposed for this program. The 1999-00 Budget Act provided $50 million for the program. No funds were distributed in 1999-00, however, because only one year of API scores was available prior to summer 2000, therefore making it impossible to determine API growth targets. The 2000-01 Budget Act and the 2001-02 Governor's Budget each include $100 million for this program. The administration's spending plan for 2000-01 is to use the $50 million from the 1999-00 Budget Act and $50 million from the 2000-01 Budget Act. As a result, $50 million from the 2000-01 Budget Act will still be available for rewards in the budget year. We recommend the Legislature use this $50 million in carryover funds for the certificated staff incentives and reduce Item 6110-133-0001 by $50 million. This action would provide the total of $100 million for budget-year expenditures proposed for this program, yet also free up $50 million in Proposition 98 funds for other K-14 education priorities of a one-time nature.

Immediate Intervention/Underperforming Schools Program

We recommend the Legislature enact legislation to broaden the eligibility criteria for the Immediate Intervention/Underperforming School Program to include all schools in the lowest two deciles of the Academic Performance Index.

The II/USP provides state and federal support for school-wide reform efforts at low-performing schools. The 2001-02 Governor's Budget proposes $113.8 million from the General Fund (Proposition 98) for the program. The budget would provide planning grants for a third cohort of 430 schools, and implementation grants for the first and second cohorts (782 schools in total). Figure 4 show the schools and number of students in each cohort for 2001-02. The Governor's budget continues to fund 78 schools with federal funds, which we discuss below.

In their first year in the program, participating schools are provided $50,000 planning grants to develop a comprehensive school reform plan. As part of the planning phase, the schools must hire qualified external evaluators to assist in developing the reform plans. Once a school's plan is approved by the SBE, the school receives annual implementation grants of up to $200 per enrolled student. Schools will receive the implementation grants for two years, and may be granted a third year of funding by the SBE if they continue to struggle to meet their API growth targets. The SBE can decide to impose sanctions after either the second or third year of funding for schools that continue to struggle. The first cohort will reach the end of their two-year implementation grant in July 2002 and potentially could face sanctions as early as fall of 2002.

Program Eligibility Has Changed. Any school whose average student score was below the 50^th percentile on the Stanford-9 was eligible to participate in the first II/USP cohort in 1999-00. Under the original criteria, over half of the schools in the state were eligible to apply for the program, and 1,419 schools applied for the 430 slots available. Chapter 695, Statutes of 2000 (SB 1552, Alpert), amended the PSAA to change the eligibility criteria to be based upon the API. As revised by Chapter 695, schools in the lower half of the API which did not meet their API growth targets were eligible. Based upon 1999-00 API scores, 938 schools were eligible in 2000. Of the 938 schools, 532 applied for the 430 slots available.

Legislature Should Broaden II/USP Eligibility to Include Lowest-Achieving Schools. The policy change of Chapter 695 to focus on schools not making their growth targets makes sense to ensure that schools that are struggling to improve receive additional resources. However, only focusing on schools not achieving their growth targets may be too narrow. For example, under the eligibility criteria in Chapter 695, the school with the lowest API score in the entire state is not eligible for this immediate assistance program since it reached its annual growth target. Even though such a low-performing school is showing signs of improvement, we believe that its improvement could potentially be expedited by participation in this program. Such a low-performing school is likely to be in greater need of external assistance than a school in the fifth decile (close to the state average) which did not meet its annual API growth target. Accordingly, we recommend the Legislature expand the eligibility for the II/USP program to all schools in the lowest two deciles of API ranking.

Extra Federal Funds Available for II/USP

We recommend the State Department of Education report at budget hearings on the extent that General Fund costs for the Immediate Intervention/Underperforming Schools Program can be reduced by the use of federal funds.

California has received $92 million in federal funding since 1998-99 which can only be used for II/USP implementation grants (see Figure 5). As mentioned above, 78 schools have received federal funds for II/USP. To date, SDE has provided $32 million in federal funds to those schools. An additional $16.3 million is proposed for those schools in the budget year. After funding these 78 schools, SDE will have an additional $44 million remaining to provide implementation grants to new schools.

The 430 schools in the second cohort will have the option of participating in II/USP using either General Fund monies or the available federal funding. Schools in the second cohort have until May 15, 2001 to apply for federal funds for their II/USP implementation grants. Federal funding has the following trade-offs:

• Guaranteed Funding Level. Federally funded schools receive a guaranteed $200 per student for three years, while state-funded schools have no per student guarantee and only two guaranteed years of funding.
• More Rigorous Requirements. Federally funded schools apply on a competitive basis, and must complete a more rigorous planning, reporting, and evaluation process.

The SDE has provided a rough estimate of the number of schools that it expects to receive federal funds. We estimate that $10 million in federal funds will be used to fund schools selected by SDE to receive federal funds, leaving approximately $34 million in federal funds available for future II/USP schools. Such low usage of federal funds for II/USP would result in greater demand for state funds for II/USP, which we discuss below. We recommend the SDE report at budget hearings on steps the Legislature can take to increase the usage of available federal funds to pay the costs of II/USP.

General Fund Appropriation for II/USP May Be Low

The amount provided in the Governor's budget for the Immediate Intervention/Underperforming Schools Program is inadequate to maintain the current-year per-pupil grant level.

In 2000-01, schools in the first cohort received $168 per-pupil implementation grants. The Department of Finance informs us that the Governor did not have a target per-pupil implementation grant for II/USP in developing the 2001-02 budget. The budget provides $92 million for implementation grants. Based upon our estimate of federal fund usage for II/USP, we estimate that the per-pupil funding level provided in the Governor's budget would result in implementation grants of $146 per pupil.

Figure 6 shows the options that the Legislature has (based upon our cost estimates) if it wants to (1) maintain the Governor's proposed level, (2) maintain last year's per-pupil grant amount of $168 per pupil, or (3) provide schools the maximum statutory grant amount of $200 per pupil. If the Legislature wants to either maintain the current-year's per-pupil funding level or increase the per-pupil funding level to the maximum, it would require additional General Fund support of $15 million or $36 million, respectively.