Legislative Analyst's Office

Analysis of the 2001-02 Budget Bill


School Accountability

The Public Schools Accountability Act (PSAA) (Chapter 3x , Statutes of 1999, [SB 1x, Alpert]) created a statewide school-level accountability system which (1) rewards schools for academic improvement, (2) provides external assistance for lower-performing schools, and (3) potentially sanctions schools which continue to fail after receiving external assistance. The Academic Performance Index (API) is the cornerstone of the PSAA. Its purpose is to measure a school's academic performance and the growth in performance over time.

Recently, the second annual set of API scores was released, allowing the first API growth scores to be calculated. Based on the API growth data, the State Department of Education (SDE) will be distributing $677 million in rewards to schools, teachers, and other school personnel in the current year. Figure 1 shows the various rewards programs tied to API results.

Figure 1

Rewards Programs Based On API a

1999-00 Through 2001-02
(Proposition 98 Appropriations, In Millions)

Program

1999-00

2000-01

2001-02
Proposed

Governor's Performance Awards

$96b

$131b

$350

Teacher Performance Awards

350

Certificated Staff Performance Incentives

50

100

100

Totals

$146

$581

$450

a Academic Performance Index.

b The Public Schools Accountability Act appropriated $96 million in the 1999-00 fiscal year. Those funds were carried over and are being used for rewards in 2000-01.

As shown in Figure 2, the budget also proposes significant resources for various programs targeted at low-performing schools, based upon the API. As Figures 1 and 2 illustrate, the state is allocating a significant amount of money—almost $1.7 billion in 1999-00 through 2001-02—for programs using this index. The box nearby provides some background information on how the API is calculated and how the measure is used.

Figure 2

Programs Targeted at Low-Performing
Schools, Based on the API a

2000-01 and 2001-02
(In Millions)

Program

2000-01

2001-02

Immediate Intervention/Underperforming Schools Program

$52.1b

$139.1b

Teaching as a Priority

118.7

118.7

Governor's Teaching Fellows

3.5

17.5

National Board Certification

10.0

10.0

Cal Grant T

10.0

10.0

Assumption Program of Loans for Education (APLE)

5.6c

14.6c

Totals

$199.9

$309.9

a Academic Performance Index.

b Includes both General Fund and federal funds.

c The APLE provided 6,500 warrants in 2000-01 with the same number proposed for 2001-02. There is no immediate cost to issuing a warrant, but the state will incur out-year costs of up to $19,000 each. In the proposed budget, the state would commit to out-year costs of up to $123.5 million for 6,500 warrants.

How Does the Academic Performance Index Work?

An API Score

The State Department of Education (SDE) calculates a score ranging from 200 to 1,000 for each school based on its students' standardized test scores (Stanford-9). The calculation is relatively complex and involves combining student scores across the different subject areas. The SDE ranks the schools and provides a decile ranking (one to ten), which is generally used for its ease of understanding. A decile ranking of ten is best.

Subgroup Population API Score

The SDE performs a similar calculation for each "numerically significant" subgroup of students. These subgroups include: African American, American Indian, Asian, Filipino, Pacific Islander, white not Hispanic, Hispanic, and socioeconomic disadvantaged. Numerically significant means either (1) at least 30 pupils and at least 15 percent of a school's enrollment or (2) at least 100 students in a school.

Growth Target

Annually, a school receives a growth target for the next school year along with its API score. The State Board of Education set an interim statewide performance target of 800 points. Schools with API scores below 800 must close the gap between their current score and the state performance target by at least 5 percent to meet their growth target. For example, if a school's 1999 API score was 500, the school's growth target would be (800 - 500) * 5 percent = 15 points. Schools with API scores at the 800 point level or above must increase their scores by 1 point to meet their growth target.

Subgroup Growth Targets

Each numerically significant subgroup at a school has a growth target of 80 percent of the schoolwide growth target. In the example used above, the schoolwide growth target was 15 points. Under that example, each numerically significant subgroup at the school must improve by at least 80 percent of 15 points - thus, by at least 12 points to meet the target.

Reward Programs

Schools are eligible for rewards based on meeting their assigned annual growth targets for their API scores and subgroup API scores. The reward programs are described below.

Governor's Performance Awards

School Site Employee Performance Bonus (One-Time)

Certificated Staff Performance Incentives

Eligibility Requirements?

  • Meet school API growth target.

  • Meet subgroup targets.

  • Have 95 percent Stanford-9 participation in grades K-8 or 90 percent participation for grades 9-11.

  • Same as Governor's Performance Awards

  • Low-performing schools (below 50th percentile on API); ranked by highest API growth rates.

  • Same subgroup and participation rate requirements as Governor's Performance Awards.

Who Receives the Funding?

  • School site councils decide uses.

  • Half decided by school site councils, other half distributed among all school site staff.

  • Certificated staff only.

  • 1,000 across state receive $25,000 each.

  • 3,750 receive $10,000 each.

  • 7,500 receive $5,000 each.

 

 

The API Is a Work in Progress

The state's accountability system is still a work in progress. The API is currently based on only one measure, namely the Stanford-9 test results. The Stanford-9 is an "off the shelf" student assessment produced by Harcourt Brace and administered to California students in grades 2 through 11. The PSAA requires that the API include at least five additional measures: graduation rates, student attendance rate, teacher attendance rate, Standards-Aligned Standardized Testing and Recording (STAR), and the High School Exit Exam (HSEE). The Legislature included this broad set of outcome measures to ensure that increases in the API were based upon real gains in achievement and not temporary fluctuations in a single measure.

The SDE has not included graduation rates, student attendance nor teacher attendance in the API because it is currently not able to collect accurate school-level data on these outcome measures. The SDE has not included Standards-Aligned STAR and HSEE results because the assessments are not yet fully operational. While California's current system is a long way from the full implementation, many efforts are under way to improve the quality of the information used in calculating a school's API (see shaded box on Improving the API).

Improving the API

Since the state rewards schools for their growth in academic performance index (API) scores, the State Department of Education (SDE) must be able to collect data on performance measures for two years before it can be included in the API measure. The timetable for possibly including additional measures in the API is as follows:

Measure to Include in API

Potential Availability

Standards-Aligned STAR

2002

High School Exit Exam

2003

Graduation rate

Unknown

Student attendance rate

Unknown

Teacher attendance rate

Unknown

Standards-Aligned Standardized Testing and Recording (STAR). As part of the spring 2001 STAR test, the SDE will administer four subject tests (math, language arts, science, and social science) aligned with California's content standards. Additionally, SDE will provide performance standards for the first time in spring 2001. A performance standard translates the results of a student's test into a scale ranging from mastery of the subject area to below basic. Assuming there are no problems with the validity of the assessments or performance standards, results from the Standards-Aligned STAR could be included in the API as early as 2002.

High School Exit Exam (HSEE). Based upon the current State Board of Education plan, all 10th grade students will take the HSEE in spring 2002. Potentially, SDE could include the results from the spring 2002 and spring 2003 administrations in the 2003 API.

Graduation Rate, Student and Teacher Attendance Rate. The Public Schools Accountability Act required the Superintendent of Public Instruction (SPI) to report by September 1999 on the accuracy of the measures to be included in the API, and to recommend necessary actions to implement an accurate reporting system for these measures. The SPI's report recommended use of the California School Information Services to collect data on graduation rates and student and teacher attendance rates. The administration rejected the SPI proposal, and $500,000 was included in the 2000-01 Budget Act for the Office of the Secretary for Education (OSE) to conduct a study of data collection alternatives for expanding the number of measures in the API. According to OSE, it will release a Request for Proposal some time this spring for this study. The findings of the study will determine the timetable for including these measures in the API.

In addition to the measures described above, the Governor's budget provides $5 million to create a longitudinal student assessment database. The main policy goal of this proposal is to enable the state to begin to construct "value-added" measures of student performance, the advantages of which we discuss at length in the "K-12 Databases" section of this chapter.

 

 Problems With the Stanford-9

The Stanford-9 provides California schools with a score reflecting how students in California perform relative to a national norm created from the results of administering the same test to a nationally "representative" group of students. We see three major problems with the Stanford-9:

We discuss each of these problems in more detail below.

Stanford-9 Not Aligned to Standards. The current version of the API is solely dependent on the Stanford-9, an assessment tool that is not aligned to the state's academic content standards. The California Academic Content Standards adopted by the State Board of Education (SBE) define what the state wants students to learn in each grade and subject (see shaded box for actions related to the academic content standards). Since the Stanford-9 tests a different set of information than the content standards, the state is sending conflicting messages about what schools should be teaching. Should teachers and administrators focus instruction on areas covered by the Stanford-9? The state is offering bonuses that can exceed $25,000 per individual for success measured by that test. Or should they teach what is expected under the academic content standards for which the state has invested $1 billion for new textbooks, the Governor is proposing $335 million for staff development, and upon which the HSEE is based?

State Academic Content Standards

The California Assessment of Academic Achievement Act (Chapter 975, Statutes of 1995 [AB 265, Alpert]), required the State Board of Education (SBE) to adopt academically rigorous content standards in the five core curriculum areas—math, English-language arts, science, social science, and English language development. The standards provide a framework for what a student should learn in five core subject areas in each grade.

Since adoption of the standards, the state has taken several actions to emphasize the importance of the standards to schools and teachers.

  • Instructional Materials. The Schiff-Bustamante Standards-Based Instructional Materials Program (Chapter 312, Statutes of 1998, AB 2041) is provid-ing school districts with $1 billion over four years to purchase instructional materials aligned with the academic content standards.

  • The Standardized Testing and Recording (STAR) Augmentation. In 1998-99, the SBE added an augmentation to the STAR test in math and English-language arts, consisting of questions specifically aligned to state content standards. In the current year, the board will develop student performance standards around the STAR augmentation to determine which students have met basic, proficient, and advanced levels of performance.

  • High School Exit Exam (HSEE). Chapter 1x, Statutes of 1999 (SB 2x, O'Connell), authorized the HSEE. This legislation requires the test to be aligned to the state academic standards.

  • Teacher Staff Development. The Governorís budget proposes $490 million— including a $335 million expansion—for teacher staff development that trains teachers in using state standards.

  • Other Actions to Emphasize Standards. The state also has aligned the Golden State Exams, and developed sample curriculum for each subject based on the standards.

 

  How Meaningful Are Stanford-9 Scores for ELLs? The second problem with the Stanford-9 is the validity of Stanford-9 scores for many ELL students. The advisory committee of experts appointed by the SBE for development of the API advised the board to exclude Stanford-9 test scores for ELL students from the API calculation. Their recommendation was based upon analysis of Stanford-9 test results for ELL students. A school may be doing an excellent job of helping a student transition to English fluency, but that student's Stanford-9 score may not reflect the improvements the student has made. The SBE did not follow the recommendation. Although not implemented by the SBE, the advisory committee's recommendation puts into question the comparability of API scores between schools with many ELL students and schools with few or no ELL students.

Test Questions Don't Change From Year to Year. The final problem with the Stanford-9 is that the set of actual questions on the test does not change from year to year. Particularly with a high stakes test, it is important to vary test questions from year to year in order to minimize possibilities for literal "teaching to the test" and outright cheating. The SDE did not stipulate the rotation of questions from year to year in its contract with the publisher of the Stanford-9. So for the five-year duration of the contract—which expires after the administration of the spring 2002 test—the test given in each grade is exactly the same in each of its administrations. Teachers know the exact questions that their students will be asked and upon which their school will be evaluated. Providing $677 million in rewards based solely upon this test creates incentives for schools to misuse their knowledge of Stanford-9 test questions.

Reduce Funding for Governor's Performance Awards

We recommend the Legislature reduce the funding proposed for the Governor's Performance Awards by $219 million until the Academic Performance Index is based upon a broader set of reliable performance indicators. (Reduce Item 6110-123-0001 by $219 million.)

Funding Too High for Untested Measure. As discussed above, the API is a work in progress and will take several years to develop fully. The administration is making progress on increasing the number of outcome measures used to calculate the API, and is planning to switch to a value-added approach, which will improve the use of the existing achievement measure. Until then, however, there are numerous problems with the Stanford-9 being used as the sole measure for the API as discussed above.

Consequently, we believe it is not only premature to attach such high stakes (and such large funding amounts) to the API but it may be counterproductive to improving learning outcomes for California's public school students by redirecting the focus of teaching from the state's broad content standards to the set of material tested on the Stanford-9. Once the Governor and SDE have been able to develop the data collection systems and assessment tools needed to support a quality accountability system, the Legislature then should determine the appropriate level of funding needed for school incentives. Accordingly, we recommend reducing funding for the Governor's Performance Awards to $131 million, the same level appropriated in the 2000-01 Budget Act. This recommendation would create an ongoing $219 million of Proposition 98 savings which the Legislature could direct more effectively to other K-12 education needs.
(Reduce Item 6110-123-0001 by $219 million.)

Inadequate Funding to Evaluate PSAA

We recommend the Legislature (1) increase funding to the State Department of Education (SDE) to contract for an evaluation of the Public Schools Accountability Act from $250,000 to $500,000, and (2) reappropriate $250,000 included in Item 6110-001-0001 of the 2000-01 Budget Act for the same purpose. We further recommend SDE report at budget hearings on the possibility of using federal funds for this evaluation.

The PSAA requires SDE to contract for an external evaluation of the PSAA—including the Governor's Performance Awards program, Immediate Intervention/Underperforming Schools Program (II/USP), and the impact of these programs on student achievement. The 2000-01 Budget Act provided SDE with $250,000 for the evaluation. In the summer of 2000, the SDE released a Request for Proposal for the evaluation but did not receive a single qualified bid. As a result, the state has not begun to evaluate this major program. The Governor proposes an additional $250,000 from the General Fund (non-Proposition 98) for the PSAA evaluation this year.

We believe that the $500,000 that would be appropriated between the two budgets is still inadequate given the complexity of issues that the evaluation must address. For example, under this evaluation, the SDE is required to contract for a comprehensive evaluation of the implementation, impact, costs, and benefits of II/USP and Governor's Performance Awards program. A preliminary report is due March 31, 2002. The state may distribute as much as $1.7 billion related to these programs in 1999-00 through 2001-02. Potentially more important, as early as fall 2002, schools and employees of schools participating in the II/USP could face sanctions from the state under the terms of the PSAA. Sanctions could include the Superintendent of Public Instruction reassigning principals, reorganizing schools, reassigning certificated staff, contracting out the management of schools or even closing schools. The state needs to make sure that the measure upon which such high-stakes decisions are made truly tracks changes in student achievement. Given the importance of this program and the extensive list of legislative requirements for the evaluation, we recommend that the Legislature (1) increase the appropriation for the evaluation from $250,000 to $500,000 in the budget year, and (2) reappropriate $250,000 included in the 2000-01 Budget Act for this purpose. (This would make $750,000 available to the study.)

As discussed below, the II/USP is partially funded with federal funds. The federal grant which provides funding for II/USP allows the state to use up to 5 percent of the grant amount to pay the costs of administering the program. We believe that evaluation of the program is an essential part of the program's administration. To the extent feasible, we believe it is preferable to use federal funds instead of the General Fund to pay for at least part of the PSAA evaluation. Federal funds could not be used for all of the evaluation costs because the PSAA evaluation must evaluate programs beyond II/USP. We recommend SDE report at budget hearings on the feasibility of using federal funds to pay for at least part of the PSAA evaluation.

Certificated Staff Performance Incentives Overfunded

We recommend that the Legislature reduce funding for the Certificated Staff Performance Incentive Awards by $50 million on a one-time basis to take advantage of carryover funding available for this program. (Reduce Item 6110-133-0001 by $50 million.)

Chapter 52, Statutes of 1999 (AB 1114, Steinberg), created the Certificated Staff Performance Incentive Act to provide $5,000 to $25,000 rewards to certificated staff at schools meeting two criteria:

Figure 3 shows the appropriations and expenditures that have been made or are proposed for this program. The 1999-00 Budget Act provided $50 million for the program. No funds were distributed in 1999-00, however, because only one year of API scores was available prior to summer 2000, therefore making it impossible to determine API growth targets. The 2000-01 Budget Act and the 2001-02 Governor's Budget each include $100 million for this program. The administration's spending plan for 2000-01 is to use the $50 million from the 1999-00 Budget Act and $50 million from the 2000-01 Budget Act. As a result, $50 million from the 2000-01 Budget Act will still be available for rewards in the budget year. We recommend the Legislature use this $50 million in carryover funds for the certificated staff incentives and reduce Item 6110-133-0001 by $50 million. This action would provide the total of $100 million for budget-year expenditures proposed for this program, yet also free up $50 million in Proposition 98 funds for other K-14 education priorities of a one-time nature.

Figure 3

Certificated Staff Incentive
Program Funding

1999-00 Through 2001-02
(In Millions)

 

Appropriation in Budget Act

Proposed
Expenditure Level

1999-00

$50

2000-01

100

$100

2001-02a

100

100

Totals

$250

$200

a Amount proposed in Governor's budget.

Immediate Intervention/Underperforming Schools Program

We recommend the Legislature enact legislation to broaden the eligibility criteria for the Immediate Intervention/Underperforming School Program to include all schools in the lowest two deciles of the Academic Performance Index.

The II/USP provides state and federal support for school-wide reform efforts at low-performing schools. The 2001-02 Governor's Budget proposes $113.8 million from the General Fund (Proposition 98) for the program. The budget would provide planning grants for a third cohort of 430 schools, and implementation grants for the first and second cohorts (782 schools in total). Figure 4 show the schools and number of students in each cohort for 2001-02. The Governor's budget continues to fund 78 schools with federal funds, which we discuss below.

Figure 4

Participation in II/USP a in 2001-02

Cohort 1 (participation started 1999-00)

  • Grant: Implementation.

  • Number of Schools: 352b

  • Number of Students: 285,000c

Cohort 2 (participation started 2000-01)

  • Grant: Implementation

  • Number of Schools: 430d

  • Number of Students: 430,500

Cohort 3 (participation starts 2001-02)

  • Grant: Planning

  • Number of Schools: 430

  • Number of Students: Unknown

a Immediate Intervention/Underperforming Schools Program.

b An additional 78 schools are funded with federal funds.

c An additional 89,000 students are funded with federal funds.

d Some schools in Cohort 2 may apply to be funded with federal funds.

In their first year in the program, participating schools are provided $50,000 planning grants to develop a comprehensive school reform plan. As part of the planning phase, the schools must hire qualified external evaluators to assist in developing the reform plans. Once a school's plan is approved by the SBE, the school receives annual implementation grants of up to $200 per enrolled student. Schools will receive the implementation grants for two years, and may be granted a third year of funding by the SBE if they continue to struggle to meet their API growth targets. The SBE can decide to impose sanctions after either the second or third year of funding for schools that continue to struggle. The first cohort will reach the end of their two-year implementation grant in July 2002 and potentially could face sanctions as early as fall of 2002.

Program Eligibility Has Changed. Any school whose average student score was below the 50th percentile on the Stanford-9 was eligible to participate in the first II/USP cohort in 1999-00. Under the original criteria, over half of the schools in the state were eligible to apply for the program, and 1,419 schools applied for the 430 slots available. Chapter 695, Statutes of 2000 (SB 1552, Alpert), amended the PSAA to change the eligibility criteria to be based upon the API. As revised by Chapter 695, schools in the lower half of the API which did not meet their API growth targets were eligible. Based upon 1999-00 API scores, 938 schools were eligible in 2000. Of the 938 schools, 532 applied for the 430 slots available.

Legislature Should Broaden II/USP Eligibility to Include Lowest-Achieving Schools. The policy change of Chapter 695 to focus on schools not making their growth targets makes sense to ensure that schools that are struggling to improve receive additional resources. However, only focusing on schools not achieving their growth targets may be too narrow. For example, under the eligibility criteria in Chapter 695, the school with the lowest API score in the entire state is not eligible for this immediate assistance program since it reached its annual growth target. Even though such a low-performing school is showing signs of improvement, we believe that its improvement could potentially be expedited by participation in this program. Such a low-performing school is likely to be in greater need of external assistance than a school in the fifth decile (close to the state average) which did not meet its annual API growth target. Accordingly, we recommend the Legislature expand the eligibility for the II/USP program to all schools in the lowest two deciles of API ranking.

Extra Federal Funds Available for II/USP

We recommend the State Department of Education report at budget hearings on the extent that General Fund costs for the Immediate Intervention/Underperforming Schools Program can be reduced by the use of federal funds.

California has received $92 million in federal funding since 1998-99 which can only be used for II/USP implementation grants (see Figure 5). As mentioned above, 78 schools have received federal funds for II/USP. To date, SDE has provided $32 million in federal funds to those schools. An additional $16.3 million is proposed for those schools in the budget year. After funding these 78 schools, SDE will have an additional $44 million remaining to provide implementation grants to new schools.

Figure 5

Federal Funding for II/USP a

(In Millions)

 

Federal
Appropriation

Spending

1998-99

$16.2

1999-00

16.1

$15.2

2000-01

26.5

16.3

2001-02

32.9

16.3

Totals

$91.9

$47.9

a Immediate Intervention/Underperforming Schools Program.

The 430 schools in the second cohort will have the option of participating in II/USP using either General Fund monies or the available federal funding. Schools in the second cohort have until May 15, 2001 to apply for federal funds for their II/USP implementation grants. Federal funding has the following trade-offs:

The SDE has provided a rough estimate of the number of schools that it expects to receive federal funds. We estimate that $10 million in federal funds will be used to fund schools selected by SDE to receive federal funds, leaving approximately $34 million in federal funds available for future II/USP schools. Such low usage of federal funds for II/USP would result in greater demand for state funds for II/USP, which we discuss below. We recommend the SDE report at budget hearings on steps the Legislature can take to increase the usage of available federal funds to pay the costs of II/USP.

General Fund Appropriation for II/USP May Be Low

The amount provided in the Governor's budget for the Immediate Intervention/Underperforming Schools Program is inadequate to maintain the current-year per-pupil grant level.

In 2000-01, schools in the first cohort received $168 per-pupil implementation grants. The Department of Finance informs us that the Governor did not have a target per-pupil implementation grant for II/USP in developing the 2001-02 budget. The budget provides $92 million for implementation grants. Based upon our estimate of federal fund usage for II/USP, we estimate that the per-pupil funding level provided in the Governor's budget would result in implementation grants of $146 per pupil.

Figure 6 shows the options that the Legislature has (based upon our cost estimates) if it wants to (1) maintain the Governor's proposed level, (2) maintain last year's per-pupil grant amount of $168 per pupil, or (3) provide schools the maximum statutory grant amount of $200 per pupil. If the Legislature wants to either maintain the current-year's per-pupil funding level or increase the per-pupil funding level to the maximum, it would require additional General Fund support of $15 million or $36 million, respectively.


Return to Education Table of Contents, 2001-02 Budget Analysis