April 2000

	Extended School Year Program At Compton Unified School District: Report on Program Effectiveness
	Required by Chapter 943, Statutes of 1998 (Senate Bill 1866, Hughes)

Executive Summary

Background

In 1995, Chapter 870 (AB 174, W. Murray) created the Extended School Year (ESY) program to improve academic outcomes at Compton Unified School District's lowest-performing schools. In 1998, Chapter 943 (SB 1866, Hughes) extended the program to more schools in the district and required the Legislative Analyst's Office (LAO) to evaluate the program's effectiveness at improving student achievement. The goal of the program is to raise up to grade level performance students identified as two years below grade level in mathematics and English language arts. Program funding is due to sunset on January 1, 2001.

LAO Findings

• The district has been unable to provide complete and reliable data to permit evaluation of the impact of the ESY program on student achievement. Until such time as the district is able to provide such data, little more can be concluded.

• The ESY program has met some of its program goals. For example, the district has been able to provide an enhanced classroom environment during the ESY program and may be improving its ability to retain higher quality teachers, although the district still has a very high rate of emergency permit teachers.
• Conflicts arising from concurrent operation of ESY and other summer programs may affect the quality of the district's other summer school programs.

LAO Recommendations

• Shared Transitional Funding. We recommend that the Legislature extend the program for two years, sharing the cost with the district.
• Data Collection and Evaluation. We further recommend dedicating a percentage of ESY program funds for testing, data collection, and evaluation.

Introduction and Background

Legislative History

In 1993, in response to serious fiscal and academic problems, the state assumed administrative control of the Compton Unified School District (CUSD) in Los Angeles County. Chapter 78, Statutes of 1993 (AB 657, W. Murray), (1) directed the Superintendent of Public Instruction to appoint an administrator who would assume the rights and duties of the school board and (2) provided a $10.5 million loan to meet the most urgent of the district's financial responsibilities. (The Legislature later increased the loan to $20 million.)

Chapter 870, Statutes of 1995 (AB 174, W. Murray), established the "Extended School Year" (ESY) program in Compton. Chapter 870 required the district to provide 39 additional school days to those pupils at its three lowest-performing elementary schools assessed as two or more grade levels behind in mathematics or English language arts. The legislation made ESY attendance compulsory for these pupils. It required the district to test the pupils before and after ESY attendance to determine progress in grade level performance. It required the State Department of Education (SDE) to approve the testing instruments, curricula, and processes for teacher selection, mentoring, and staff development for ESY. The statute specified that the additional school days be "devoted" to instruction in basic skills in mathematics and English language arts. Thus, the required program was not really an "extended school year," as that term is commonly used, but instead a remedial summer school program focused on two subject areas.

In September 1998, Chapter 943 (SB 1866, Hughes), expanded the ESY program to additional schools in the district, authorized the program for all the K-12 grades, and set an annual state funding cap of $3 million for the program. (The district chose to provide ESY to students in grades K through 8.) Seventeen of the district's 34 schools participated in the ESY program in the summer of 1999, providing more than 5,000 students with remedial instruction (about one-sixth of total district enrollment).

Along with expanding the ESY program, Chapter 943 directed the Legislative Analyst's Office (LAO) to evaluate the effectiveness of the ESY program in improving pupil academic outcomes, based on pupil test data and baseline data collected by the SDE under the requirements of Chapter 870.

The program has a sunset date of January 1, 2001. Thus, the last ESY program funded under current law will be held in the summer of 2000.

District Implementation

In addition to student achievement goals, the district has added the further goal of improving teacher quality through hiring criteria, staff development, and a determined effort to retain successful teachers. The district also worked toward establishing a classroom environment conducive to accelerated literacy learning by lowering the teacher/student ratio, and providing a teacher aide in each classroom for the first four hours of each school day.

In implementing the ESY program, the district has dedicated the first four hours of each day's instruction to English language arts and reading, and the last two hours to mathematics and science. It uses the same instructional materials utilized during the regular school year. At the end of the program, the district assesses the students using multiple measures to determine retention or promotion.

District Promotion/Retention Policy

State law requires every school board to approve a policy regarding the promotion and retention of pupils. Below we briefly describe Compton's policy and the key role played by ESY.

• Identification. The district uses multiple tools to assess student performance--including student grades, assessments by the teachers, and standardized test results. All K-8 students two or more grade levels behind based upon these measures, must participate in ESY. Those students in grades 3, 5, and 8 unable to improve their performance over the course of the ESY are retained (must repeat the grade).
• Extended School Year. The ESY program gives students another chance to meet grade level achievement standards. During the 39-day additional instructional period, teachers measure student improvement with periodic oral exams, weekly writing assessments, and a standardized assessment at the end of the program. Data from these assessments as well as teacher input are used for promotion/retention decisions.
• Ongoing Remediation. Remediation does not necessarily end with the ESY program, but continues in the regular school year for many students. The district provides the same intensive environment for this ongoing remediation, including reduced-sized classes, four hours of intensive reading and language arts each morning, a teaching assistant in each classroom, and many of the same teachers who teach in the ESY program.

Participation and Funding

Figures 1 and 2 display basic information on the ESY program. Figure 1 shows the number of participating students and teachers each year along with the average student/teacher ratio. Figure 2 shows the revenue the district received from the state for ESY and the amount expended on the program. The figure shows the resulting hourly cost per student for the program.

The cost of the ESY program ranged from $2.41 to $3.19 per pupil hour over the four years of the program. In the first two years of the program, state revenue exceeded district expenditures by $375,000 and $450,000, respectively. These extra monies were available for general purposes of the district. In the third and fourth years of the program, the district chose to use the funding to provide services to an additional 1,052 and 1,885 students, respectively, above the legislated funding assumption of 3,115 students.

Program Analysis and Findings

The Effect of ESY on Student Achievement

As mentioned above, the district does use multiple measures in the spring to determine which students will participate in the ESY program, and it uses multiple measures at the end of the summer to assess which students are promoted. Unfortunately, it does not give a test at the end of the school year or on the first day of the ESY program, and then administer the same or comparable test at the end of the summer. Thus, the district did not administer the pre/post assessments required by statute.

For three out of the last four years, the district has administered the California Achievement Test, version 5 (CAT-5) in April and again in October, thereby "bracketing" three of the four ESY sessions. The CAT-5 is a widely used norm-referenced assessment test and is one of the tests that current law specifically permits for ESY purposes. Even though it is not possible to fully separate student gains on the CAT-5 associated with participation in the ESY program from gains that may have been made over the three months of regular school between April and October, statistically positive test gains on the CAT-5 could be used to at least draw some tentative conclusion about the effectiveness of the ESY program.

Unfortunately, neither the district nor the SDE has been able to provide complete and reliable data from the CAT-5 for students in the ESY program. Despite repeated requests over the last 18 months (including numerous visits to the district), the first data set produced by the district arrived in February 2000. The data set was incomplete (fewer than 500 students out of the 1,697 students who participated in ESY in its first year--summer 1996) and inaccurate. For example, over 75 percent of the 500 student scores were listed as zero. This reported result is not possible under the scoring conventions of the CAT-5.

At the time of this report, neither the district nor the SDE had produced usable data. After several visits and numerous hours working with the district's data assessment team, we have concluded the district lacks the capacity to collect, manage, and use testing data in a systematic way. Consequently, we are unable to evaluate the effect of the ESY program on the basis of student test scores, as required by statute.

Other Findings

Teacher Quality

Historically, the district has suffered from a lack of trained credentialed teachers, a high rate of teacher absenteeism, and an inability to retain quality teachers. The district has sought to address these concerns as an important part of its ESY program.

Initially, the ESY program implemented selective performance-based hiring criteria and a centralized hiring process through a district-level selection committee. When the district expanded participation to 17 schools, the recruiting of quality teachers fell largely to the principals. Below, we provide some information on the district's efforts.

According to the district's policy, teachers were to be evaluated for participation in the ESY program based upon their students' gains in test scores during the regular school year, classroom observation, and evaluations by their principals. (We were unable to verify, however, the extent to which student test scores actually were used in the evaluations.) In addition, ESY teachers were required to make a five-year commitment to the program and commit to attend all instructional and staff development days.

Figure 3 shows the basic characteristics of teachers who taught in the ESY program compared to the district's teaching force as a whole, based on an LAO survey of teachers. As the figure indicates, there was not much difference between the ESY teachers and the district averages with respect to percent of teachers with emergency permits and years of experience. The figure also indicates, however, that the district has had some success in minimizing the number of novice teachers in the ESY program.

Staff Development. Giving teachers the professional tools they need is another goal of the program. The ESY teachers are required to attend all 39 days of ESY instruction plus six additional days of staff development aimed at the literacy curriculum. It is difficult, however, to separate the benefits of staff development during the regular school year from the summer ESY staff development. When asked to evaluate the influence of the ESY staff development, a large proportion of teachers listed "previous staff development" as more influential on their teaching method than ESY staff development.

Classroom Environment

The district was able to create a more enriched environment in the ESY program than during the regular school year by reducing class size somewhat, providing greater access to Sylvan Learning Centers (discussed below), and implementing an intensive remedial curriculum.

Class Size. The district did meet its goal of providing instruction in a reduced class size environment on average. However, the class sizes varied from 15 to 30, and on average were only slightly less than the district's regular school year average of 26 to 1.

The Sylvan Learning Centers. Sylvan Learning Centers provide an intensive reading program which pulls students out of the regular class and matches three students with one teacher for an hour. Students in the ESY program are afforded greater access to Sylvan Learning Centers than other students during the regular school year. The ESY students at 10 of the 17 schools that offered an ESY program during the summer of 1999 had access to Sylvan Learning Centers. Sylvan guarantees improvement on standardized reading tests--provided the student meets the minimum number of instructional hours--and is generally considered a successful and effective program. The district, however, was unable to provide a report on the effectiveness of the Sylvan program in Compton. Without a report or reliable data, the effectiveness of the Sylvan reading program in Compton remains unclear.

Curriculum. The ESY program provides students targeted for retention a curriculum which supports an intense learning environment structured for increasing reading and language arts literacy. The first four hours of each day are devoted to this task. Teacher aides are available in each class during these hours to assist in the process. The curriculum for the program is based upon the same instructional requirements as the regular school year, only in a more intensive mode. An ESY class also uses the same books and instructional materials, as well as the same goals and achievement standards, as the regular school year.

The ESY Interaction With Existing Summer School Programs

The state currently funds several supplemental instruction/summer school programs, as outlined in Figure 4. The goals of the ESY program are similar to goals of the three remedial programs described in the figure, with the main difference being that state funding for the summer school programs is capped at 120 hours per pupil, while the ESY program provides 220 hours of classes.

Compton operates the other summer school programs in addition to the ESY program. At school sites where both ESY and other summer school programs were in place, we observed that the programs seemed to be somewhat disruptive of each other. For example, one group of students would finish school for the day--with all the attendant noise and disruption--while other students were still in class. In addition, according to district staff, the district had difficulty finding teachers for the other summer school programs because the pay rate was significantly lower than the ESY salary. In view of the above, the district's emphasis on the ESY program could be affecting the quality of the service provided in the other summer school programs.

Recommendations

Should the Program Be Continued?

As noted earlier, the ESY program is currently scheduled to end after this summer. Figure 5 displays several options for the Legislature to consider regarding future funding of the program.

The first option--extending the program for another five years--is the most expensive one. Given the failure of the district and the SDE to demonstrate the effectiveness of the program in improving pupil academic outcomes, we do not consider this option to be justified. Option two, which provides full state funding for a shorter period of time while the district works out a transition plan, is a reasonable alternative. In our view, its main drawback is that state support goes from full funding to zero at the end of the program, making the transition for the district potentially more difficult. The third option is a shared-cost transitional approach. Under this option, the state would continue to provide additional funding for the ESY program for two more years. The state, however, would only pay for the added costs above regular summer school programs (approximately $1.5 million annually).

The last option, of simply allowing the funding to sunset, would not necessarily cause the ESY program to be discontinued. It could, we believe, force the district to scale back the number of students the program could serve, and reduce the resources available to the program on an ongoing basis. However, if the ESY program is, in the district's evaluation, the best place for CUSD's resources to go, it could continue to spend summer program money on its implementation.

LAO Recommendation

Unfortunately, as discussed above, there is insufficient data to permit a definitive conclusion as to whether the ESY program has been effective so far. However, neither can we conclude that the program has been ineffective. This makes it difficult to recommend program continuation, but also difficult to recommend program termination. The state, in its present position of trustee for the district, still bears special responsibility for assuring improvement of the educational programs available to Compton's public school pupils. In view of this, we think some "benefit of the doubt" should go toward some form of program continuation, provided (1) there is some sharing of cost between the state and the district and (2) the district and SDE make needed improvements in data collection and program assessment.

In view of the above, we recommend the third option--a shared-cost transitional approach. This option would result in the state "stepping down" its assistance, making it easier for the district to roll its ESY program into its entire summer school operations. One advantage to this approach is that it requires the district to allocate its resources for all its summer programs as efficiently as possible.

Under this approach, the district would receive funding for all students eligible for state-mandated summer school services (students retained or students at risk of failing the High School Exit Exam.) However, the district might have to draw upon its own resources to assure that the same level of summer school services (services not mandated by the state) can be provided to other students. The district will have additional discretionary funds in excess of $4 million annually beginning in the 2001-02 fiscal year, because it will have completed making annual debt service payments of that amount to the state in repayment of their $20 million loan. Finally, our recommended approach would provide Compton with additional resources for summer programs for a two-year period. After that two-year extension, Compton would be in a comparable state funding situation as other districts serving pupil populations with similar needs.

Data Collection and Evaluation

Throughout this report, we have noted the district's inability to provide complete and accurate data for use in this analysis. We believe this problem is systemic and rises to the level of an important issue regarding any future funding for the ESY program. We also note that the SDE had ultimate responsibility under the ESY statutes to assure that adequate testing and data collection took place. The SDE did not follow through on this responsibility.

The paramount goal of the ESY program is to improve student achievement. Without accurate and reliable data, there is no way of knowing if the program is effective or cost beneficial. Chapter 943 requires the use of standardized test results for all students prior to their participation in the ESY program and again at the end of their participation. Data collected from such tests would help determine grade level for retention and advancement, teacher capacity and performance, and location of the lowest-performing schools. The data also could indicate whether the program has been generally effective or not. The data could not--even if perfectly collected and analyzed--tell the Legislature whether this type of program is a useful "pilot" that should be replicated in other school districts. To answer this question would require a substantially more rigorous experiment, involving more districts, alternative types of remedial programs for the sake of comparison, and random assignments of schools, pupils, and teachers to these programs. If the Legislature decides to extend the program, we recommend language be added that dedicates a percentage of ESY program funding for testing, data collection, and evaluation by an independent agent approved by the SDE.

For the long term, the district needs to improve its data management system to facilitate both student and program evaluations. The district should consider participating in the California Student Information Services (CSIS) grant program to receive external assistance in developing its future student data management system. The CSIS is a multiyear effort to standardize the management of student records across the state to (1) facilitate records transfer, (2) improve districts' ability to use student level data in decision making, and (3) assist districts in state reporting. Compton could benefit greatly from such assistance.