Interim Evaluation of Community College Basic Skills Transformation Grants

Introduction

State Created Program to Improve Basic Skills Education at Community Colleges. Chapter 22 of 2015 (SB 81, Committee on Budget and Fiscal Review) created a grant program to improve basic skills education at the community colleges. Grant recipients were to implement certain evidence‑based improvement strategies. Chapter 22 required grant recipients to collect and report to the California Community Colleges (CCC) Chancellor’s Office a variety of information, including the types of strategies implemented, the size of cohorts served by the new strategies, and the outcomes for students served by these strategies. Chapter 22 also directed our office to evaluate the program’s effectiveness based upon the data compiled by the Chancellor’s Office. The legislation directed us to submit an interim evaluation by December 1, 2019 and a final evaluation by December 1, 2021. This brief fulfills the interim reporting requirement.

Background

In this section, we provide background on basic skills education at the community colleges, describe the Basic Skills Transformation Grants created in 2015, and describe more recent policy changes that require community colleges to make broader changes to the way they support students in completing transferable college‑level math and English courses.

Basic Skills Education

Community Colleges Offer Math and English Courses Below College Level. To obtain an associate degree and/or transfer to a four‑year university, students must complete college‑level courses in math and English (among other requirements). For students who are not deemed prepared for a college‑level course, colleges offer preparatory courses, including courses in elementary and secondary reading, writing, and math, as well as courses in English as a second language. Students deemed unprepared may be required to complete one or more of these basic skills courses before enrolling in college‑level courses. For example, a student whose educational goal requires completion of college statistics may need to first complete pre‑algebra, elementary algebra, and/or intermediate algebra. Basic skills courses do not apply towards meeting degree or transfer requirements.

Students Previously Assessed Based on Standardized Tests. Traditionally, students were assessed as prepared or unprepared for college‑level courses based on standardized tests. Students were required to take these tests prior to enrolling in community college courses. Students would then be placed into the appropriate course based on their test results. (As we discuss later in this section, these tests are no longer used due to recent policy changes.)

State Provides Ongoing Funding for Basic Skills Courses. Most basic skills funding is generated through apportionments, which provide funding to community college districts on a per‑student basis. In the past, the state also has provided categorical funding specifically for basic skills education. From 2007‑08 through 2017‑18, the state provided funding through the Basic Skills Initiative. Districts could use the funds from this program for counseling and tutoring students enrolled in basic skills courses, as well as designing curriculum and training the faculty teaching these courses. Over most of this period, the state provided $20 million annually for the initiative. In 2018‑19, this funding was consolidated into a larger block grant—the Student Equity and Achievement Program. Community colleges can use the $475 million provided through the block grant for basic skills, but they also can use the funds for other counseling and support, as well as for implementing strategies to close equity gaps. (As we discuss on the following page, the state has also provided one‑time grant funds for basic skills.)

Students Deemed Unprepared Are Less Likely to Graduate. Among degree, certificate, or transfer‑seeking students who entered the community college system in 2011‑12, only 25 percent were deemed prepared for college‑level coursework in both math and English. Prepared students have much higher completion rates than unprepared students. Of the students entering the system in 2011‑12, 70 percent of prepared students completed a degree, obtained a certificate, or transferred to a four‑year university within six years, compared to 41 percent of unprepared students. The share of students classified as prepared varies significantly by racial/ethnic group. The share of prepared students is relatively low for black and Latino students (13 percent and 20 percent, respectively), while over half of white and Asian students are classified as prepared. Age also is related to preparation, with younger students more likely to be classified as prepared upon entering the community college system.

Basic Skills Transformation Grants

State Funded One‑Time Basic Skills Transformation Grants. The 2015‑16 budget included $60 million one‑time Proposition 98 funding for grants to assist colleges in adopting or expanding the use of evidence‑based models for basic skills assessment, placement, instruction, and student support. Due to the large demand from colleges for these grants, the Legislature provided an additional $30 million one‑time Proposition 98 funding for this purpose in 2016‑17. A college could receive up to $1.5 million in grant funding that it could spend over three years. Colleges applied and were awarded funding through a competitive process.

Initiative Focused on Promoting Certain Evidence‑Based Improvement Strategies. Chapter 22 identified six evidence‑based strategies for improving basic skills education (Figure 1). It required grant recipients to adopt or expand the use of at least two of them. Colleges could use grant funding for implementing their selected strategies. Statute specifies the overarching intent of implementing the strategies is to increase the proportion of students who (1) complete college‑level math and English in a sequence of three or fewer courses and (2) earn an industry‑relevant degree or certificate within two years. These changes were driven by findings from community colleges in California and other states showing that certain basic skills strategies resulted in higher proportions of students completing college‑level math and English courses in shorter periods of time.

Total of 64 Colleges Received Grants. Of the 114 community colleges operating during the grant period, 85 colleges applied for grants. As part of the application process, colleges were required to describe their readiness to implement new strategies, set targets for the number of students to be served by these new strategies, and describe their plan for addressing the training and support faculty would need to implement these strategies. The CCC Chancellor’s Office deemed 64 of the 85 applications as meeting minimum requirements. Grant funding was distributed to these colleges in July 2016, with the expectation that colleges would begin implementation during the 2016‑17 academic year.

Recent Policy Changes

Subsequent Legislation Made Major Changes to Basic Skills Education. Two years after creating the Basic Skills Transformation Grants, the Legislature enacted Chapter 745 of 2017 (AB 705, Irwin). The legislation changed the rules for basic skills education in three main ways.

• Assessment. Colleges must now use students’ high school coursework, course grades, and/or grade point average (GPA) in making placement decisions. Colleges may not use standardized tests to determine student placement unless the assessment has been approved by the CCC Board of Governors. To date, the Board of Governors has not approved any assessment for this purpose.
• Placement. Colleges also must now make transferable college‑level courses the default placement for students. (Students may choose to enroll in courses below their default or recommended placement.) This change in state policy was driven by a growing research consensus that placing students directly into transferable college‑level courses was the most effective basic skills strategy implemented to date.
• Timing. Assembly Bill 705 sets the expectation that students complete transferable college‑level coursework within one year.

Chancellor’s Office Provided Implementing Guidance. In fall 2017, the CCC Chancellor’s Office began providing guidance to colleges in implementing AB 705. The Chancellor’s Office expects colleges to place all students that graduated high school within the past ten years directly into transferable college‑level math and English, but provide concurrent academic support to students with lower GPAs. The specific GPA cut‑offs are based on research conducted by the Multiple Measures Assessment Project, a collaborative of community college researchers and practitioners. In English, for example, the Chancellor’s Office recommends concurrent academic support for students with a high school GPA lower than 2.6. The Chancellor’s Office expects colleges to fully implement the new policies by fall 2019 for math and English courses.

New Law Has Implications for Colleges Receiving Basic Skills Transformation Grants. Because of the changes AB 705 makes to basic skills education, some colleges receiving Transformation Grants are having to rethink their basic skills approaches. Most notably, Transformation Grant recipients previously were allowed to focus on shortening basic skills course sequences (rather than placing students directly into college‑level courses). Transformation Grant recipients also originally had a less stringent timing expectation, with students encouraged to complete college‑level material within a three‑course sequence, typically taking a year and a half. Now, colleges are to de‑emphasize these longer course sequences, as well as have students complete college‑level work more quickly.

Findings

In this section, we first describe our findings regarding implementation of the Basic Skills Transformation Grants initiative. Chapter 22 required grant recipients to collect and report certain data from 2015‑16 through 2019‑20, with the CCC Chancellor’s Office required to compile that data annually. As noted later in this section, the data collected by the Chancellor’s Office was of poor quality and could not be used to evaluate the grant program. To complete our evaluation of the grant program, we spoke with several community college administrators and analyzed trends in student outcomes using statewide data collected by the Chancellor’s Office.

Implementation

Criteria Used for Determining Awardees Was Reasonable. Based on our review of the CCC Chancellor’s Office’s request for applications, we have no concerns with the criteria used for determining grant awardees. The information the Chancellor’s Office requested from colleges appears to be well aligned with the statutory requirements of the initiative. The Chancellor’s Office also appears to have made reasonable judgment calls in reviewing applications—rejecting those that did not include all required information and may, therefore, have been less well developed. For example, the Chancellor’s Office rejected applications that lacked information on the number of students currently served by evidence‑based basic skills strategies and the number of students the college was proposing to serve using the Transformation Grant funding.

Grant Recipients Had Somewhat Higher Completion Rates Than Other Colleges. Notably, the colleges receiving grants had slightly better student outcomes than those colleges with rejected applications or not applying. The percentage of first‑time students enrolled in fall 2015 who completed transferable college‑level English within one year was 24 percent for Transformation Grant recipients. This compares with 22 percent for colleges whose applications were rejected and 20 percent for colleges that did not apply for a Transformation Grant.

Data Collected by CCC Chancellor’s Office Is of Poor Quality. To collect the required data from grant recipients, the Chancellor’s Office developed a detailed spreadsheet for colleges to complete and submit once per year. The spreadsheet includes information on the strategies a college implemented and its student outcomes, disaggregated by race/ethnicity, high school GPA, and the student’s initial placement. In preparing our report, we had access to data submitted for the first two years of implementation (2016‑17 and 2017‑18). The spreadsheets shared with us, however, contained numerous errors and inconsistencies. As evident from the spreadsheets, the Chancellor’s Office did not thoroughly review the data or follow‑up with colleges to have them submit accurate data. Due to this poor data quality, we are unable to use this data to summarize the strategies used by colleges or evaluate the effects of the initiative. Colleges are required to report two additional years of data (2018‑19 and 2019‑20) for the final program evaluation due in 2021. Without greater oversight and quality control from the CCC Chancellor’s Office, this additional data also is likely to be unusable.

Colleges Used Funding Primarily for Curriculum Development and Training. Though colleges were required to report which evidence‑based strategies they implemented, they were not required to report how they spent their grant funds. Based on our conversations with colleges, they indicated they used grant funds primarily to modify existing math and English curricula, train faculty in using the redesigned curricula, and train staff in supporting students through their course sequences. These purposes are consistent with the Legislature’s intent for how colleges were to use the grant funds.

Some Colleges Did Not Sustain Activities When Grant Funds Were Exhausted. Some colleges indicated to us that they used their one‑time grant funds to hire additional support staff, with the goal of making some support services (such as tutoring) more accessible for students. In some cases, colleges shared with us that they had since found ongoing fund sources (such as the Student Equity and Achievement Program) to continue employing these staff. In other cases, colleges indicated they were not able to continue these services after their Transformation Grant funds expired.

Trends in Student Outcomes

More Students Are Being Placed Directly Into Transfer‑Level Courses. Over the last few months, several groups have released reports examining how community college course offerings have changed in response to basic skills reforms. Reports by the Public Policy Institute of California (PPIC) and the RP Group (a nonprofit organization operated by community college researchers) have found significant increases in the percentage of students being placed into transferable college‑level coursework. The PPIC report also found that students who enrolled in transferable college‑level coursework with co‑requisite support had higher completion rates than students who enrolled in basic skills courses. A Campaign for College Opportunity review of the fall 2019 course section offerings at 47 colleges also found a significant increase in transferable college‑level course offerings and co‑requisite courses. This report, however, identified a few concerns of implementation falling short. In particular, the report noted that some colleges continued to include a significant number of basic skills courses in their course catalogs and appeared to be offering too few courses in transferable college‑level statistics and quantitative reasoning—the most common math requirement for community college students.

Completion Rates for Transfer‑Level Math and English Courses Have Improved. Given the poor quality of the data reported by Transformation Grant recipients, we examined an alternative data source to track course completion rates. Specifically, we used the state’s main data system for all colleges—known as the Management Information System (MIS)—to examine trends in the share of students completing transferable college‑level math and English courses. Such analysis allows us to see trends in student outcomes, but we cannot link these outcomes directly with specific strategies grant recipients adopted. As Figure 2 shows, systemwide completion rates in both math and English courses have improved over the past five years. The proportion of first‑time students who enrolled and completed a transferable college‑level English course within one year increased from 18 percent in the fall 2013 cohort to 31 percent in the fall 2018 cohort. In math, these rates increased from 8 percent in the fall 2013 cohort to 14 percent in fall 2018. For both math and English, the largest improvements occurred in the fall 2018 cohort, which could be driven by colleges beginning early implementation of AB 705. As Figure 2 shows, completion rates increased for all racial/ethnic groups, although previously existing achievement gaps remain. We found no significant differences in improvement between Transformation Grant recipients and all other colleges.

Issues for Legislative Consideration

Final Evaluation of Transformation Grants Is No Longer Pertinent. When the Legislature authorized the evaluations of the Basic Skills Transformation Grant initiative back in 2015, it presumably desired to learn which of the six intervention strategies were most effective at improving student completion rates in transferable math and English courses. Now, with the enactment of AB 705 in 2017, all colleges must use the same two prescribed intervention strategies (high school grades and default placement into transferable courses). As a result, the evaluation focus has shifted to how effective these two (rather than six) strategies are at improving course completion and graduation rates. With these developments, the distinct effects of Transformation Grants can no longer be disentangled from the changes to basic skills education that all colleges are making as a result of AB 705.

Outcome Data for All Colleges Will Continue to Be Available Through MIS. Because all colleges must implement the new strategies, research groups have been using MIS data to examine the early effects of AB 705. The Chancellor’s Office collects MIS information annually and makes it publicly available. The Chancellor’s Office also is required to report some MIS data, including course completion and graduation rates, to the Legislature as part of an annual performance report.

Recommend Legislature Rethink Its Evaluation Goals. Given the enactment of AB 705 makes the final evaluation of the Transformation Grant initiative obsolete, we recommend the Legislature eliminate this evaluation requirement. If the Legislature is interested in seeing if AB 705 improves course completion and graduation rates, it can monitor outcomes using CCC’s annual performance report. As this information already is available annually, the Legislature could reconsider its evaluation goals to determine if any information not available through MIS might be of interest. If such information is identified, the Legislature then could authorize a new corresponding evaluation.