Center For Education Efficacy, Excellence, and Equity logo

Introduction

The question of whether summer vacation is detrimental to student learning has been the subject of education research for more than a century^{1White, W. (1906)}. Descriptions such as “summer slide” or “summer learning loss” denote that learning progress often slows—and, in many cases, declines—over the summer break^{2I use “summer learning” to refer to both gains and losses that may occur between spring and fall assessments. This choice aims to foreground the act of learning rather than the threat of “loss.”}. But these summer changes may
not look the same for all students: evidence suggests that some learn new
skills while others forget or lose mastery^{3Cooper, H., et al. (1996)}.

Some scholars estimate that students lose a quarter or more of the prior school year’s progress in math^{4Briggs, D. C., & Wellberg, S. (2022)} between spring and fall assessments—the equivalent of four steps forward, one step back.

Existing research has been mixed when it comes to whether learning slows or reverses during summers and, importantly, whether what happens between spring and fall assessment windows exacerbates opportunity gaps along socioeconomic or racial lines^{5Kuhfeld and Lewis (2023) provide a thorough summary of this debate; see also Leefelt (2015), Kuhfeld, (2019), and Workman, von Hippel, & Merry (2023).} . Moreover, enrichment at home or through school- or community-based initiatives may help to mitigate a slowdown in learning from spring to fall^{6von Hippel, P. T. et al. (2018).}.

Learning Trends in the Context of the Pandemic

Against this backdrop, the COVID-19 pandemic introduced not only a public health crisis but also an extended period of virtual learning for most students enrolled in schools in the United States (US)^{7A Year of COVID-19: What It Looked Like for Schools. Education Week (2021).}. Disruptions to schooling pale in comparison to the deep personal
losses that many young people, especially those in communities of color, experienced^{8Mackey, K. et al. (2021).}.

The 2022 release of scores on the National Assessment of Educational Progress (NAEP), known as “the Nation’s Report Card,” confirmed a historic decline in students’ math and reading performance, with steeper declines in math^{9That is, results from the National Assessment of Educational Progress (NAEP) test, taken by a nationally representative sample of fourth and eighth grade students.}. Analysis of district-level patterns reveals that the pandemic disproportionately affected learning in lower-income communities^{10 Work from Reardon and Kane at the Educational Opportunity Project; see https://educationrecoveryscorecard.org/.} . Likewise, analysis of spring data from curriculum and assessment providers indicates that students are behind where their same-grade peers were in the years preceding the pandemic^{11See Bahena & Morales (2023) and Kuhfeld & Lewis (2022) for more discussion of these data.}.

Using data from the i-Ready spring and fall diagnostics assessments of more than four million students from more than 9,000 schools, I examine scores and shifts in learning.

The analysis focuses on a subset of schools that used the i-Ready platform both prior to the pandemic (summer 2018 and 2019) and during at least one of pandemic-era summers (2021 and 2022). I include students who completed first through fourth grade in a given spring, for a sample with both spring and fall enrollments that correspond to traditional elementary school grades (i.e., grades 1-5).

In Sections 2 and 3, I present an overview of learning trends for math and reading in the summers of 2018 to 2022^{15I omit the summer of 2020 due to pandemic-related data limitations.} . I then examine whether patterns diverge for those who began the prior school year below rather than on grade level^{16I use “on grade level” to refer to any student whose performance is at least on grade level; this category includes students who score on or above grade level.}.

Summer Learning in Math & Reading

Math

In the figure at right, I look at spring-to-fall changes in scores as a measure of summer learning changes for the summers of 2018 and 2019 (pre-pandemic) alongside the summers of 2021 and 2022. Two changes shown in this graph are worth noting here.

First, scores are lower now than they were prior to the pandemic. This change exceeds the differences seen between spring and fall assessments (“summer learning losses”).

Second, the figure depicts dots for each spring and fall that are connected by a solid line: this line represents the change in score (loss or gain) between a given spring and fall. The lines slope downward in the pre-pandemic period, consistent with a decline in math scores over the summer. The average change in scores ranged from four to ten points, or roughly 20 to 40 percent of the school year’s gains^{17This is consistent with losses of .20 to .29 standard deviation units in the summer of 2019.} . Summer losses are smaller in 2021 and 2022. The lines that connect each set of dots are flatter than in pre-pandemic summers, consistent with losses of fewer than five points.

Together, these two observations point to an interesting trend in summer learning changes in the COVID-19 era: on average, students are scoring lower on spring math assessments than they did prior to the pandemic. But they do not appear to lose as much of their academic year learning over the summer as their same-age peers had in summers before 2020. This allows the average student to come back to school slightly closer to pre-pandemic levels than they were in the spring—though still below those levels.

Reading

 

Turning to reading, spring-to-fall score changes look quite different than they did for math. In general, reading scores do not change much between a given spring and the corresponding fall; as the figure at left shows, they slope slightly upward for some grades (especially those completing grade 1) and downward for others, with no clear pattern by age or grade.

Reading scores are slightly lower in the pandemic era compared to pre-pandemic; time trends in spring scores are discussed in greater detail in work from the E4 Center by Bahena and Morales that draws on a similar dataset.

Given that reading scores show limited variation between spring and fall across the timeframe I consider, I discuss summer learning in math over the summers of 2018-2022 in greater detail than I do summer learning in reading^{18I omit the summer of 2020 given data limitations with the spring 2020 assessment, as discussed in Curriculum Associates, 2020.}.

In Section 3, I group students based on whether they began the academic year on grade level or below grade level based on their fall diagnostic score.

Student Subgroups: On or Below Grade Level

Math

Students who performed below grade level on the fall i-Ready diagnostic prior to the pandemic experienced more academic-year growth, on average, than their peers who began the academic year on grade level. For example, the average first grade student who was below grade level in fall of 2018 saw their math score increase by 48 points between fall and spring assessments, compared to an average increase of 32 points among students who were on grade level (50 percent more growth). Put differently, students who began the academic year below grade level have spring scores that are still below, but not as far below, their on-grade-level peers at the end of the academic year.

It is encouraging to see the gap in math performance narrow over the course of the school year, between students who began the year on grade level and those who did not. What may be more surprising is that this gap continues to narrow over the summer. For example, fourth grade students who began the 2018-19 academic year below grade level in math scored, on average, 49 points below their peers in spring 2019 and 43 points below in fall 2019. While this summer shift is modest, it suggests some degree of “catching up” during both school years and summers.

In other words, these data tell us that, on average, students who began the year on grade level experience more summer learning loss than their peers who were below grade level. For example, fourth grade students who began the 2018-19 academic year below grade level in math experienced a summer learning loss of six points (21 percent of their academic-year gains), compared to their on-grade-level peers, who lost 11 points (44 percent of their academic-year gains). .

Summers of 2021 and 2022

The figures on page 6 and 7 show spring and fall scores for each grade and year that I consider, with separate lines for students who began the academic year below grade level and those who began on grade level.

As shown in Section 2, performance in the pandemic era is lower than it was prior to the pandemic, both for students who are on grade level and those who are not. The lines representing summers between each spring and fall assessment also appear flatter in 2021 and 2022 than pre-pandemic.

In turning to the question of how these subgroups of students have fared in light of the COVID-19 pandemic, we compare the performance of each group to the performance of their same-grade peers prior to the pandemic (see figures on page 7). That is, we compare the scores of students who scored below grade level in the fall of 2020 or 2021 to their same-grade peers who scored below grade level in the fall of 2018.

We see that students who began the 2021-22 school year on grade level finished that year
24 to 35% behind where their on-grade-level peers had scored on the math assessment in the spring of 2019. This pandemic-era gap then narrowed over the summer, with students returning to school 11 to 14 percent of an academic year behind where their same-grade peers had been in fall of 2019.

The figure above shows that between spring and fall of 2022 these students made even more progress towards eliminating pandemic-era learning losses, a shift that is suggestive of ongoing change over time ^{20It is possible that the pattern observed is entirely due to regression to the mean; but for it to explain the entirety of this pattern, the nature of this measurement error would need to have changed over time, a possibility that warrants further study. I see this as a plausible but relatively unlikely explanation.}.

The picture is a bit more concerning for students who returned to in-person learning below grade level in math. Their spring 2021 and 2022 math assessments were 15 to 22 percent
of an academic year behind where their below-grade-level peers had been in the spring of
2019. These gaps narrowed over the summer, to between six and 12 percent of an academic year behind where their below-grade-level peers had been in the fall of 2019.

These data suggest that, although students who began the year below grade level experience less summer learning loss than their on-grade-level peers, they are not making up as much ground over the summer as their peers did in prior years. Perhaps more pressing: I see no evidence that students who returned to in-person learning below grade level in math are starting to recover from pandemic-era unfinished learning.

If this trend were to continue, it could mean students who returned to in-person schooling below grade level continue to fall further behind their on-grade-level peers than I would have expected based on what I saw before the pandemic. Additionally, the figure above shows that the proportion of students who are performing below grade level in math has increased relative to pre-pandemic years. Together, these patterns suggest that the group of students whose performance is considered below grade level is growing.

Reading

As with reading scores discussed in Section I, I do not see evidence of substantial summer losses in reading in the 2018-2022 school years, either for students who began the academic year on grade level or for their peers who began the school year below grade level. For example, among students who finished second grade in spring 2019, those who had begun the year below grade level lost four points in reading; their peers who began the year on grade level lost two points (losses of nine percent and six percent of their academic-year gains, respectively). Among those who completed fourth grade in spring 2019, both groups returned to school in the fall with an average score less than one point above or below their spring performance.

The figure on the previous page depicts pandemic-era scores in terms of the growth each group’s same-grade peers experienced in 2018-19. We see a pattern that is quite similar to the one we observed for these subgroups in math.

In fact, the average student who began second through fourth grade reading on grade level in Fall 2021 returned to school in Fall 2022 performing ahead of where their same-grade peers were in reading in Fall 2019. Students who returned to in-person learning performing on grade level in reading have, on average, eliminated any pandemic-era unfinished learning.

The pattern for students who returned to in-person learning performing below grade level in reading is similar to what we saw in math. Students’ spring scores were 16 to 21 percent of an academic year behind where their below-grade-level peers had performed in 2018-19. These gaps narrowed slightly over the summer, to between 8 and 18 percent of an academic year behind where their below-grade-level peers had been in the fall of 2019.

Together, the math and reading data indicate that students who returned to in-person learning below grade level are, on average, not making continued progress towards pandemic-era unfinished learning. While gaps are narrowing between spring and fall assessments, this progress is not enough to get students back to where their below-grade-level peers were performing prior to the pandemic.

Conclusion

Summer learning trajectories, a topic of interest in education research even before the pandemic, have received heightened attention as states and districts work to support students who lost ground as a result of the COVID-19 pandemic. This report provides information about spring and fall scores both prior to the pandemic and in the summers of 2021 and 2022. I make several contributions:

Is it “summer slide”?
Our findings align with past research that has identified substantial summer learning loss in math, but not in reading^{22Briggs, D. C., & Wellberg, S. Evidence of “Summer Learning Loss” on the i-Ready Diagnostic Assessment.}. I observe a decrease in math scores between spring and fall assessments for almost every grade-year combination I examine. Also in line with past research, I do not find evidence of substantial learning losses in reading. I add to this literature by confirming that reading scores do not tend to decrease over the summer for both grade-level groups: those who began the academic year on grade level and those who began below grade level.

Pandemic-era unfinished learning
I provide a current look at the state of student learning, with data through the fall of 2022. These most recent data raise concerns that achievement has not fully rebounded from the pandemic-era unfinished learning^{23In contrast, after Hurricane Katrina, achievement was observed to rebound approximately two years after the natural disaster (Sacerdote, 2012). See also Kuhfeld, Soland, and Lewis, 2022.}. In the spring of 2021 and 2022, elementary school students performed 24 to 43 percent of an academic year behind where their same-grade peers were in math in spring 2019. This gap was roughly reduced by half over the summer (to 13 to 21 percent of an academic year). However, I see no indication of a trend towards eliminating this pandemic-era unfinished learning; that is, math scores in 2021-22 were not substantially higher than those observed in 2020-21.

Recovery for students on, but not below, grade level
The most concerning element of these findings comes when we look at 2021 and 2022 progress, relative to pre-pandemic, based on whether students scored on, versus below, grade level at the start of each school year. We see that students who returned to in-person learning performing on grade level have made progress towards eliminating pandemic-era unfinished learning in both math and reading.

In contrast, the growing group of students who began the 2020-21 or 2021-22 school year performing below grade level remains behind where their below-grade-level peers performed in 2018-19. This pandemic-era gap is present in both math (six to 12 percent of an academic year) and reading (eight to 18 percent of an academic year).

Looking ahead
Kuhfeld and her colleagues report that gaps in performance may have worsened in the second, compared to the first, pandemic school year^{25Kuhfeld, Soland, & Lewis (2022)}. Our findings are similar. However, it is important to note that we are still relatively early in the pandemic recovery era. We urge caution in interpreting these findings as representative of a “post-pandemic” period, given that the virus continued to infect and kill many Americans throughout the time period that our data cover^{26Data from the World Health Organization confirm that the US experienced more than 1,000 confirmed COVID-related deaths per week through the end of 2022.} .

While additional research is needed, including from the 2022-23 school year and beyond, policymakers and education leaders should be on the lookout for whether learning progress is moving in the right direction. It is clear that our most vulnerable students may need enhanced supports to recover from pandemic-era unfinished learning.

Making connections: School contexts and social policy
Coronavirus recovery legislation included nearly 200 billion dollars allocated to K-12 education^{27Jordan, P. (2022)}, of which roughly half had been spent by early 2023^{28Refers to funds being obligated (under contract for a specific purpose)}. The Department of Education reports that these funds supported the expansion of summer learning programming in all 50 states, Washington, D.C., and Puerto Rico. These supports may have helped to reduce summer learning losses that might otherwise have been larger in the summers of 2021 and 2022.

Summer programming tends to prioritize the enrollment of students with lower grades or standardized test scores. But there are reasons to worry that access is not equitable, including barriers related to transportation, awareness of application processes, and language accessibility of communications with parents and caregivers ^{29See Are Summer School Programs Reaching Students Who Need Them Most? and New Data: Was 2022’s Summer Learning ‘Explosion’ Enough To Reverse COVID Losses?}.

Let us know

How can school districts ensure that expanded services reach the students who need them most?

Which sources of enrichment work best to address unfinished learning?
At E4, we are eager to learn more about the impacts of high-dosage tutoring and whether certain domains of math learning have promising impacts on overall progress.

What are the bright spots from summer learning programming?
Can they be replicated in other states or districts?

Why do you think I see different trends in reading and math?

Are summer losses in math larger in particular subdomains (e.g., algebra or word problems)?
Are there certain types of questions that students answered correctly in the spring but struggled with in the fall?

Are there other aspects of summer learning we should think about?

Get in touch at e4center@northwestern.edu and on Twitter at @E4Center_NU.