Good News For New Orleans - Douglas Harris - Education Next
View this article online here: http://educationnext.org/good-news-new-orleans-evidence-reform-student-achievement/
Content
feature
EM
GOOD NEWS FOR
NEWORLEANS
BA
Early evidence shows reforms
lifting student achievement
O
ED
PHOTOGRAPH / CHRIS GRANGER,THE TIMES-PICAYUNE, LANDOV
G
R
What happened to the New Orleans public schools
following the tragic levee breeches after Hurricane Katrina
is truly unprecedented. Within the span of one year, all
public-school employees were fired, the teacher contract
expired and was not replaced, and most attendance zones
were eliminated. The state took control of almost all public
schools and began holding them to relatively strict standards
of academic achievement. Over time, the state turned all the
schools under its authority over to charter management
organizations (CMOs) that, in turn, dramatically reshaped
the teacher workforce.
A few states and districts nationally have experimented
with one or two of these reforms; many states have increased
the number of charter schools, for example. But no city had
gone as far on any one of these dimensions or considered
trying all of them at once. New Orleans essentially erased its
traditional school district and started over. In the process,
the city has provided the first direct test of an alternative to
the system that has dominated American public education
for more than a century.
Dozens of districts around the country are citing the
New Orleans experience to justify their own reforms. In
addition to being hailed by Democratic president Barack
Obama and Louisiana’s Republican governor, Bobby Jindal,
parliamentary delegations from at least two countries have
visited the city to learn about its schools.
by DOUGLAS N. HARRIS
8
ED U C A T I O N N E X T / F A L L 2 0 1 5
educationnext.org
R
BA
EM
A New Trajectory (Figure 1)
G
Following Hurricane Katrina and the reforms enacted in its aftermath, the
performance of New Orleans students rose steadily compared to that of
students in other Louisiana districts that were affected by the hurricane.
O
ED
State
Average
Scale test score
(standard deviation)
-0.1
-0.2
Hurricane
Katrina hits
New Orleans
-0.3
-0.4
-0.5
-0.6
-0.7
2002 2003 2004 2005 2006 2007 2008 2009 2010
New Orleans
2011
2012
Matched Comparison
NOTES: Separate analyses show that New Orleans students returning after the storm,
who could be studied only through 2009, also made gains relative to their own prior
performance, but these differences were often not statistically significant. Scale scores
are averaged across grades 3 through 8 and across English Language Arts, math,
science, and social studies. Scale scores are standardized so that zero refers to the
statewide average.
SOURCE: Author’s calculations
educationnext.org
F A L L 2 0 1 5 / E DU CA TION NEX T
9
Before the Storm
BA
EM
The unprecedented nature of the reforms and level of
national and international attention by themselves make the
New Orleans experience a worthy topic of analysis and debate.
But also consider that the underlying principles are what many
reformers have dreamed about for decades—that schools would
be freed from most district and union contract rules and allowed
to innovate. They would be held accountable not for compliance
but for results.
There is clearly a lot of hype. The question is, are the
reforms living up to it? Specifically, how did the reforms
affect school practices and student learning? My colleagues
and I at the Education Research Alliance for New Orleans
(ERA-New Orleans) at Tulane University have carried out a
series of studies to answer these and other questions. Our work
is motivated by the sheer scale of the Katrina tragedy and the
goal of supporting students, educators, and city leaders in their
efforts to make the city’s schools part of the city’s revitalization
effort. The rest of the country wants to know how well the New
Orleans school reforms have worked. But the residents of New
Orleans deserve to know. Here’s what we can tell them so far.
R
Assessing the effects of this policy experiment involves comparing the effectiveness of New Orleans schools before and after
the reforms. As in most districts, before Hurricane Katrina, an
elected board set New Orleans district policies and selected
superintendents, who hired principals to run schools. Principals
hired teachers, who worked under a union contract. Students
were assigned to schools based mainly on attendance zones.
The New Orleans public school district was highly dysfunctional. In 2003, a private investigator found that the district
system, which had about 8,000 employees, inappropriately
provided checks to nearly 4,000 people and health insurance
to 2,000 people. In 2004, the Federal Bureau of Investigation
(FBI) issued indictments against 11 people for criminal
offenses against the district related to financial mismanagement. Eight superintendents served between 1998 and 2005,
lasting on average just 11 months.
This dysfunction, combined with the socioeconomic background of city residents—83 percent of students were eligible
for free or reduced-price lunch—contributed to poor academic
results. In the 2004â05 school year, Orleans Parish public
schools ranked 67th out of 68 Louisiana districts in math and
reading test scores. The graduation rate was 56 percent, at least
10 percentage points below the state average.
As a result, some reforms were already under way when
Katrina hit in August 2005. The state-run Recovery School
District (RSD) had already been created to take over lowperforming New Orleans schools. The state had appointed an
emergency financial manager to handle the district’s finances.
There were some signs of improvement in student outcomes
just before the storm, but, as we will see, these were relatively
modest compared with what came next.
G
A Massive Experiment
O
After Katrina, state leaders quickly moved almost all public
schools under the umbrella of the RSD, leaving the higher-performing ones under the Orleans Parish School Board (OPSB).
Gradually, the RSD turned
schools over to charter operators,
and the teacher workforce shifted
toward alternatively prepared
teachers from Teach for America
and other programs. So new was
the system that a new name was
required—longtime education
reformer Paul Hill called it the
“portfolio” model.
Researchers often refer to
such sudden changes as “natural experiments” and study
them using a technique called
“difference-in-differences.” The
idea is to first take the difference between outcomes before
and after the policy, in the place
where it was implemented—the
treatment group. This first difGradually, the RSD turned schools over to charter operators, and the teacher workforce shifted
ference is insufficient, however,
toward alternatively prepared teachers from Teach for America and other programs.
because other factors may have
ED U C A T I O N N E X T / F A L L 2 0 1 5
educationnext.org
PHOTOGRAPH / JOHN MCCUSKER,THE TIMES-PICAYUNE,LANDOV
ED
10
feature
NEW ORLEANS REFORMS HARRIS
In the school year just prior to Hurricane Katrina,
Orleans Parish public schools ranked 67th out of 68
Louisiana districts in math and reading test scores.
were moving in parallel before the reforms, however, suggesting that our matching process produced a comparison group
that is more appropriate than the state as a whole.
The performance of New Orleans students shot upward
after the reforms. In contrast, the comparison group largely
continued its prior trajectory. Between 2005 and 2012, the
performance gap between New Orleans and the comparison
group closed and eventually reversed, indicating a positive
effect of the reforms of about 0.4 standard deviations, enough to
improve a typical student’s performance by 15 percentile points.
The estimates we obtain when we focus just on returnees
are smaller and often not statistically significant, although the
discrepancies are predictable: first, the returnees were probably
more negatively affected by trauma and disruption; second,
creating a new school system from scratch takes time, so we
would expect any effects to be larger in later years; and third,
the effects of the reforms seem more positive in early elementary
grades, and the returnees were generally in middle school when
they returned. Even so, the combination of analyses suggests
effects of at least 0.2 standard deviations, or enough to improve
a typical student’s performance by 8 percentage points.
But there is still the possibility that what appear to be
reform effects are actually the result of other factors.
G
R
BA
EM
affected the treatment group at the same time. This calls for
making the same before-and-after comparison in a group
that is identical, except for being unaffected by the treatment.
Subtracting these two—taking the difference of the two differences between the treatment and comparison groups—yields
a credible estimate of the policy effect.
We have carried out two difference-in-differences strategies:
1) Returnees only. We study only those students who
returned to New Orleans after Hurricane Katrina. The advantage of this approach is that it compares the same students over
time. One disadvantage is that it omits nonreturnees. Also, we
can only study returnees over a short period of time—after
2009, they no longer have measurable outcomes to study.
2) Different cohorts. We consider the achievement growth
of different cohorts of students before and after the reforms—
for example, students in 3rd grade in 2005 and students in 3rd
grade in 2012. The advantages here are that we can include
both returnees and nonreturnees, and we can use this strategy
to study longer-term effects. But the students are no longer
the same.
In both strategies, the New Orleans data set includes all publicly funded schools in the city, including those governed by
the district (OPSB), since all public schools were influenced by
the reforms. The main comparison group includes other districts
in Louisiana that were affected by Hurricane Katrina, and by
Hurricane Rita, which came soon afterward. This helps account
for at least some of the trauma and disruption caused by the
storms, the quality of schools students attended in other regions
while their local schools were closed, and any changes in the state
tests and state education policies that affected both groups.
educationnext.org
ED
Figure 1 shows the scores for each cohort, separately for
New Orleans and the matched comparison group. The scores
cover grades 3 through 8, are averaged across subjects, and are
standardized so that zero refers to the statewide mean. The
first thing to notice is that before the reforms, students in New
Orleans performed far below the Louisiana average, at about
the 30th percentile statewide. Students from the comparison
districts also lagged behind the rest of the state, but by a lesser
amount. The New Orleans students and the comparison group
O
Effects on Average Achievement
Addressing Additional Concerns
The goal of any analysis like this is to rule out explanations for the changes in outcomes other than the reforms
themselves. Our main comparisons deal with many potential
problems, such as changes in state tests and policies. Here we
consider in more depth four specific factors that could bias the
estimated effects on achievement: population change, interim
school effects, hurricane-related trauma and disruption, and
test-based accountability distortions.
Population change. Hurricane Katrina forced almost everyone to leave the city. Some returned and some did not. The
most heavily flooded neighborhoods were (not coincidentally)
those where family incomes were lowest, and people in these
neighborhoods returned at much lower rates than people who
lived in other parts of the city. Given the strong correlation
between poverty and student outcomes, this could mean that
higher test scores shown in Figure 1 are driven not by the
F A L L 2 0 1 5 / E DU CA TION NEX T
11
There was an upward change in the trajectory of student
test scores in New Orleans after the reforms of about 0.2
to 0.4 standard deviations, enough to improve a typical
student’s performance by 8 to 15 percentile points.
but we can estimate the combination of the two. A study by
the RAND Corporation of students from Louisiana districts
affected by the hurricane suggests that these two factors had
a short-term net negative effect on evacuees’ performance of
0.03 to 0.06 standard deviations. Our analysis suggests that
the negative influence is even larger for New Orleans students,
most likely because of the more extensive destruction in the
city compared with most other areas along the state’s coast.
Thus, at least in the years just after the reforms, the factors
pushing student outcomes down were at least as large as the
population changes pushing them up.
Test-based accountability distortions. One key part of the
New Orleans reforms was the idea that the state would shut
down schools within three to five years if they did not generate
a high enough School Performance Score, a measure based on
test scores and graduation rates. Prior research suggests that
such intensive test-based accountability can lead to behaviors, such as teaching to the test, that increase scores without
improvements in underlying learning or through reduced
learning in nontested subjects.
To address this problem, we estimate effects separately
by subject, recognizing that the stakes attached to math and
language scores were roughly double the stakes for science
and social studies scores during the period under analysis.
Also, the state’s social promotion policy raises the stakes for
students in grades 4 and 8. We find no evidence that the size
of effects varied systematically with the stakes attached to
the subjects or grades. However, it is hard to rule out other
potential test-based accountability distortions with our data.
As further evidence, we considered descriptive information
on nontest outcomes. State government reports indicate that,
relative to the state as a whole, the New Orleans high school
graduation rate and college entry rate (among high school
graduates) rose 10 and 14 percentage points, respectively.
So, in theory, there are many challenges to estimating the
effects of the New Orleans package of school reforms. The
combined effect of these alternative factors on long-term
achievement gains appears small, however, especially when
compared with our initial estimate of the reform effects.
There is a clear pattern across these methods. The estimates
are consistently within the same range, and even the lower end
of that range suggests large positive effects.
G
R
BA
EM
reforms but by schools serving more-advantaged students.
Observers have pointed out that the share of the student
population eligible for free or reduced-price lunch (FRL) actually increased slightly in New Orleans after the storm. But there
are many reasons not to trust FRL data. For example, they reflect
crude yes/no measures and are unlikely to capture extreme poverty of the sort common in New Orleans. Also, what really matters here is not whether poverty increased in New Orleans, but
whether poverty increased more than in the comparison group.
Therefore, in addition, we gathered data from the U.S. Census,
which measures changes in income and the percentages of the
population with various levels of education. We also carried
out the difference-in-differences analysis in these demographic
measures to understand the changes in New Orleans relative to
the matched comparison group of hurricane-affected districts,
and then simulated the effect of changes in family background
characteristics on test scores using data from the federal Early
Childhood Longitudinal Study.
We also examined pre-Katrina characteristics to see whether
the returnees were different from nonreturnees and found that
returnees did have slightly higher scores. In fact, we come to
the same conclusion in both analyses: the expected increase in
student outcomes after the hurricanes due to population change
is no more than 0.02 to 0.06 standard deviations, or about 10
percent of the difference-in-differences estimates in Figure 1.
Interim school effects. Some of the changes in student learning may reflect neither the prestorm nor poststorm quality
of New Orleans schools, but the performance of schools that
students briefly attended outside the city after the evacuation.
Other research on these students by Dartmouth economist
Bruce Sacerdote suggests that New Orleans evacuees experienced larger improvements in school quality than evacuees
from other districts.
Trauma and disruption. Any benefit of having good interim
schools might be offset by the trauma and disruption of the
storm itself and its aftermath. The majority of New Orleans
returnees probably knew someone among the nearly 2,000
people who died in the Katrina aftermath. Also, almost all students experienced significant disruption, moving to unfamiliar
neighborhoods and schools for extended periods. Reports of
post-traumatic stress disorder remain common.
It is difficult to isolate trauma and interim school effects,
ED
ED U C A T I O N N E X T / F A L L 2 0 1 5
O
12
educationnext.org
feature
NEW ORLEANS REFORMS HARRIS
Equity of Outcomes
G
R
BA
EM
be hard to say the outcomes from the New Orleans reforms
Public schools exist to ensure that all children have an are inequitable relative to what came before them. That said,
opportunity to succeed in life. Thus we consider not only the they were highly inequitable to start with, and there is clearly
average effects of the reform package, but also whether the room for improvement.
most-disadvantaged students benefited.
We first define equity in terms of how New Orleans, as an
urban district, performed relative to districts serving more- What Really Changed?
To help improve the schools going forward, it is important
advantaged students. Both before and after the reforms, at least
80 percent of New Orleans students were minority or eligible to know how school practices and other intermediate outcomes
for FRL. It is therefore noteworthy that the reforms brought changed. In a series of 15 ongoing studies, my collaborators at
the city’s students near to the state average on a wide range of ERA-New Orleans and I have examined four main components
of the reforms: choice and competition, teachers and leaders,
academic outcomes (see Figure 1).
It is also important to consider the distribution of effects within charters and CMOs, and test-based accountability.
Some of the reform effect may be driven by parental choice
the city, and here the results are more mixed. All major subgroups
of students—African American, low-income,
special education, and English Language
Learners (ELL)—were at least as well-off after
the reforms, in terms of achievement. Critics of
charter schools express concern about possible
increases in racial isolation (some would say
“segregation”). Among all of the various subgroups we considered, only Hispanic students
seem to have experienced increases in isolation.
There have also been concerns about schools
unfairly targeting low-income and African
American students in disciplinary decisions.
While we have not yet studied whether any
student groups have been specifically targeted,
we can say that the number of suspensions and
expulsions has dropped since the reforms, for
African American students and others alike.
There are a few less-positive signs, however. In our analysis of what families look
for when choosing schools, we found that
the lowest-income families place less weight
on the School Performance Score than other
families. Their circumstances may lead them
to focus more on practical considerations
such as distance to school and extended hours
(to avoid extra child-care costs). Similarly, in
our analysis of student mobility, we see that
low-scoring students are less likely than highscoring students to migrate toward schools
with high scores. Finally, up until a few years
ago, principals reported cherry-picking students by, for example, counseling out students
deemed poor fits and holding invitation-only
events to attract certain students.
Given the large improvements in average
outcomes in a district that is almost entirely
low-income and minority, and the mixed In terms of achievement, all major subgroups of students were at least as well-off
evidence on other equity indicators, it would after the reforms.
O
PHOTOGRAPH / SUSAN POAG,THE TIMES-PICAYUNE,LANDOV
ED
educationnext.org
F A L L 2 0 1 5 / E DU CA TION NEX T
13
and competition. The supply of schools in New Orleans appears
highly differentiated. Some schools specialize in math and science, others in the arts. Some schools offer language immersion
programs, while other schools have fairly traditional curricula.
Some schools have selective admissions, while others are open
enrollment or seek diverse student bodies. We also find that
New Orleans families diverge in their schooling preferences,
so having this degree of differentiation in schooling options is
likely to help match what families want with what schools offer
(see “The New Orleans OneApp,” features, and “Many Options
in New Orleans Choice System,” research, Fall 2015).
It is still unclear, however, whether these changes in the
market have contributed to the improvements in student outcomes. Even supporters of the reform efforts sometimes bristle
when I use the word “market” and “competition” to describe
the new system. Instead, they point to two other parts of the
reform package: the authority of the state to close schools and
the authority schools have over their teaching staffs.
Sixteen New Orleans schools have been completely closed
and another 30 have been taken over in some fashion by either
the RSD or OPSB—a large number in a city that has only about
90 public schools in total. Consistent with written state policies,
G
R
BA
EM
we find that the School Performance Score is the strongest measurable driver of closure and renewal decisions. Moreover, in
finding CMOs to open new schools and take over old ones, the
RSD has preferred those with a track record of academic success.
School leaders in New Orleans talk frequently about how
critical flexibility in personnel management is to their overall
school success. Free of state and local mandates and constraints
from union contracts, leaders reopening schools after the
storm could hire anyone they wanted, including uncertified
teachers, and dismiss teachers relatively easily. As CMOs took
over, more of the teacher workforce came from alternative
preparation programs such as Teach for America and The
New Teacher Project. Consistent with some other studies,
analyses commissioned by the state suggest that graduates of
these programs contribute more to student achievement than
graduates of traditional preparation programs.
The combination of policies had two types of effects on the
teacher workforce. First, the percentages of teachers with regular
certification and with 20 or more years of experience dropped
by about 20 points each. Also, due to both the short-term commitments of some alternatively certified teachers and school
autonomy over personnel, the teacher turnover rate nearly
doubled. The fact that such
large improvements in student
learning could be achieved with
these common metrics going in
the “wrong direction” reinforces
a common finding in education
research: teacher credentials and
turnover are not always good
barometers of effectiveness.
Finally, we turn to a topic
that is not typically thought of
as part of the reform package
but may be an essential component: costs and resources.
Our analysis suggests that from
2004â05 to 2011â12, the same
years covered by our achievement analysis, total public
schooling expenditures per
student increased by $1,000 in
New Orleans relative to other
districts in the state. Some of
the increase probably reflects
one-time start-up costs of new
schools, and we are working
to understand what share falls
in that category. Regardless,
there is wide agreement that the
Relative to the state as a whole, the New Orleans high school graduation rate rose 10 percentage
reforms did not come cheap.
points after the New Orleans reforms.
None of this really tells
O
ED U C A T I O N N E X T / F A L L 2 0 1 5
educationnext.org
PHOTOGRAPH / KATHLEEN FLYNN,THE TIMES-PICAYUNE,LANDOV
ED
14
feature
NEW ORLEANS REFORMS HARRIS
The effects of the reforms in New Orleans are large compared
with other completely different strategies for school improvement,
such as class-size reduction and intensive preschool.
us exactly which of the factors drove the improvements in
student outcomes—no doubt they are interconnected—but
it does provide some indication of how schools and families
responded to the policy shift.
EM
Implications for New Orleans
G
R
BA
These findings have important implications for the New
Orleans public schools, the many other urban districts pursuing the portfolio approach, and for the state and federal
policies—especially test-based and market-based accountability—from which the New Orleans reforms emerged.
For New Orleans, the news on average student outcomes
is quite positive by just about any measure. The reforms seem
to have moved the average student up by 0.2 to 0.4 standard
deviations and boosted rates of high school graduation and
college entry. We are not aware of any other districts that have
made such large improvements in such a short time.
The effects are also large compared with other completely
different strategies for school improvement, such as class-size
reduction and intensive preschool. This seems true even after
we account for the higher costs. While it might seem hard
to compare such different strategies, the heart of the larger
school-reform debate is between systemic reforms like the
portfolio model and resource-oriented strategies.
With the possible exception of distortions from test-based
accountability, which are harder to identify, the reforms managed to avoid most of the side effects that many feared. But our
findings also suggest areas of potential improvement. While
the reforms have been successful on some dimensions of
equity, it seems necessary to do more to ensure that all groups
within the city benefit. All types of public school systems
struggle with providing equitable access to quality schools,
and the New Orleans system is no exception.
student test scores made it the second-lowest-ranked district
in the second-lowest-ranked state in the country.
New Orleans is an attractive city for young educators. The
national response to the hurricane aftermath was heartening,
and for many young people, contributing to the rebuilding
effort became a calling. Later, as the reform effort took hold,
New Orleans also became the nation’s epicenter of school
reform, an ideal place for aspiring reform-minded educators.
Because the city is smaller than many urban districts, school
leaders could be very selective in choosing from the pool of
educators who wanted to come and work there.
The effects might also be smaller, at least in the short run,
if the reforms were adopted on a statewide basis, because the
reform is dependent on a specific supply of teachers. It seems
difficult enough attracting effective teachers and leaders to
work long hours at modest salaries in New Orleans; doing it
throughout Louisiana is unrealistic without a major change in
the educator labor market. Nonetheless, it would be a mistake
to dismiss the relevance of the New Orleans experience for
others. It is relevant precisely because it is so unusual. The
city’s reforms force us to question basic assumptions about
what Kâ12 publicly funded education can and should look like.
There is more to the debate than we can cover here, including
fundamental philosophical issues about whose objectives and
values should count in making schooling decisions. But there
is also wide agreement that the academic outcomes considered
here are important, so learning how much the reforms contribute to changes in academic measures should also be a key part
of the conversation. Better understanding of all the elements of
the reforms is something we owe to the city, its children, and
everyone who suffered and perished in this terrible tragedy.
educationnext.org
ED
Unfortunately, the effects of even the most successful programs are often not replicated when tried elsewhere, and there
are good reasons to think the conditions were especially ripe
for success in New Orleans:
There was nowhere to go but up. Pre-Katrina, the New
Orleans public school system was highly dysfunctional, and
O
Implications for the Nation
Douglas N. Harris is professor of economics at Tulane
University and founder and director of the Education
Research Alliance for New Orleans. The research cited here
is coauthored with others on the ERA-New Orleans research
staff (Paula Arce-Trigatti, Nathan Barrett, Lindsay Bell
Weixler, Christian Buerger, Matthew Larsen, Jane Arnold
Lincove, Whitney Ruble, Robert Santillano, and Jon Valant)
and members of the ERA-New Orleans National Research
Team (Huriya Jabbar, Jennifer Jennings, Spiro Maroulis,
Katharine Strunk, Patrick Wolf, and Ron Zimmer).
All errors are the author’s.
F A L L 2 0 1 5 / E DU CA TION NEX T
15
EM
GOOD NEWS FOR
NEWORLEANS
BA
Early evidence shows reforms
lifting student achievement
O
ED
PHOTOGRAPH / CHRIS GRANGER,THE TIMES-PICAYUNE, LANDOV
G
R
What happened to the New Orleans public schools
following the tragic levee breeches after Hurricane Katrina
is truly unprecedented. Within the span of one year, all
public-school employees were fired, the teacher contract
expired and was not replaced, and most attendance zones
were eliminated. The state took control of almost all public
schools and began holding them to relatively strict standards
of academic achievement. Over time, the state turned all the
schools under its authority over to charter management
organizations (CMOs) that, in turn, dramatically reshaped
the teacher workforce.
A few states and districts nationally have experimented
with one or two of these reforms; many states have increased
the number of charter schools, for example. But no city had
gone as far on any one of these dimensions or considered
trying all of them at once. New Orleans essentially erased its
traditional school district and started over. In the process,
the city has provided the first direct test of an alternative to
the system that has dominated American public education
for more than a century.
Dozens of districts around the country are citing the
New Orleans experience to justify their own reforms. In
addition to being hailed by Democratic president Barack
Obama and Louisiana’s Republican governor, Bobby Jindal,
parliamentary delegations from at least two countries have
visited the city to learn about its schools.
by DOUGLAS N. HARRIS
8
ED U C A T I O N N E X T / F A L L 2 0 1 5
educationnext.org
R
BA
EM
A New Trajectory (Figure 1)
G
Following Hurricane Katrina and the reforms enacted in its aftermath, the
performance of New Orleans students rose steadily compared to that of
students in other Louisiana districts that were affected by the hurricane.
O
ED
State
Average
Scale test score
(standard deviation)
-0.1
-0.2
Hurricane
Katrina hits
New Orleans
-0.3
-0.4
-0.5
-0.6
-0.7
2002 2003 2004 2005 2006 2007 2008 2009 2010
New Orleans
2011
2012
Matched Comparison
NOTES: Separate analyses show that New Orleans students returning after the storm,
who could be studied only through 2009, also made gains relative to their own prior
performance, but these differences were often not statistically significant. Scale scores
are averaged across grades 3 through 8 and across English Language Arts, math,
science, and social studies. Scale scores are standardized so that zero refers to the
statewide average.
SOURCE: Author’s calculations
educationnext.org
F A L L 2 0 1 5 / E DU CA TION NEX T
9
Before the Storm
BA
EM
The unprecedented nature of the reforms and level of
national and international attention by themselves make the
New Orleans experience a worthy topic of analysis and debate.
But also consider that the underlying principles are what many
reformers have dreamed about for decades—that schools would
be freed from most district and union contract rules and allowed
to innovate. They would be held accountable not for compliance
but for results.
There is clearly a lot of hype. The question is, are the
reforms living up to it? Specifically, how did the reforms
affect school practices and student learning? My colleagues
and I at the Education Research Alliance for New Orleans
(ERA-New Orleans) at Tulane University have carried out a
series of studies to answer these and other questions. Our work
is motivated by the sheer scale of the Katrina tragedy and the
goal of supporting students, educators, and city leaders in their
efforts to make the city’s schools part of the city’s revitalization
effort. The rest of the country wants to know how well the New
Orleans school reforms have worked. But the residents of New
Orleans deserve to know. Here’s what we can tell them so far.
R
Assessing the effects of this policy experiment involves comparing the effectiveness of New Orleans schools before and after
the reforms. As in most districts, before Hurricane Katrina, an
elected board set New Orleans district policies and selected
superintendents, who hired principals to run schools. Principals
hired teachers, who worked under a union contract. Students
were assigned to schools based mainly on attendance zones.
The New Orleans public school district was highly dysfunctional. In 2003, a private investigator found that the district
system, which had about 8,000 employees, inappropriately
provided checks to nearly 4,000 people and health insurance
to 2,000 people. In 2004, the Federal Bureau of Investigation
(FBI) issued indictments against 11 people for criminal
offenses against the district related to financial mismanagement. Eight superintendents served between 1998 and 2005,
lasting on average just 11 months.
This dysfunction, combined with the socioeconomic background of city residents—83 percent of students were eligible
for free or reduced-price lunch—contributed to poor academic
results. In the 2004â05 school year, Orleans Parish public
schools ranked 67th out of 68 Louisiana districts in math and
reading test scores. The graduation rate was 56 percent, at least
10 percentage points below the state average.
As a result, some reforms were already under way when
Katrina hit in August 2005. The state-run Recovery School
District (RSD) had already been created to take over lowperforming New Orleans schools. The state had appointed an
emergency financial manager to handle the district’s finances.
There were some signs of improvement in student outcomes
just before the storm, but, as we will see, these were relatively
modest compared with what came next.
G
A Massive Experiment
O
After Katrina, state leaders quickly moved almost all public
schools under the umbrella of the RSD, leaving the higher-performing ones under the Orleans Parish School Board (OPSB).
Gradually, the RSD turned
schools over to charter operators,
and the teacher workforce shifted
toward alternatively prepared
teachers from Teach for America
and other programs. So new was
the system that a new name was
required—longtime education
reformer Paul Hill called it the
“portfolio” model.
Researchers often refer to
such sudden changes as “natural experiments” and study
them using a technique called
“difference-in-differences.” The
idea is to first take the difference between outcomes before
and after the policy, in the place
where it was implemented—the
treatment group. This first difGradually, the RSD turned schools over to charter operators, and the teacher workforce shifted
ference is insufficient, however,
toward alternatively prepared teachers from Teach for America and other programs.
because other factors may have
ED U C A T I O N N E X T / F A L L 2 0 1 5
educationnext.org
PHOTOGRAPH / JOHN MCCUSKER,THE TIMES-PICAYUNE,LANDOV
ED
10
feature
NEW ORLEANS REFORMS HARRIS
In the school year just prior to Hurricane Katrina,
Orleans Parish public schools ranked 67th out of 68
Louisiana districts in math and reading test scores.
were moving in parallel before the reforms, however, suggesting that our matching process produced a comparison group
that is more appropriate than the state as a whole.
The performance of New Orleans students shot upward
after the reforms. In contrast, the comparison group largely
continued its prior trajectory. Between 2005 and 2012, the
performance gap between New Orleans and the comparison
group closed and eventually reversed, indicating a positive
effect of the reforms of about 0.4 standard deviations, enough to
improve a typical student’s performance by 15 percentile points.
The estimates we obtain when we focus just on returnees
are smaller and often not statistically significant, although the
discrepancies are predictable: first, the returnees were probably
more negatively affected by trauma and disruption; second,
creating a new school system from scratch takes time, so we
would expect any effects to be larger in later years; and third,
the effects of the reforms seem more positive in early elementary
grades, and the returnees were generally in middle school when
they returned. Even so, the combination of analyses suggests
effects of at least 0.2 standard deviations, or enough to improve
a typical student’s performance by 8 percentage points.
But there is still the possibility that what appear to be
reform effects are actually the result of other factors.
G
R
BA
EM
affected the treatment group at the same time. This calls for
making the same before-and-after comparison in a group
that is identical, except for being unaffected by the treatment.
Subtracting these two—taking the difference of the two differences between the treatment and comparison groups—yields
a credible estimate of the policy effect.
We have carried out two difference-in-differences strategies:
1) Returnees only. We study only those students who
returned to New Orleans after Hurricane Katrina. The advantage of this approach is that it compares the same students over
time. One disadvantage is that it omits nonreturnees. Also, we
can only study returnees over a short period of time—after
2009, they no longer have measurable outcomes to study.
2) Different cohorts. We consider the achievement growth
of different cohorts of students before and after the reforms—
for example, students in 3rd grade in 2005 and students in 3rd
grade in 2012. The advantages here are that we can include
both returnees and nonreturnees, and we can use this strategy
to study longer-term effects. But the students are no longer
the same.
In both strategies, the New Orleans data set includes all publicly funded schools in the city, including those governed by
the district (OPSB), since all public schools were influenced by
the reforms. The main comparison group includes other districts
in Louisiana that were affected by Hurricane Katrina, and by
Hurricane Rita, which came soon afterward. This helps account
for at least some of the trauma and disruption caused by the
storms, the quality of schools students attended in other regions
while their local schools were closed, and any changes in the state
tests and state education policies that affected both groups.
educationnext.org
ED
Figure 1 shows the scores for each cohort, separately for
New Orleans and the matched comparison group. The scores
cover grades 3 through 8, are averaged across subjects, and are
standardized so that zero refers to the statewide mean. The
first thing to notice is that before the reforms, students in New
Orleans performed far below the Louisiana average, at about
the 30th percentile statewide. Students from the comparison
districts also lagged behind the rest of the state, but by a lesser
amount. The New Orleans students and the comparison group
O
Effects on Average Achievement
Addressing Additional Concerns
The goal of any analysis like this is to rule out explanations for the changes in outcomes other than the reforms
themselves. Our main comparisons deal with many potential
problems, such as changes in state tests and policies. Here we
consider in more depth four specific factors that could bias the
estimated effects on achievement: population change, interim
school effects, hurricane-related trauma and disruption, and
test-based accountability distortions.
Population change. Hurricane Katrina forced almost everyone to leave the city. Some returned and some did not. The
most heavily flooded neighborhoods were (not coincidentally)
those where family incomes were lowest, and people in these
neighborhoods returned at much lower rates than people who
lived in other parts of the city. Given the strong correlation
between poverty and student outcomes, this could mean that
higher test scores shown in Figure 1 are driven not by the
F A L L 2 0 1 5 / E DU CA TION NEX T
11
There was an upward change in the trajectory of student
test scores in New Orleans after the reforms of about 0.2
to 0.4 standard deviations, enough to improve a typical
student’s performance by 8 to 15 percentile points.
but we can estimate the combination of the two. A study by
the RAND Corporation of students from Louisiana districts
affected by the hurricane suggests that these two factors had
a short-term net negative effect on evacuees’ performance of
0.03 to 0.06 standard deviations. Our analysis suggests that
the negative influence is even larger for New Orleans students,
most likely because of the more extensive destruction in the
city compared with most other areas along the state’s coast.
Thus, at least in the years just after the reforms, the factors
pushing student outcomes down were at least as large as the
population changes pushing them up.
Test-based accountability distortions. One key part of the
New Orleans reforms was the idea that the state would shut
down schools within three to five years if they did not generate
a high enough School Performance Score, a measure based on
test scores and graduation rates. Prior research suggests that
such intensive test-based accountability can lead to behaviors, such as teaching to the test, that increase scores without
improvements in underlying learning or through reduced
learning in nontested subjects.
To address this problem, we estimate effects separately
by subject, recognizing that the stakes attached to math and
language scores were roughly double the stakes for science
and social studies scores during the period under analysis.
Also, the state’s social promotion policy raises the stakes for
students in grades 4 and 8. We find no evidence that the size
of effects varied systematically with the stakes attached to
the subjects or grades. However, it is hard to rule out other
potential test-based accountability distortions with our data.
As further evidence, we considered descriptive information
on nontest outcomes. State government reports indicate that,
relative to the state as a whole, the New Orleans high school
graduation rate and college entry rate (among high school
graduates) rose 10 and 14 percentage points, respectively.
So, in theory, there are many challenges to estimating the
effects of the New Orleans package of school reforms. The
combined effect of these alternative factors on long-term
achievement gains appears small, however, especially when
compared with our initial estimate of the reform effects.
There is a clear pattern across these methods. The estimates
are consistently within the same range, and even the lower end
of that range suggests large positive effects.
G
R
BA
EM
reforms but by schools serving more-advantaged students.
Observers have pointed out that the share of the student
population eligible for free or reduced-price lunch (FRL) actually increased slightly in New Orleans after the storm. But there
are many reasons not to trust FRL data. For example, they reflect
crude yes/no measures and are unlikely to capture extreme poverty of the sort common in New Orleans. Also, what really matters here is not whether poverty increased in New Orleans, but
whether poverty increased more than in the comparison group.
Therefore, in addition, we gathered data from the U.S. Census,
which measures changes in income and the percentages of the
population with various levels of education. We also carried
out the difference-in-differences analysis in these demographic
measures to understand the changes in New Orleans relative to
the matched comparison group of hurricane-affected districts,
and then simulated the effect of changes in family background
characteristics on test scores using data from the federal Early
Childhood Longitudinal Study.
We also examined pre-Katrina characteristics to see whether
the returnees were different from nonreturnees and found that
returnees did have slightly higher scores. In fact, we come to
the same conclusion in both analyses: the expected increase in
student outcomes after the hurricanes due to population change
is no more than 0.02 to 0.06 standard deviations, or about 10
percent of the difference-in-differences estimates in Figure 1.
Interim school effects. Some of the changes in student learning may reflect neither the prestorm nor poststorm quality
of New Orleans schools, but the performance of schools that
students briefly attended outside the city after the evacuation.
Other research on these students by Dartmouth economist
Bruce Sacerdote suggests that New Orleans evacuees experienced larger improvements in school quality than evacuees
from other districts.
Trauma and disruption. Any benefit of having good interim
schools might be offset by the trauma and disruption of the
storm itself and its aftermath. The majority of New Orleans
returnees probably knew someone among the nearly 2,000
people who died in the Katrina aftermath. Also, almost all students experienced significant disruption, moving to unfamiliar
neighborhoods and schools for extended periods. Reports of
post-traumatic stress disorder remain common.
It is difficult to isolate trauma and interim school effects,
ED
ED U C A T I O N N E X T / F A L L 2 0 1 5
O
12
educationnext.org
feature
NEW ORLEANS REFORMS HARRIS
Equity of Outcomes
G
R
BA
EM
be hard to say the outcomes from the New Orleans reforms
Public schools exist to ensure that all children have an are inequitable relative to what came before them. That said,
opportunity to succeed in life. Thus we consider not only the they were highly inequitable to start with, and there is clearly
average effects of the reform package, but also whether the room for improvement.
most-disadvantaged students benefited.
We first define equity in terms of how New Orleans, as an
urban district, performed relative to districts serving more- What Really Changed?
To help improve the schools going forward, it is important
advantaged students. Both before and after the reforms, at least
80 percent of New Orleans students were minority or eligible to know how school practices and other intermediate outcomes
for FRL. It is therefore noteworthy that the reforms brought changed. In a series of 15 ongoing studies, my collaborators at
the city’s students near to the state average on a wide range of ERA-New Orleans and I have examined four main components
of the reforms: choice and competition, teachers and leaders,
academic outcomes (see Figure 1).
It is also important to consider the distribution of effects within charters and CMOs, and test-based accountability.
Some of the reform effect may be driven by parental choice
the city, and here the results are more mixed. All major subgroups
of students—African American, low-income,
special education, and English Language
Learners (ELL)—were at least as well-off after
the reforms, in terms of achievement. Critics of
charter schools express concern about possible
increases in racial isolation (some would say
“segregation”). Among all of the various subgroups we considered, only Hispanic students
seem to have experienced increases in isolation.
There have also been concerns about schools
unfairly targeting low-income and African
American students in disciplinary decisions.
While we have not yet studied whether any
student groups have been specifically targeted,
we can say that the number of suspensions and
expulsions has dropped since the reforms, for
African American students and others alike.
There are a few less-positive signs, however. In our analysis of what families look
for when choosing schools, we found that
the lowest-income families place less weight
on the School Performance Score than other
families. Their circumstances may lead them
to focus more on practical considerations
such as distance to school and extended hours
(to avoid extra child-care costs). Similarly, in
our analysis of student mobility, we see that
low-scoring students are less likely than highscoring students to migrate toward schools
with high scores. Finally, up until a few years
ago, principals reported cherry-picking students by, for example, counseling out students
deemed poor fits and holding invitation-only
events to attract certain students.
Given the large improvements in average
outcomes in a district that is almost entirely
low-income and minority, and the mixed In terms of achievement, all major subgroups of students were at least as well-off
evidence on other equity indicators, it would after the reforms.
O
PHOTOGRAPH / SUSAN POAG,THE TIMES-PICAYUNE,LANDOV
ED
educationnext.org
F A L L 2 0 1 5 / E DU CA TION NEX T
13
and competition. The supply of schools in New Orleans appears
highly differentiated. Some schools specialize in math and science, others in the arts. Some schools offer language immersion
programs, while other schools have fairly traditional curricula.
Some schools have selective admissions, while others are open
enrollment or seek diverse student bodies. We also find that
New Orleans families diverge in their schooling preferences,
so having this degree of differentiation in schooling options is
likely to help match what families want with what schools offer
(see “The New Orleans OneApp,” features, and “Many Options
in New Orleans Choice System,” research, Fall 2015).
It is still unclear, however, whether these changes in the
market have contributed to the improvements in student outcomes. Even supporters of the reform efforts sometimes bristle
when I use the word “market” and “competition” to describe
the new system. Instead, they point to two other parts of the
reform package: the authority of the state to close schools and
the authority schools have over their teaching staffs.
Sixteen New Orleans schools have been completely closed
and another 30 have been taken over in some fashion by either
the RSD or OPSB—a large number in a city that has only about
90 public schools in total. Consistent with written state policies,
G
R
BA
EM
we find that the School Performance Score is the strongest measurable driver of closure and renewal decisions. Moreover, in
finding CMOs to open new schools and take over old ones, the
RSD has preferred those with a track record of academic success.
School leaders in New Orleans talk frequently about how
critical flexibility in personnel management is to their overall
school success. Free of state and local mandates and constraints
from union contracts, leaders reopening schools after the
storm could hire anyone they wanted, including uncertified
teachers, and dismiss teachers relatively easily. As CMOs took
over, more of the teacher workforce came from alternative
preparation programs such as Teach for America and The
New Teacher Project. Consistent with some other studies,
analyses commissioned by the state suggest that graduates of
these programs contribute more to student achievement than
graduates of traditional preparation programs.
The combination of policies had two types of effects on the
teacher workforce. First, the percentages of teachers with regular
certification and with 20 or more years of experience dropped
by about 20 points each. Also, due to both the short-term commitments of some alternatively certified teachers and school
autonomy over personnel, the teacher turnover rate nearly
doubled. The fact that such
large improvements in student
learning could be achieved with
these common metrics going in
the “wrong direction” reinforces
a common finding in education
research: teacher credentials and
turnover are not always good
barometers of effectiveness.
Finally, we turn to a topic
that is not typically thought of
as part of the reform package
but may be an essential component: costs and resources.
Our analysis suggests that from
2004â05 to 2011â12, the same
years covered by our achievement analysis, total public
schooling expenditures per
student increased by $1,000 in
New Orleans relative to other
districts in the state. Some of
the increase probably reflects
one-time start-up costs of new
schools, and we are working
to understand what share falls
in that category. Regardless,
there is wide agreement that the
Relative to the state as a whole, the New Orleans high school graduation rate rose 10 percentage
reforms did not come cheap.
points after the New Orleans reforms.
None of this really tells
O
ED U C A T I O N N E X T / F A L L 2 0 1 5
educationnext.org
PHOTOGRAPH / KATHLEEN FLYNN,THE TIMES-PICAYUNE,LANDOV
ED
14
feature
NEW ORLEANS REFORMS HARRIS
The effects of the reforms in New Orleans are large compared
with other completely different strategies for school improvement,
such as class-size reduction and intensive preschool.
us exactly which of the factors drove the improvements in
student outcomes—no doubt they are interconnected—but
it does provide some indication of how schools and families
responded to the policy shift.
EM
Implications for New Orleans
G
R
BA
These findings have important implications for the New
Orleans public schools, the many other urban districts pursuing the portfolio approach, and for the state and federal
policies—especially test-based and market-based accountability—from which the New Orleans reforms emerged.
For New Orleans, the news on average student outcomes
is quite positive by just about any measure. The reforms seem
to have moved the average student up by 0.2 to 0.4 standard
deviations and boosted rates of high school graduation and
college entry. We are not aware of any other districts that have
made such large improvements in such a short time.
The effects are also large compared with other completely
different strategies for school improvement, such as class-size
reduction and intensive preschool. This seems true even after
we account for the higher costs. While it might seem hard
to compare such different strategies, the heart of the larger
school-reform debate is between systemic reforms like the
portfolio model and resource-oriented strategies.
With the possible exception of distortions from test-based
accountability, which are harder to identify, the reforms managed to avoid most of the side effects that many feared. But our
findings also suggest areas of potential improvement. While
the reforms have been successful on some dimensions of
equity, it seems necessary to do more to ensure that all groups
within the city benefit. All types of public school systems
struggle with providing equitable access to quality schools,
and the New Orleans system is no exception.
student test scores made it the second-lowest-ranked district
in the second-lowest-ranked state in the country.
New Orleans is an attractive city for young educators. The
national response to the hurricane aftermath was heartening,
and for many young people, contributing to the rebuilding
effort became a calling. Later, as the reform effort took hold,
New Orleans also became the nation’s epicenter of school
reform, an ideal place for aspiring reform-minded educators.
Because the city is smaller than many urban districts, school
leaders could be very selective in choosing from the pool of
educators who wanted to come and work there.
The effects might also be smaller, at least in the short run,
if the reforms were adopted on a statewide basis, because the
reform is dependent on a specific supply of teachers. It seems
difficult enough attracting effective teachers and leaders to
work long hours at modest salaries in New Orleans; doing it
throughout Louisiana is unrealistic without a major change in
the educator labor market. Nonetheless, it would be a mistake
to dismiss the relevance of the New Orleans experience for
others. It is relevant precisely because it is so unusual. The
city’s reforms force us to question basic assumptions about
what Kâ12 publicly funded education can and should look like.
There is more to the debate than we can cover here, including
fundamental philosophical issues about whose objectives and
values should count in making schooling decisions. But there
is also wide agreement that the academic outcomes considered
here are important, so learning how much the reforms contribute to changes in academic measures should also be a key part
of the conversation. Better understanding of all the elements of
the reforms is something we owe to the city, its children, and
everyone who suffered and perished in this terrible tragedy.
educationnext.org
ED
Unfortunately, the effects of even the most successful programs are often not replicated when tried elsewhere, and there
are good reasons to think the conditions were especially ripe
for success in New Orleans:
There was nowhere to go but up. Pre-Katrina, the New
Orleans public school system was highly dysfunctional, and
O
Implications for the Nation
Douglas N. Harris is professor of economics at Tulane
University and founder and director of the Education
Research Alliance for New Orleans. The research cited here
is coauthored with others on the ERA-New Orleans research
staff (Paula Arce-Trigatti, Nathan Barrett, Lindsay Bell
Weixler, Christian Buerger, Matthew Larsen, Jane Arnold
Lincove, Whitney Ruble, Robert Santillano, and Jon Valant)
and members of the ERA-New Orleans National Research
Team (Huriya Jabbar, Jennifer Jennings, Spiro Maroulis,
Katharine Strunk, Patrick Wolf, and Ron Zimmer).
All errors are the author’s.
F A L L 2 0 1 5 / E DU CA TION NEX T
15
