The following is a paper that appeared in the journal SCIENCE earlier this year (Science 13 January 2012; Vol. 335 no. 6065 pp. 171-172; DOI 10.1126/science.1205418) written by Julian Betts and Richard Atkinson. The authors make three suggestions for helping in research in this area: 1) Lottery data should be shared with public agencies; 2) an independent observer should be present for lotteries, and 3) longitudinal student level test data should be made more easily available to researchers. -Dr. Petrosino
Charter schools have been promoted as a solution to what many view as the public school malaise in the United States. Charter schools, although publicly funded, can operate fairly independently of large district bureaucracies and teacher unions, for instance, by setting their own curriculum and teaching methods and by avoiding the system that grants the most senior teachers first choice of job openings, regardless of their classroom effectiveness. Proponents hope that the semi-independent governance structure of charters will encourage these schools to generate fresh ideas. President Obama has followed Presidents Bush and Clinton in identifying charter schools as a key element of school reform.
Are charter schools boosting achievement? Is there variation across charter schools in effectiveness? These are key policy questions, as failing charter schools should be shut down, and successful charter schools replicated (1). Unfortunately, most studies of charter schools' impact on student achievement use unsophisticated methods that tell us little about causal effects. We highlight below some key problems, and suggest policies and practices that could improve research and, we hope, education.
Lotteries: Promising, But Not Perfect
A recent meta-analysis discarded roughly 75% of studies because they failed to account for differences between the background and academic histories of students attending charter schools and those attending traditional public schools (2). Most studies simply take a snapshot of student performance at a single point in time. Such studies cannot disentangle school quality from the preexisting achievement level and trajectory of students who decide to attend a given school. The potential for student self-selection into charter schools is great, which makes naïve comparison of student outcomes at charter schools and traditional public schools misleading. Parents may not apply to a charter school because of the distance to the school or lack of time to fulfill volunteer work that such schools sometimes request. Families that apply may be unusually motivated. The decision by charter school operators about where to locate also influences who attends, which makes simple comparisons with traditional public schools difficult. More often than not, the difference in average test scores between charter schools and traditional public schools reflects who enrolls at the schools more than the quality of education being provided (3, 4).
But rigorous research on charters is beginning to appear, and much of this takes advantage of the way in which charter schools admit students. State laws dictate that if a charter is oversubscribed, then an admissions lottery must be held. Because only chance distinguishes who does and does not receive admission, the students who lose the lottery represent the ideal control group. Lottery-based studies of charter schools have been done in Boston (Massachusetts); New York City; a small national sample of middle schools; and a few schools in Chicago (Illinois), San Diego (California), and Lynn (Massachusetts). These studies tend to find that charter schools either outperform or perform at the same level as traditional public schools (5–12). These studies, however, cover only about 90 charter schools, roughly 2% of charter schools nationally (13).
We strongly support the wider use of randomized controlled trials of the impact of charter schools on student outcomes. At the same time, we acknowledge that this approach has limitations. Foremost among these is that most charter schools are not oversubscribed. For example, the U.S. Department of Education released a lottery-based study of charter middle schools that found that only 130 out of 492 such schools nationwide used admission lotteries (10). This raises the possibility that a study of oversubscribed charters will not tell us anything about the effectiveness of the majority of charter schools that are not sufficiently popular to be oversubscribed. The natural targets for research, districts with many oversubscribed charter schools, may have unusually good charter schools. Indeed, one study showed that parents in Texas are more likely to remove their children from underperforming charter schools than from charter schools that outperform nearby traditional public schools (14). Lottery-based studies of middle- and high-school charter schools in Boston produced among the highest estimates of impacts on reading and math achievement in studies of those grade spans (5). Another lottery-based study of New York City charter schools produced the largest estimated impacts among studies of elementary and middle schools in combination (8). In contrast, a lottery-based national study of charter middle schools that did not solely seek districts with the greatest demand for charter school slots found no significant gains from winning a lottery (10).
For several reasons, it will be important to study the many charter schools that are not oversubscribed, using the best research designs possible. In addition to obtaining a more representative portrait of charter schools, studying the qualitative features of a broad set of schools will allow both theoretical insight and institutional knowledge that can help to separate causation from correlation.
Essential to studies of undersubscribed charters schools is to account for individual students' past achievement. Even with this, it will be difficult to estimate the causal effects of attending a charter school because of unobserved factors that influence who attends charter schools. We have three locations in which to compare results of lottery-based and careful non-lottery-based studies (15, 16). The two approaches produce somewhat similar results, although the non-lottery-based studies have sometimes produced lower estimated effects, perhaps because of inadequate controls for unobservable characteristics of students and their families.
Better Policy, Better Data, Better Research
Research on charter schools must evolve in several other ways. Because charter schools have freedom to experiment, not all of them will perform equally well. Thus, research should estimate the impact of specific charter schools (or, at the very least, types of charter schools). Once we have identified the most successful models, in order to replicate them, we need better information on what aspects of these schools lead to better performance. Do pedagogical or curricular approaches, or the qualifications of teachers, explain any of the differences? Does it matter whether charters are organized locally or are affiliated with charter management (nonprofit) or educational management (for-profit) organizations? Do aspects of the policy environment, such as state law and approaches taken by local authorizers, matter? It will be increasingly important for the literature to report not just “average” effects of charter schools but effects of individual schools, while getting inside the “black box” to learn more about distinctive educational features of each charter school.
There are other roadblocks to the use of admission lottery data for analyzing effects of charters. Fortunately, individual states could remove most of these barriers by overhauling the laws governing charter schools, charter school authorizers, and the bureaucracy that can limit the availability of student test score data to the research community.
First, in most states, individual school districts are the main agencies that can authorize the opening of a charter school. In a few states, public universities or a state agency can directly charter schools as well. State laws typically do not require that charter schools share lottery information with the authorizing entity or the state. This is shortsighted. The aforementioned national study found that of the 130 charter middle schools that used lotteries, only 77 agreed to participate by sharing their lottery data (10). Lottery data should not be viewed as the property of the charter school; rather, it is incumbent upon authorizers to gather and scrutinize these data, not least to verify that the lotteries are being done in a fair manner. States should thus require each charter school to share its lottery data with the authorizing authority and also with the state's department of education, subject to standard safeguards to avoid release of individuals' identities.
Second, for researchers to conduct a successful evaluation of charter schools, either an observer needs to be present or detailed characteristics of the lottery process need to be reported (17). If some lottery winners turn down the admission offer, it is crucial that researchers understand how the school admitted students from its wait list of students who were not initially admitted in the lottery. If the school admits students from the wait list on a nonrandom basis, showing favoritism toward certain students, all pretense of randomization is lost. As another example, researchers need to know whether students were placed into separate lotteries by grade, or were given preferences; for instance, if a sibling already attended the school. Such information allows researchers to preserve an experimental analysis by stratifying the data. Charter schools should be required to submit not only a list of lottery winners and those who did not win the lottery, but details on how students were admitted from wait lists, whether separate lotteries were held for different groups of students, and what preferences were applied.
Finally, with a few exceptions, most states make it difficult if not impossible to obtain student-level test-score data for research purposes. States should routinely authorize researchers from academic institutions to obtain longitudinally linked student test-score data. This would raise all education research, not just on charter schools, to a more rigorous level by enabling researchers to use rigorous nonexperimental approaches in the many cases where schools are not oversubscribed. This would also open up the possibility of supplementing analyses of test-score gains with longer-term outcomes such as high school graduation and college attendance (18). These are likely to be better predictors of long-term adult success of students than test scores alone. Such studies are much needed, because of concerns that states' achievement tests focus on a fairly narrow set of skills, e.g., making it difficult to reveal differences between charter schools and other schools on measures of learning higher-order reasoning and writing ability.
It may seem like a tall order for so many states to enact these reforms independently. The federal government could play an important role in encouraging states to follow through. Federal initiatives such as the No Child Left Behind Act and the Race to the Top fund have made federal financial support for states' education systems contingent upon states enacting certain reforms. The federal government could tie funds to support charters or other schools to these reforms, and thus help identify and replicate the most successful schools, then shut down the schools that underperform.
↵* R. C. Atkinson is President Emeritus of the University of California and former Director of the National Science Foundation.
References
- ↵
- ↵
- ↵ For an example of how simple comparison of test scores between charter and traditional public schools reflects initial student achievement but not actual gains in performance, see (4).
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵
- ↵ Both lottery-based and nonexperimental estimates for Boston are provided in (5). In three of four cases, the nonexperimental estimate is lower than the corresponding lottery-based estimate. Lottery-based estimates for New York City are provided in (8). A nonexperimental replication of the New York City results is provided in (16), and the authors obtain the same estimate for math but an estimate for reading that is two-thirds of the lottery-based estimate. An attempt to replicate the lottery-based results in (12) without lottery data yields similar results, but only if each student's past achievementis controlled for (4).
- ↵
- ↵
- ↵
Picture: Mumford and Sons perform in Hoboken, NJ (Wednesday, August 1, 2012). Pier A
