In a recent television interview with Bill O’Reily, President Obama discussed the D.C. Opportunity Scholarship program and stated the private school vouchers “didn’t actually make that much of a difference” and have not “significantly improved the performance of kids in these poorest communities.”
President Obama seems to be relying on the final report of the D.C. Opportunity Scholarship Program which stated, “there is no conclusive evidence that the OSP affected student achievement.” But it seems Obama’s (and some of the media’s) familiarity with the report ends here.
It’s true the final report did not find statistically significant reading gains, but earlier reports over the first three years did. The report also found large gains in graduation rates. And importantly, even the lead author of the final report, Patrick Wolf, supports expanding the D.C. voucher program. A deeper understanding of the report explains why.
First, the study examines the impact of being offered a voucher (after applying, qualifying and winning in the lottery process) – not the impact of using a voucher. This was done to set a really high bar for determining whether the vouchers made a difference. To achieve statistically significant achievement results, all the kids who won a voucher and used it to attend private schools had to score high enough to lift the scores of all the kids who won a voucher but stayed in public schools.
Next, random assignment studies (as great as they are) suffer from a major methodological flaw called “the real world.” Students were randomly assigned to a control group (no voucher offered) and a treatment group (a voucher was offered). Students in the control didn’t have vouchers, but that didn’t stop them from enrolling in private schools or charter schools. Students who were offered vouchers weren’t required to use them and if they did, they didn’t have to stay in the private schools.
By the final year report, 47 percent of students in the control group (who were not offered a voucher) ended up in private schools or charter schools at some point during the study. Regarding the treatment group, 78 percent of the students offered a voucher used a voucher, but only 27 percent used it to attend a private school during every year of the study. That means 51 percent of students offered a voucher used it inconsistently — returning to public, charter and private schools as they pleased.
In other words, one could summarize the study as examining the impact of some students using school choice vs. slightly fewer students using school choice. The DC study is not, as President Obama believes, proof that vouchers do not work.