Testing and Accountability

NAEP History, Geography, Civics scores paint dismal picture

Matthew Ladner April 27, 2020

Diffusion, then, rather than discovery, is the duty of our government. With us, the qualifications of voters are as important as the qualifications of governors, and even comes first in the natural order. -Horace Mann

Preparing students to responsibly exercise citizenship represented a foundational aspiration in the founding of the American public-school system. Then as now, we did not simply aspire to have schools equip students with the knowledge and skills necessary for economic flourishing; we also wanted students to develop a grounding in history and civics.

Our education system is failing to accomplish this goal.

Last week, the National Assessment of Educational Progress (NAEP) released nationwide eighth-grade data on the 2018 exams taken in United States History, Geography and Civics. Fewer than one-third of American public-school students scored proficient on any of the three exams. In fact, on each exam, the percentage of public-school students scoring “Below Basic” exceeded the percentage scoring “Proficient.”

For every eighth-grade public-school student who scored “Proficient,” there were more than 2.5 who scored “Below Basic,” the lowest achievement level. We can’t be sure what the ratio of “Proficient” to “Below Basic” would be among American eighth-graders if we abolished their history classes, gave them a library card and offered to pay them a dollar for every history book they read, but it might surpass 2.5 to 1.

While scores aren’t available for private schools in general, scores for Catholic school students are – and they tell a different story.

Scores of Catholic school students show an advantage, in several ways: when comparing them to free and reduced-price lunch-eligible public-school students; non-eligible free and reduced-price lunch populations; and racial and ethnic groups. If you compare Catholic school students of parents without college degrees to public-school students of parents without college degrees, Catholic school students get the better of it. The same holds true when you compare Catholic school students of college-educated parents to public-school students of college-educated parents.

Ironically, we heard last week a faint echo of the Know-Nothing Party’s anti-Catholic school rhetoric directed at home schooling. Harvard Magazine quoted Harvard Law professor Elizabeth Barthole:

“From the beginning of compulsory education in this country, we have thought of the government as having some right to educate children so that they become active, productive participants in the larger society,” she says. This involves in part giving children the knowledge to eventually get jobs and support themselves.

“But it’s also important that children grow up exposed to community values, social values, democratic values, ideas about nondiscrimination and tolerance of other people’s viewpoints,” she says, noting that European countries such as Germany ban homeschooling entirely and that countries such as France require home visits and annual tests.

People can have honest disagreements about home-school policy, but I invite the reader to ponder the first chart in this post and attempt to square it with the quote above. Should we be worried about the civic values and knowledge of the 3 percent of students who home-schooled before the pandemic? Or might the appalling lack of history, geography and civics displayed by the 87 percent or so students attending public schools constitute a more pressing concern?

April 27, 2020 1 comment

Analysis Charter Schools Education Choice Education Spending factcheckED Featured Private School Scholarships School Choice Testing and Accountability

FACT-CHECKED: Tallahassee educator’s attack on ‘monstrous’ system is misleading

Jon East January 23, 2020

Jon East

Editor’s note: The goal of fact-checkED is to bring clinical precision to complex issues that are easily misunderstood, aiming to counteract incorrect information before it continues to circulate.

Sally Butzin has a rich and accomplished history in public education, as a classroom teacher, instructional innovator and published author. But her latest column in the Tallahassee Democrat offers an unfortunate brew of bitterness and hyperbole about modern reforms.

Butzin cast Florida today as something of an educational wasteland, reminiscing about the early 1990s when “innovation was encouraged” and contrasting that with the election of Gov. Jeb Bush in 1998 when “the bottom fell out.” She deplored the “emphasis on testing and accountability” but especially blames charter schools and private school vouchers. They constitute a “monstrous parallel system” that “has bled the life out of public education” and “left millions of children behind.”

We’ll leave to others the political defense of Bush, who ushered in a wave of education reform in his eight years in office, and instead focus on three of her more tangible claims.

Claim No. 1: “These new schemes had no accountability to taxpayers.”

The “schemes,” in this context, refer to charter schools and private school scholarships.

First, charter schools are created through a contract with each public school district that involves oversight into educational goals and tax dollars received. Their students are required to take the same state assessment tests as all other district students and the schools receive the same letter grade ratings as public schools. Further, under state law, a charter school can be shut down, and many have been, for receiving two grades of F.

Second, the state’s largest scholarship program, the Florida Tax Credit Scholarship serving 108,570 economically disadvantaged students in 1,836 private schools, is overseen by the state Department of Education. Students are required to take nationally norm-referenced tests approved by the state and those test results are reported publicly each year. Schools that receive more than $250,000 in scholarship dollars are required to submit financial reports by certified public accountants, and scholarship funding organizations are audited each year by the State Auditor General and private accounting firms. The students, schools and scholarship organizations are subject to roughly 17,000 words of statutory and agency regulations.

Third, both charter schools and scholarship schools face a form of accountability that can be more intense than state regulations and audits. Because they are schools of choice and students are free to attend or not, they can go out of business if parents are dissatisfied and choose elsewhere.

These points are brief in part because Butzin’s claim is categorical. She wrote that they have “no accountability,” which is flatly false.

Claim No. 2: “This parallel system has bled the life out of public education, as billions of dollars flowing to it have left millions of children behind.”

This claim seems to have a life of its own, despite the fact that it is refuted by extensive independent research. At its core, it assumes that public schools should be paid tax dollars without regard to how many students attend them, even though Florida has always funded education on a per-student basis. And it ignores the fact that students choosing these two alternative options receive fewer dollars than they would if they remained in a traditional public school.

This is the third time in the short history of fact-checkED that we have analyzed this financial cost claim, and for more detail we invite you to read our first effort. Once again, we rate this claim as false.

Claim No. 3: “Now, it would be one thing if this parallel system was improving outcomes for all our children. But this is simply not the case.”

Butzin did not share her data on outcomes, but the National Assessment of Educational Progress (NAEP) is generally regarded as one of the most reliable indicators of academic achievement in elementary and secondary schools. NAEP, often called the “Nation’s Report Card,” paints an entirely different picture.

By every measure, the NAEP reading and math test scores of fourth- and eighth-graders have increased over the past two decades – the period after which Butzin says “the bottom fell out.” The percentage of fourth-graders who meet basic standards in math is up by nearly two-thirds, and the percentage rated proficient or above has tripled. The percentage of fourth-graders who meet basic standards in reading is up by a third, and the proficient and above is up by two-thirds. Similarly, the percentage of eighth-graders who meet basic in math is up by a fifth, and proficient is up by four-fifths. The percentage of eighth-graders at basic in reading is up by a tenth, and the percentage above proficient up by half.

And, according to the Urban Institute’s adjustment of rankings for race, income and English language learner status, Florida ranks first in fourth-grade reading, first in fourth-grade math, third in eighth-grade reading and eighth in eighth-grade math.

A few other measures are worth mentioning. The most recent Education Week Quality Counts report ranked Florida fourth in the nation in terms of K-12 academic achievement. The College Board ranked Florida third in percentage of graduating high school seniors who have passed college-caliber Advanced Placement exams. The high school graduation rate has jumped from 60.1 percent to 86.9 percent in the past two decades.

We don’t know the data Butzin used, but we rate her claim as highly misleading.

January 23, 2020 0 comment

2020 Presidential Election Analysis Education Choice Education Politics Education Spending factcheckED Featured School Choice Tax Credit Scholarships Testing and Accountability

fact-checkED: Sanders’ education choice claims in Florida op-ed misleading

Jon East January 16, 2020

Jon East

With the presidential primary election in delegate-rich Florida just two months away, Democratic candidates are beginning to knock on the door.

In a Monday column in the South Florida Sun-Sentinel, Vermont Sen. Bernie Sanders found little to like about Florida’s public education system. But our focus is on educational choice and three assertions he made while insisting that “Florida is ground zero of a school privatization movement intent on destroying public education.”

“Each year almost $1 billion in state money goes to private schools instead of public schools.”

“These private schools operate with little to no accountability.”

“In many cases, their students’ math and reading skills have declined.”

These claims are misleading, at best.

Claim No. 1: The $1 billion citation draws attention but does not prove Sanders’ point. He implies that private school voucher and scholarship programs harm public schools financially, which is a longstanding and erroneous argument advanced by the Florida Education Association (FEA) that we have dealt with previously in this space. There is simply no evidence to support it.

Eight different independent financial studies all have concluded the scholarships, which cost less per student than the total amount spent in a traditional public school, save tax money that can be used to enhance public schools. Florida TaxWatch weighed in last year, finding existing scholarships cost 60 cents on the public school dollar. And the FEA lost its legal challenge to the 19-year-old Tax Credit Scholarship program in part because the trial and appellate courts summarily rejected its argument that public schools “have been and will continue to be injured by the scholarship program’s diversion of resources from the public schools.”

Claim No. 2: The qualifier in this assertion – that private schools operate with “little to no” accountability – gives Sanders some wiggle room in part because accountability is hard to objectively measure. But Sanders seems to dismiss entirely the accountability that is inherent with any school that survives only if parents choose it for their children.

That market concept may not appeal to a self-professed socialist but does have real-world consequences and, in the case of the state’s two scholarships for underprivileged students, gives genuine educational power to parents who typically have little.

As to the regulatory oversight, Florida’s largest program, the Tax Credit Scholarship serving 108,570 economically disadvantaged students this year, is ranked among the nation’s most aggressive. The students, schools and scholarship organizations are subject to roughly 17,000 words of statutory and agency regulations.

Among those requirements: scholarship students must take state-approved standardized tests; those test results are reported publicly every year; schools must submit annual financial reports by certified public accountants; and scholarship organizations must be audited each year by the state Auditor General and independent accounting firms.

Sanders is entitled to believe the program is not subject to sufficient regulation, but we rate his claim of “little to no” accountability as misleading.

Claim No. 3: Once again, Sanders uses an interesting qualifier – “in many cases” – in his assertion that scholarship students’ “math and reading skills have declined.” Eleven years’ worth of standardized test reports refute his claim.

Students who receive the Tax Credit Scholarship must take a nationally norm-referenced test each year approved by the state, and most take the Stanford Achievement. The test scores are sent to Florida State University’s Learning Systems Institute, which is paid to aggregate and report them publicly.

For the most recent year, 2017-18, scholarship students scored on average at the 47.4 percentile in reading and 45.2 percentile in math. That’s basically average, which is encouraging given that these scholarships students are among the poorest in the state and were among the lowest-performing students in the public schools they left behind.

More to the point, their annual gains have been remarkably consistent. The scores reflect that these low-income students have achieved the same annual gains as students of all income levels nationally.

In other words, they have increased, not declined.

Certainly the average doesn’t reflect every student. In the most recent report, about as many students – 0.6 percent – achieved extraordinary gains of more than 40 percentile points as those who dropped an equivalent amount. But Sanders’ qualifier that “many” students declined seems intended to distort the overall test findings. We rate it mostly false.

January 16, 2020 0 comment

Achievement Gap Analysis Commentary and Opinion Education Spending Featured Testing and Accountability

USA K-12 achievement stalled on international exams

Matthew Ladner December 9, 2019

Matthew Ladner

PISA, the OECD’s Programme for International Student Assessment, measures 15-year-olds’ ability to use their reading, mathematics and science knowledge and skills to meet real-life challenges.

The performance of American 15-year-olds in reading and math has remained stagnant for the past two decades according to results released this past week from the Organisation for Economic Co-operation and Development (OECD). Meanwhile, the achievement gap in reading between high- and low-performing students has grown wider.

The less-than-stellar results from the exam, the Program for International Student Assessment (PISA), mirror recently released scores from the National Assessment of Educational Progress (NAEP), which I described in an earlier post as an Agincourt-level disaster.

Chile is the country most closely resembling the math and reading scores of American black students; Turkey most closely matches the combined achievement of American Hispanic students. Yet the United States spends more than twice as much per pupil as either Chile or Turkey.

Even the scores for American white students, while internationally competitive, appear less than impressive.

Estonia scores a bit higher while spending half as much per pupil as the United States. Moreover, scholars have made us aware that higher-income American families spend lavishly on enrichment options for their K-12 students (tutoring, club sports, etc.), and that this spending has increased steadily over time.

How many Estonian families do you reckon spend $8,872 per child per year on enrichment spending? Since the average American income is more than $26,000 per person higher than that in Estonia after adjusting for purchasing power, I’m going to walk out on a limb and dare a guess: Not many.

And, while enrichment spending apparently has little to do with improvement among low-income students (the trend in such spending is flat since the early 1980s), that isn’t the case among advantaged students. Richer in schools twice as well funded but underperforming is not a great place for America’s highest-performing subgroup to find itself vis-à-vis Estonia.

Even without these latest PISA results, but reinforced in light of them, it’s clear that without the benefit of lavish enrichment spending and other related advantages, the high levels of spending in American schools appears broadly ineffectual for students of color.

December 9, 2019 3 comments

Commentary and Opinion Education and Public Policy Jonathan Butcher School Choice Testing and Accountability

Guest commentary: NAEP should be a wake-up call for ed reformers, policymakers

Jonathan Butcher November 21, 2019

Jonathan Butcher

A recent comparison of K-12 children around the U.S. brought bad news for education reformers—an amorphous group of policymakers and advocates who are akin to locksmiths searching for the right combination of resources and policy ideas to unlock student potential. The news was bad for students, too, but since the scores do not affect a student’s report card, the results mattered more to the aforementioned locksmiths today.

The results will matter for students tomorrow.

As readers of this blog will know, the Nation’s Report Card, also known as the National Assessment of Educational Progress (NAEP), is a set of math and reading tests given to a sample of fourth- and eighth-graders around the country every two years. The U.S. Department of Education also administers NAEP tests in other subjects and specific cities.

The more frequent math and reading results allow education locksmiths to gauge whether any number of inputs in the nation’s schools—from spending increases to new laws—are having their desired effects. With both more money and federal law firmly in place, observers were frustrated that national averages for 2019 reading and math scores fell from 2017 (with the 1-point improvement in fourth-grade math a small exception).

Rigorous research is required to appropriately link policies and school budgets to NAEP outcomes, but Florida’s aggressive upgrade to its K-12 education design in the early 2000s tracked closely with improved NAEP scores among minority students. The additional parent options in education, along with a focus on reading among third-graders and more attention to the use of Advanced Placement testing has been the subject of policy discussion around the country since.

This year, reformers pointed to similar reading-related policies in Mississippi to explain the state’s improvement in fourth-grade scores, while calling for “urgent action” elsewhere. The Council of Chief State School Officers even said the “urgency of improving outcomes for all students” was enough to plan … a meeting. This “urgent action” is why the results will matter for students tomorrow: That is when the changes will affect the classroom. Parents can only hope it is not too late by then.

Nationally, average scores among fourth- and eighth-grade students increased in math by 27 points and 20 points, respectively, between 1990 and 2009. Yet since 2009, and a decade is almost the length of a child’s K-12 career, the improvement stalled. Reading scores have improved by just three points in both grades since 1992.

It is these scores and other comparisons that have not changed that should bother reformers and policymakers alike in 2019. Last summer, Harvard University’s Paul Peterson wrote in Education Next that the test score gap on NAEP and international tests between students from low-income families and their more affluent peers has not changed since the 1960s. Peterson wrote later that the “performances on math, reading, and science tests of the most advantaged and the most disadvantaged students differ by approximately four years’ worth of learning, a disparity that has remained essentially unchanged for nearly half a century.”

Such findings temper enthusiasm for any year-to-year increases. So, too, do the results from NAEP’s other test, the Long-Term Trend Assessment (LTT). Average scores for 17-year-olds in math and reading have not changed since the 1970s (the LTT was last given in 2012 and is scheduled to resume in 2020). Students appear to lose any gains made in elementary and middle school by graduation.

Over time, these results have been depressingly more of the same, which should incentivize education reformers and policymakers not to do the same things. Such as: In the last two years, many state policymakers were either absent or complicit in teacher union attempts to keep students in assigned district schools by curbing charter school growth in Los Angeles and Chicago and blocking new private learning options in Kentucky and West Virginia.

These options are the most significant departures from the routine of school-by-zip-code because parents can help a struggling child immediately by moving them to a new setting with a charter school, private school scholarship or education savings account. Students will not have to wait for the urgent meetings to finish, hoping to find success this time.

Now that would be different.

November 21, 2019 0 comment

Analysis Education and Public Policy Education Politics Testing and Accountability

NAEP 2019: Where do we go from here?

Matthew Ladner November 14, 2019

Matthew Ladner

The aughts were a time of K-12 academic improvement. For some states, including Florida, that improvement started earlier, but included the era illustrated in this chart.

All states began participating in NAEP exams in 2003. Most states experienced a pronounced improvement in eighth-grade reading and eighth-grade math from 2003 to 2009. On these tests, 10 points equals approximately one grade level’s worth of average progress.

As you can see, there are far more bars going in a positive direction in the first chart and relatively few going in a negative direction between 2003 and 2009. Now let’s examine the same sort of chart between 2009 and 2019.

Things went badly after 2009 in most states. Some broad theories readily come to mind but are weaker than they might seem.

Most states adopted the Common Core standards after 2009, for instance. Prominent analysts of the center right and left have studied the impact of Common Core on academic achievement and have reached a common conclusion: not much positive or negative impact. Look at Texas on both charts; positive experience in 2003 to 2009, deeply negative experience from 2009 to 2019. Texas never adopted Common Core.

Obviously, lots of states that did adopt also had bad decades, but Mississippi had the biggest 2009 to 2019 gains. Judged against lofty promises of improvement, the Common Core project clearly fell flat at a high cost in terms of political and financial capital. That is very different than saying they worsened outcomes, but it’s still very bad.

It was the economy! My friend Mike Petrilli has been exploring the theory that the Great Recession is a driver of decline in NAEP performance. Texas, again, is a fly in this ointment.

The Texas economy got off light in the Great Recession and boomed economically for much of the 2009 to 2019 period, but Texas scores worsened statewide. The so-called “Sand States” – Arizona, California, Florida and Nevada – were the hardest hit in the housing downturn, but they all had a much better decade driving academic improvement than Texas.

Per-pupil spending has been increasing since fourth-graders of 2019 took the NAEP, and for most of the time eighth-graders have been in school. We should hope that Mike is wrong about this given that the country is overdue for a recession by historical standards.

Sandy Kress makes the case that the blame lies in the Obama administration’s decision to let the pressure off test-based accountability. The trends certainly show something went south after 2009, and Sandy may be right. It’s worth considering whether something had to give with the 100 percent-ish proficiency on a deadline NCLB requirement.

Before President Obama took office, I had feared states might simply drop their passing thresholds if the federal government stuck to its guns. Moreover, the Common Core project coupled with teacher evaluation can be seen as a doubling down on high-stakes testing strategies, one that radicalized opponents countered in a variety of (largely predictable) ways.

So, what to make of all of this?

Providing academic transparency to parents, taxpayers and policymakers remains vital, but it’s clear the public is not on board for heavy-handed, top-down testing mandates. We live in a democracy, and the demos is not enamored with the idea of closing schools based on test scores and firing teachers based on test scores, and are sick of the amount of test prep going on in school these days. If you doubt any of that, feel free to focus-group it, or alternatively go outside and talk to normal people.

In many states, much of the low-hanging academic improvement fruit of the NCLB era has been reversed post 2009. Sun Tzu wrote that “a victorious general wins and then seeks battle whereas a defeated army seeks battle and then seeks victory.” It’s clear which of these two profiles education reformers more closely resemble today.

A good first step to recovery would be to focus on policies that have the prospect of developing supportive constituencies rather than annoying the public. Those who adhere to the status quo have the upper hand in most states even without antagonizing the public.

November 14, 2019 1 comment

Achievement Gap Analysis Education Reporting Testing and Accountability

NAEP 2019: Where does Florida stand?

Patrick R. Gibbons November 5, 2019

Patrick R. Gibbons

Year-to-year student achievement will always fluctuate, but how does Florida stand in the larger picture? While results from the 2019 National Assessment of Educational Progress should concern everyone involved with K-12 education, Florida’s long-term trend is holding despite the recent score stagnation. And as we reported last week, Florida maintained high rankings when adjusting for demographics.

Among eighth-grade math students, Florida has improved 24 points compared to the national average of 19. Black students are up 28 points compared to 23 points nationally, and Hispanic students are up 30 points compared to 22 points nationally.

Eighth-grade reading, typically a sore spot for Florida given the tremendous success in the early years, is up 8 points compared to 21 years ago. But the situation is much worse for the national average, where there has been no improvement on eighth-grade reading scores since 1998.

Black eighth-graders in Florida, despite being down from last year, still have improved by 8 points since 1998. The average black student nationally has seen no improvement since 1998.

Hispanic students fare better on English reading, up 12 points in Florida and 9 points nationally since 1998.

November 5, 2019 0 comment

Demographic Research Education Reporting Testing and Accountability

NAEP 2019: Mississippi improvement more impressive under closer examination

Matthew Ladner November 4, 2019

Matthew Ladner

Federal officials released results from the 2019 National Assessment of Educational Progress, or NAEP, Oct. 30. The congressionally mandated project administered by the National Center for Education Statistics provides group-level data on student achievement in reading and mathematics for grades 4 and 8.

While last week’s NAEP news was glum nationwide, Mississippi students performed relatively well. You have to dig into the details to see just how well.

First, a bit of backstory.

Mississippi is one of the nation’s poorest states and has the largest African-American student population in the country. The state has ranked at or near the bottom on NAEP scores for many years. Over the past decade, policymakers have adopted several K-12 policies familiar to Floridians: A-F letter grades for schools, a strong emphasis on early literacy, a charter school law and an education savings account program for students with disabilities.

The state’s choice sector went from non-existent to nascent in recent years, although it’s not yet at a scale where it can play more than a complimentary role to the public education system. Yet based on this year’s NAEP, as seen in this crosstab of student race/ethnicity on fourth-grade reading, something seems to have gone very right.

Average scale scores for Grade 4 reading by National School Lunch Program eligibility across race/ethnicity for Mississippi, nation

Free and reduced-price meal eligible white students in Mississippi, as well as free and reduced-price meal eligible black students, outscored their peers by 9 points. Not to be outdone, eligible Hispanic students beat the national average by 15 points. These differences are statistically significant.

Meanwhile, Mississippi black students who do not qualify for free and reduced-price meals (middle to high income) outscored their peers nationwide by 22 points.

Mississippi didn’t just clobber national averages; the state took it to the nation’s wealthiest states as well. Northeast states (Connecticut, Maine, Massachusetts, New Hampshire, New Jersey, New York, Pennsylvania, Rhode Island and Vermont) have long held the highest NAEP scores, but they tend to hold the same advantaged end of various achievement gaps as wealthy states. Many of those states spend far more per pupil than Mississippi as well.

NAEP allows comparisons not just between states, but also between states and regions. So, here are the trends for fourth-grade reading broken down by student free and reduced-price meal eligibility status.

Back in 2009, students in Northeastern states in both low-income and middle/high income categories were well ahead of that cohort in Mississippi, but in 2019, Mississippi led among low-income students and tied among non-eligible free and reduced-price meal students. Mississippi students progressed from being well behind students in Northeastern states to being virtually tied in both categories on eighth-grade reading over the past decade.

As a relatively late adopter of reform efforts, Mississippi had the chance to learn from earlier successes and earlier mistakes. Alas, there have been more of the latter than the former.

Massachusetts began its reform efforts in 1993 and had the highest scores on all four NAEP exams until 2019. Florida’s scores surged after 1998 with a different, but also multi-faceted set of reforms. Arizona students were unique in achieving statistically significant gains on all six NAEP exams between 2009 and 2015, and Mississippi joined Arizona in that category in 2017.

Initial success does not guarantee sustained success, but for now, Mississippi leads in the academic improvement clubhouse. I hope it continues its success – and that the rest of us can give the state some competition.

November 4, 2019 0 comment