http://www.ajc.com/news/cheating-our-children-suspicious-1397022.html

Interesting they report improbable scoring trends were twice as likely in charter schools

Cheating our children: Suspicious school test scores across the nation

By Heather Vogell, John Perry and Alan Judd and M.B. Pell

The Atlanta Journal-Constitution

Suspicious test scores in roughly 200 school districts resemble those that entangled Atlanta in the biggest cheating scandal in American history, an investigation by The Atlanta Journal-Constitution shows.

Hyosub Shin, hshin@ajc.com St. Louis: Patrick Henry Downtown Academy’s principal was placed on leave last year for falsifying attendance records. Because attendance rates are used to calculate state funding, it’s possible the alleged fraud attracted state aid to the school that it didn’t deserve. Even though the state has not found cheating at Henry, an AJC analysis uncovered unusual scores dating back to 2007.

The newspaper analyzed test results for 69,000 public schools and found high concentrations of suspect math or reading scores in school systems from coast to coast. The findings represent an unprecedented examination of the integrity of school testing.
The analysis doesn’t prove cheating. But it reveals that test scores in hundreds of cities followed a pattern that, in Atlanta, indicated cheating in multiple schools.
A tainted and largely unpoliced universe of untrustworthy test results underlies bold changes in education policy, the findings show. The tougher teacher evaluations many states are rolling out, for instance, place more weight than ever on tests.
Perhaps more important, the analysis suggests a broad betrayal of schoolchildren across the nation. As Atlanta learned after cheating was uncovered in half its elementary and middle schools last year, falsified test results deny struggling students access to extra help to which they are entitled, and erode confidence in a vital public institution.
“These findings are concerning,” U.S. Secretary of Education Arne Duncan said in an emailed statement after being briefed on the AJC’s analysis.
He added: “States, districts, schools and testing companies should have sensible safeguards in place to ensure tests accurately reflect student learning.”
In nine districts, scores careened so unpredictably that the odds of such dramatic shifts occurring without an intervention such as tampering were worse than one in 10 billion.
In Houston, for instance, test results for entire grades of students jumped two, three or more times the amount expected in one year, the analysis shows. When children moved to a new grade the next year, their scores plummeted — a finding that suggests the gains were not due to learning.
Overall, 196 of the nation’s 3,125 largest school districts had enough suspect tests that the odds of the results occurring by chance alone were worse than one in 1,000.
For 33 of those districts, the odds were worse than one in a million.
A few of the districts already face accusations of cheating. But in most, no one has challenged the scores in a broad, public way.
The newspaper’s analysis suggests that tens of thousands of children may have been harmed by inflated scores that could have precluded tutoring or more drastic administrative actions.
The analysis shows that in 2010 alone, the grade-wide reading scores of 24,618 children nationwide — enough to populate a midsized school district — swung so improbably that the odds of it happening by chance were less than one in 10,000.
Cheating is one of few plausible explanations for why scores would change so dramatically for so many students in a district, said James Wollack, a University of Wisconsin-Madison expert in testing and cheating who reviewed the newspaper’s analysis.
“I can say with some confidence,” he said, “cheating is something you should be looking at.”
Statistical checks for extreme changes in scores are like medical tests, said Gary Phillips, a vice president and chief scientist for the large nonprofit American Institutes for Research, who advised the AJC on its methodology.
“This is a broad screening,” he said. “If you find something, you’re supposed to go to the doctor and follow up with a more detailed diagnostic process.”
The findings come as government officials, reeling from recent scandals, are beginning to acknowledge that a troubling amount of score manipulation occurs. Though the federal government requires the tests, it has not mandated screening scores for anomalies or investigating those that turn up.
Daria Hall, director of k-12 policy with the nonprofit The Education Trust, said education officials should take steps to ensure the validity of test results because of the critical role they play in policy and practice.
“If we are going to make important decisions based on test results — and we ought to be doing that — we have to make important decisions about how we are going to ensure their trustworthiness,” she said. “That means districts and states taking ownership of the test security issue in a way that they haven’t to date.”
‘Way too much pressure’
Both critics and supporters of testing said the newspaper’s findings are further evidence that in the frenzy to raise scores, the nation failed to pay enough attention to what was driving the gains.
“We are putting way too much pressure on people to raise scores at a very large clip without holding them accountable for how they are doing it,” said Daniel Koretz, a Harvard Graduate School of Education testing expert.
Test-score pressure is palpable in schools grappling with urban blight and poverty.
These are the schools that the 2001 No Child Left Behind Act was supposed to fix.
But at Patrick Henry Downtown Academy in St. Louis, airy red brick towers rising above the school belie a grimmer reality on the ground. Children leaving one recent afternoon passed piles of trash and a .45 caliber bullet tucked into the curb. Inside, their classrooms are beset by mold, rats, discipline problems and scandal.
Last year, the former principal — once hailed as among the district’s strongest — was accused by Missouri officials of falsifying attendance rolls to get more state money.
State investigators didn’t publicly question Henry’s test scores.
But the AJC’s analysis found suspicious scores in the school dating back to 2007. In 2010, for instance, about 42 percent of fourth-graders passed the state math test. When the class took the tests as fifth-graders the next year — with state investigators looking into cheating and other fraud allegations — just 4 percent passed math.
Experts say student learning doesn’t typically jump backwards.
Henry’s scores were consistently among the lowest in the state — except for the occasional sudden leap.
After school one recent afternoon, Deborah Dodson, who sends two children to the school, said she saw a teacher provide inappropriate one-on-one assistance during a state test. And she’s heard from other parents that teachers will give students answers.
Some students who aren’t likely to test well don’t receive tests at all, she said. “They don’t do anything by the book,” Dodson said. “That school and how they do things is not right.”
Rural, city schools flagged
The AJC used freedom of information laws to collect test scores from 50 states to look for the sort of patterns that signaled cheating in Atlanta. A Georgia investigation last year found at least 178 Atlanta educators — principals, teachers and other staff — took part in widespread test-tampering.
In each state, the newspaper used statistics to identify unusual score jumps and drops on state math and reading tests by grade and school. Declines can signal cheating the previous year. The calculations also sought to rule out other factors that can lead to big score shifts, such as small classes and dramatic changes in class size.
Some school leaders accused of cheating have attributed steep gains to exemplary teaching. But experts said instruction isn’t likely to move scores to the degree seen in the AJC’s analysis.
Through teaching alone, Wollack said, “it’s going to be pretty tough to have that sort of an impact.”
The AJC developed a statistical method to identify school systems with far more unusual tests than expected, which could signal endemic cheating such as that which occurred in Atlanta. The newspaper’s score analysis used conservative measures that highlighted extremes and were likely to miss many instances of cheating.
Big-to-medium-sized cities and rural districts harbored the highest concentrations of suspect tests. No Child Left Behind may help explain why. The law forced districts to contend with the scores of poor and minority students in an unprecedented way, judging schools by the performance of such “subgroups” as well as by overall achievement.
Hence, high-poverty schools faced some of the most relentless pressure of the kind critics say increases cheating.
Improbable scores were twice as likely to appear in charter schools as regular schools. Charters, which receive public money, can face intense pressure as supposed laboratories of innovation that, in theory, live or die by their academic performance.
Common problems unite the big-city districts with the most prevalent suspicious scores: Many faced state takeovers if scores didn’t improve quickly. Teachers’ pay or even their continued employment sometimes depended on test performance. And their students — mostly poor, mostly minority — were among those needing the most help.
The analysis, for instance, flagged more than one in six tests in St. Louis some years. In Detroit, it was one in seven.
Dozens of school systems in mid-sized cities — such as Gary, Ind., East St. Louis, Ill., and Mobile, Ala. — exhibited high concentrations of suspicious tests, too.
Though high-poverty city schools were more likely to have suspicious tests, improbable scores also showed up in an exclusive public school for the gifted on the Upper West Side of Manhattan. And they appeared in a rural district roughly 70 miles south of Chicago with one school, dirt roads and a women’s prison.
The findings call into question the approach that dominated federal education policy over the past decade: Set a continuously rising bar and leave schools and districts essentially alone to figure out how to surmount it — or face penalties.
“If you want to keep your job, keep your school out of the news, keep winning awards and advance in your career, you need to make your school look better,” said Joseph Hawkins, a former testing official with the Montgomery County, Md., school system.
Koretz, the Harvard expert, said cheating is one extreme on a continuum that, at its other end, includes gaming the test in legal ways — such as through test-prep drills — that don’t significantly increase students’ overall knowledge or skills.
Even as state test scores have soared, students’ performance on national and international exams has been more mediocre. Cheating and gaming may help explain why.
“The big picture is: Are we seeing apparent gains in student achievement that are bogus?” Koretz asked.
Decade of tumult
Test scores show that instead of progressing steadily in their academics, districts have endured a decade of tumult.
In some of the nation’s biggest cities, dynamic district leaders preached “data-driven” decision-making and even linked test scores to bonuses or principal hiring and firing decisions. Many boasted of taking a corporate approach to education, focusing on student test achievement as the single most important measure of success.
Some of the most persistently suspicious test scores nationwide, however, occurred in districts renowned for cutting-edge reforms.
In Atlanta, for instance, former Superintendent Beverly Hall won national recognition as Superintendent of the Year in 2009. State investigators later confirmed scores that year were widely manipulated by educators who assisted students improperly and outright changed tens of thousands of their answers on state tests.
In some Atlanta schools, cheating was an open secret for years. After students turned in their tests, teachers and administrators erased and corrected their mistakes — even holding a “changing party” at a teacher’s home. In another school, staff opened plastic wrap securing test booklets with a razor, then melted the wrap shut again after making forbidden copies.
State investigators accused a total of 38 principals with participating in test-tampering. One allegedly wore gloves while erasing to avoid leaving fingerprints.
Ultimately, the cheating supported a massive effort to bolster the Atlanta superintendent’s image as a tough reformer who had turned around a struggling system.
In 2002, Houston was the first winner of the Broad Prize, which has become the most coveted award in urban education. The Eli and Edythe Broad Foundation praised Houston’s intense focus on test results. More recently, Houston has been among the leaders in tying teacher pay to student test scores.
But twice in the past seven years, the AJC found, Houston exhibited fluctuations with virtually no chance of occurring except through tampering.
In 2005, scores fell precipitously in five dozen classes in 38 schools after a statistical analysis by the Dallas Morning News suggested test-tampering in Houston. The district fired teachers and principals and improved test security.
In 2011, however, as three-fourths of Houston teachers earned performance-based bonuses, scores rose improbably in a similar number of classes in the same number of schools. In the same year, Houston confirmed nine cheating allegations and fired or took other action against 21 employees.
Through Jason Spencer, a spokesman for the district, Houston officials questioned whether cheating caused all of the unusual score changes the AJC found. He said the district doesn’t think its pay-for-performance plan has made cheating more likely.
“We feel like we put a lot of safeguards in place,” he said, but added: “We know it happens. We would never pretend it’s not an issue.”
Teachers and other school staff in Atlanta were eligible for mostly small bonuses if scores hit district targets. Perhaps more worrisome for principals were the penalties: Former Superintendent Hall boasted of replacing about 90 percent of principals and told new hires they had three years to deliver high scores. Her mantra: “no exceptions, no excuses.”
Three studies of merit-pay programs did not show they consistently produce higher test scores, either legitimately or through cheating, said Matthew Springer, director of the National Center on Performance Incentives at Vanderbilt University.
Yet, he added that “it’s incredibly important that we systematically monitor these programs for opportunistic gaming of the system.”
Pushback from officials
Some school districts and states have taken an apathetic, if not defiant, stance in the face of cheating accusations in recent years.
The AJC sent detailed findings to districts with some of the most suspicious clusters of scores. For those not already publicly looking at cheating, the responses were similar: Officials said they were unaware of most anomalies, but protested characterizing the score changes as cheating.
Several local and state school officials objected to conducting the analysis at all, saying it doesn’t consider enough variables.
Some districts simply denied any problems exist. Detroit, for instance, claimed its scores were not “unusual or out of line in any way” and that Michigan officials had not identified irregularities “with respect to an erasure analysis, suspected cheating, or any other issue.”
In fact, Michigan’s education agency identified six Detroit schools as having statistically unlikely gains on a state test in 2009. At one school, the state determined, sixth-graders averaged 7.4 wrong-to-right erasures. Their peers statewide averaged fewer than one such change.
Analyzing Detroit’s scores from 2008 and 2009, the AJC found suspicious swings in 14 percent of classes. The statistical probability: zero.
Regardless, Detroit officials offered an explanation that experts have said is among the least likely: better teaching.
Steven Wasko, an assistant superintendent in Detroit, said the district has offered before- and after-school programs, expanded summer school, and added extra reading and math instruction. “Increases in student performance,” Wasko said in an email, “could be attributed in part to these factors.”
In a statement, St. Louis school district officials acknowledged the strangeness of score changes, but disagreed that cheating was to blame. They said neither the district nor state education officials have any “credible evidence that testing improprieties have occurred at the schools in question.”
Officials acknowledged, however, that the district has a cheating investigation open at one school. The state said that since 2010 it has received allegations of cheating at two other St. Louis schools identified as suspicious by the AJC analysis. Accusations of cheating persist.
State officials say they do not screen test scores for possible cheating and do not consider unusually high gains to be a sign of test-tampering — if schools provide an explanation.
“We hope to see great gains in our proficiency levels,” said Michele Clark, a spokeswoman for the Missouri education department.
Dallas officials said that when irregularities surfaced several years ago, they instituted new test security measures and started screening for anomalies.
Few big-city districts have attacked cheating as aggressively as Baltimore.
After he became the district’s chief executive in 2007, Andrés Alonso heard a whistle-blower complain at a PTA meeting about the district’s lax investigation into cheating allegations at her school.
With accused educators sitting nearby, Alonso recalled recently, the room became “a deafening vacuum.”
Alonso ordered a new investigation, which spread into 15 other schools. The district posted independent monitors in each school during tests. In the suspected schools, scores fell dramatically. In other schools, scores continued to rise.
Alonso asked state officials to check test papers for illicit erasures and changes. Their analysis confirmed his suspicions.
At Fort Worthington Elementary, for instance, as many as 20 mistakes were corrected on some students’ tests, often in a lighter shade of pencil.
All of Fort Worthington’s classes posted improbable gains in 2008, the AJC’s analysis shows. The performance level held for two more years, when the school faced the threat of state takeover. After the cheating was detected, statistically unlikely score drops multiplied, occurring in three-quarters of the school’s classes. Similar patterns show up across the district.
Sitting outside the school in her aging station wagon one late winter day, Vernetta Jones-Marshall said Fort Worthington is doing the best it can.
“I don’t even know if it was really a true statement,” Jones-Marshall, 57, said of the cheating allegations as she waited to pick up her son, a fifth-grader. “We didn’t make a big deal about it.”
Cheating is a big deal to Alonso, however.
Most educators act with integrity, he said, but others “feel a sense of impunity” because school officials haven’t always held cheaters accountable.
“I was doing this before the Atlanta story broke,” he said. “This was me feeling that nothing mattered more than the integrity of the school system.”
Call for vigilance
Leaders need to maintain that tough stance even after cheating disappears from the headlines, experts say.
In Dallas, for instance, the score analysis shows the number of suspicious gains dropped after cheating allegations surfaced in late 2004 — but then began inching up again a few years later.
For years, Los Angeles’ scores were among the least suspicious for big-city districts. But when California stopped conducting routine erasure analysis in 2008 for budget reasons, the number of improbable score changes in L.A. climbed steeply.
States and districts find little advice when they do decide to conduct erasure or statistical screenings of test scores.
Federal education officials and testing experts have begun working on new recommendations for detecting and investigating test-score anomalies.
Wollack, the Wisconsin testing expert, said there is room to improve. “Some of the investigations that have taken place in the past have been less than thorough, have been less exhaustive than they should have been,” he said. “Cheating went undetected as a result.”
Districts don’t have a big incentive to unearth ugly truths about their own testing programs. What’s more, most screening methods miss instances of cheating by setting high thresholds in an effort not to falsely identify innocent schools.
“It’s clear there are schools, there are districts, that are under that threshold that are still engaged in some level of misconduct,” Wollack said.
Critics of testing have complained for years that increased pressure brought on by accountability measures leads to more testing abuses.
Education historian and New York University Professor Diane Ravitch said the incessant focus on testing has eroded the quality of instruction.
“All of this is predictable,” said Ravitch, a former top U.S. Department of Education official who in recent years reversed her support for testing and tough accountability measures. “We’re warping the education system in order to meet artificial targets.”
Through programs such as Race to the Top, federal education officials have pushed states to adopt more aggressive teacher evaluation systems that, typically, consider test scores.
“Whatever the stakes were under No Child Left Behind,” Ravitch said, “they are going to be much higher, now that teachers are being told your scores are going to be public and you’re going to be fired if they don’t go up X number of years in a row.”
But Daria Hall, of the Education Trust, said most educators don’t cheat, and testing data is essential for determining if students have basic skills — such as the ability to read.
“What parent doesn’t want to know how their child is doing in reading and in math? What teacher doesn’t want to know how their student is doing?” she said. “You can’t take away the source of the information. We have to make the information better.”
Crisis of confidence
For parents, questions of academic integrity can lead to a crisis of confidence.
The chronically low-performing Nashville district illustrates the conundrum. Test scores in some of the district’s schools have alternately soared and swooped to improbable degrees.
Nashville school officials said the data raises concerns about their effectiveness as educators, but not cheating. They echoed other districts’ objections to the analysis, including their relatively high percentage of students learning English and the number of students changing schools from one year to the next.
In Hermitage, a working-class section east of downtown Nashville, Megan McGowan said she was torn about whether to send her son to Dupont Tyler Middle School.
Tests carry too much weight, she said, and teachers face tremendous pressure to produce results. Still, she said, cheating is inexcusable. If it happened at Dupont Tyler, she said, she’d think twice about sending her son there.
“I expect teachers to be ethical,” she said.

Norm's Notes

Sunday, March 25, 2012

Cheating our children: Suspicious school test scores across the nation

Cheating our children: Suspicious school test scores across the nation

Related

1 comment:

Counter

About Me