Monday, March 28, 2011

When test scores soared in D.C., were the gains real?

When test scores soared in D.C., were the gains real?


WASHINGTON — In just two years, Crosby S. Noyes Education Campus went from a school deemed in need of improvement to a place that the District of Columbia Public Schools called one of its "shining stars.


By Jack Gruber, USA TODAY

EnlargeClose
Marvin Tucker raised questions about high test scores of daughter Marlana, right, now 16, that showed math proficiency despite her struggle with the basics when she attended D.C.'s Noyes.
Standardized test scores improved dramatically. In 2006, only 10% of Noyes' students scored "proficient" or "advanced" in math on the standardized tests required by the federal No Child Left Behind law. Two years later, 58% achieved that level. The school showed similar gains in reading.
Because of the remarkable turnaround, the U.S. Department of Education named the school in northeast Washington a National Blue Ribbon School. Noyes was one of 264 public schools nationwide given that award in 2009.
Michelle Rhee, then chancellor of D.C. schools, took a special interest in Noyes. She touted the school, which now serves preschoolers through eighth-graders, as an example of how the sweeping changes she championed could transform even the lowest-performing Washington schools. Twice in three years, she rewarded Noyes' staff for boosting scores: In 2008 and again in 2010, each teacher won an $8,000 bonus, and the principal won $10,000.
A closer look at Noyes, however, raises questions about its test scores from 2006 to 2010. Its proficiency rates rose at a much faster rate than the average for D.C. schools. Then, in 2010, when scores dipped for most of the district's elementary schools, Noyes' proficiency rates fell further than average.
A USA TODAY investigation, based on documents and data secured under D.C.'s Freedom of Information Act, found that for the past three school years most of Noyes' classrooms had extraordinarily high numbers of erasures on standardized tests. The consistent pattern was that wrong answers were erased and changed to right ones.

Click to view documents

This is a series of documents obtained by USA TODAY through public-records requests. It details a back-and-forth between two District of Columbia agencies on test-score investigations.
Noyes is one of 103 public schools here that have had erasure rates that surpassed D.C. averages at least once since 2008. That's more than half of D.C. schools.
Erasures are detected by the same electronic scanners that CTB/McGraw-Hill, D.C.'s testing company, uses to score the tests. When test-takers change answers, they erase penciled-in bubble marks that leave behind a smudge; the machines tally the erasures as well as the new answers for each student.
In 2007-08, six classrooms out of the eight taking tests at Noyes were flagged by McGraw-Hill because of high wrong-to-right erasure rates. The pattern was repeated in the 2008-09 and 2009-10 school years, when 80% of Noyes classrooms were flagged by McGraw-Hill.

By Manuel Balce Ceneta, AP
Michelle Rhee, then-chancellor of D.C. schools, visits with J.O. Wilson Elementary third-grader Kmone Feeling last August.
On the 2009 reading test, for example, seventh-graders in one Noyes classroom averaged 12.7 wrong-to-right erasures per student on answer sheets; the average for seventh-graders in all D.C. schools on that test was less than 1. The odds are better for winning the Powerball grand prize than having that many erasures by chance, according to statisticians consulted by USA TODAY.
"This is an abnormal pattern," says Thomas Haladyna, a professor emeritus at Arizona State University who has studied testing for 20 years.
A trio of academicians consulted by USA TODAY — Haladyna, George Shambaugh of Georgetown University and Gary Miron of Western Michigan University — say the erasure rates found at Noyes and at other D.C. public schools are so statistically rare, and yet showed up in so many classrooms, that they should be examined thoroughly.
USA TODAY examined testing irregularities in the District of Columbia's public schools because, under Rhee, the system became a national symbol of what high expectations and effective teaching could accomplish. Federal money also was at play: Last year, D.C. won an extra $75 million for public and charter schools in the U.S. government's Race to the Top competition. Test scores were a factor.
USA TODAY initially looked at Noyes only because of its high erasure rates. Later, the newspaper found that Wayne Ryan, the principal from 2001 to 2010, and the school had been touted as models by district officials. They were the centerpiece of the school system's recruitment ads in 2008 and 2009, including at least two placed in Principal magazine.
"Noyes is one of the shining stars of DCPS," one ad said. It praised Ryan for his "unapologetic focus on instruction" and asked would-be job applicants, "Are you the next Wayne Ryan?"
In response to questions from USA TODAY, Kaya Henderson, who became acting chancellor of the D.C. Public Schools after Rhee resigned in October, said last week that "a high erasure rate alone is not evidence of impropriety."
D.C. "has investigated all allegations of testing impropriety," Henderson said. "In those situations in which evidence of impropriety has been found, we have enforced clear consequences for the staff members involved, without hesitation."
Henderson, who was Rhee's deputy, said the system would identify only schools where violations of security protocol were found. "For the majority of schools" investigated, there was "no evidence of wrongdoing," she said. Out of fairness to staff members, she said, she declined to identify all the schools that were investigated.
There can be innocent reasons for multiple erasures. A student can lose his place on the answer sheet, fill in answers on the wrong rows, then change them when he realizes his mistake. And, as McGraw-Hill said in a March 2009 report to D.C. officials, studies also show that test-takers change answers more often when they are encouraged to review their work. The same report emphasizes that educators "should not draw conclusions about cheating behavior" from the data alone.
Haladyna notes, however, that when entire classrooms at schools with statistically rare erasures show fast-rising test scores, that suggests someone might have "tampered with the answer sheets," perhaps after the tests were collected from students. Although not proof of cheating, such a case underscores the need for an investigation, he says.
At Noyes, USA TODAY found several grades with wide swings in their proficiency rates from one year to the next. In 2008, 84% of fourth-grade math students were listed as proficient or advanced, up from 22% for the previous fourth-grade class. The math scores for the fourth-grade class in 2010 dropped off to 52% proficient or better.
For the school as a whole, test scores seemed to ride a roller coaster: In reading, from 2006 through 2010, the annual percentage of all Noyes students testing as proficient or higher went from 24% to 44% to 62% to 84% to 61%, according to official records. Reading scores at all D.C. elementary schools slumped on average by 4 percentage points from 2009 to 2010; Noyes' scores plunged 23 points.

'It's our children'

In 2008, the office of the State Superintendent of Education recommended that the scores of many schools be investigated because of unusually high gains, but top D.C. public school officials balked and the recommendation was dropped.
After the 2009 tests, the school district hired an outside investigator to look at eight D.C. public schools –– one of them was Noyes, USA TODAY learned — and to interview some teachers.
John Fremer, president of Caveon Consulting Services, the company D.C. hired, says the investigations were limited. The teachers were asked what they knew about the erasure rates but not whether cheating had taken place, Fremer says. They told Caveon that they "did what they were supposed to do and they didn't do anything wrong," he says.
Henderson, the D.C. chancellor, says D.C. educators interviewed by Caveon "gave specific reasons for high erasure rates. ... Some emphasized to their students that (they) ... should always go back, review their answers and make corrections, if needed.
"Other teachers," she says, "encouraged students to eliminate wrong answers in the test booklet by marking an 'X' next to wrong answers, which could account for an unusual number of erasures if students marked their 'X' on the answer sheet instead of the test booklet."
School district officials would not release the reports Caveon compiled. Caveon has been hired again to investigate the results of 2010 tests in which 41 DCPS schools, including Noyes, had at least one classroom flagged for high erasure rates. USA TODAY could not determine which schools are being scrutinized.
Parents and some State Board of Education members say they were never told which schools had high erasure rates or other irregularities.
Zell Foster, whose daughter Paige is an eighth-grader at Noyes, says that even if the school district didn't find any violations at the school in 2009, parents should have been informed that an investigation was underway. She says neither the school nor the district sent home notices about the erasures.
"It's not fair. It's our children," Foster says. "We shouldn't be in the dark."
Mark Jones, a member of the State Board of Education, says district officials appear not to have dug deeply into why some schools had such high erasure rates, but if they did, they have not shared what they found. He says parents need to know because they make decisions about where to send their children to school based on test scores.
"We should clearly have the data, whether it's good or bad," says Jones, who has two daughters in a public elementary school. The district at the very least should have told parents "we have anomalies and we are investigating," Jones says.
Ryan declined to answer questions from USA TODAY through the district's spokeswoman and did not respond to telephone calls or e-mail. Last year, he was promoted to instructional superintendent in the D.C. schools, overseeing a cluster of schools.
Rhee resigned after the mayor who appointed her, Adrian Fenty, lost his re-election bid last fall. She has since organized a non-profit, StudentsFirst, which is trying to raise $1 billion to promote education reform. When reached by telephone, Rhee said she is no longer the chancellor and declined to comment further.
D.C. officials declined to let USA TODAY visit schools or talk to principals, including Adell Cothorne, the principal who succeeded Ryan at Noyes for the 2010-11 school year.

An impasse over erasures

McGraw-Hill's practice is to flag only the most extreme examples of erasures. To be flagged, a classroom had to have so many wrong-to-right erasures that the average for each student was 4 standard deviations higher than the average for all D.C. students in that grade on that test. In layman's terms, that means a classroom corrected its answers so much more often than the rest of the district that it could have occurred roughly one in 30,000 times by chance. D.C. classrooms corrected answers much more often.
In 2008, the Office of the State Superintendent of Education (OSSE) — the D.C. equivalent of a state education department –– asked McGraw-Hill to do erasure analysis in part because some schools registered high percentage point gains in proficiency rates on the April 2008 tests.
Among the 96 schools that were then flagged for wrong-to-right erasures were eight of the 10 campuses where Rhee handed out so-called TEAM awards "to recognize, reward and retain high-performing educators and support staff," as the district's website says. Noyes was one of these.
Rhee bestowed more than $1.5 million in bonuses on principals, teachers and support staff on the basis of big jumps in 2007 and 2008 test scores.
At three of the award-winning schools — Phoebe Hearst Elementary, Winston Education Campus and Aiton Elementary — 85% or more of classrooms were identified as having high erasure rates in 2008. At four other schools, the percentage of classrooms in that category ranged from 17% to 58%.
Although all of the experts consulted by USA TODAY said such aberrations should trigger investigations at the school level, that did not happen in D.C. in 2008. No schools were investigated.
In November 2008, Deborah Gist, then the state superintendent of education, recommended that D.C. public schools and several charter schools investigate why their erasure rates were so high. "It is important to note that these (data) analyses do not suggest reasons for the high erasure rates," Gist wrote to the schools. "However, it is important that all procedures available to us are employed to guarantee the validity of the state assessment system."
Seven charter schools responded to OSSE and carried out probes. Gist's proposal met resistance from Rhee's staff, documents obtained by USA TODAY show. Memoranda flew back and forth for five months as D.C. school officials questioned the methodology and the rationale for an investigation.
Documents show that Rhee's chief data and accountability officer, Erin McGoldrick, requested more information from OSSE in February 2009. She asked for more details on the two lists of schools OSSE submitted for possible investigation. The lists were compiled using two different statistical methods for identifying examples of high wrong-to-right erasures. Noyes was on both lists.
"DCPS must be confident in the data provided before undertaking a full investigation," McGoldrick said, because of the "disruption and alarm an investigation would likely create at schools."
In April, state superintendent Gist left Washington to take a job as head of Rhode Island's state school system. Her successor, Kerri Briggs, then dropped the request for D.C. public schools to investigate its schools. Both Gist and Briggs, now director for education reform at the George W. Bush Institute in Texas, declined to comment.
A memo later prepared by Victor Reinoso, Washington's deputy mayor for education, noted that McGraw-Hill itself had cautioned that officials "should not draw conclusions about cheating behavior" from the data analysis. A USA TODAY review of the McGraw-Hill document, however, showed that a company analyst also said the data could properly be used to identify "possible cheating incidents for follow-up investigation."

The balance of power

The impasse over the 2008 scores illustrates the unusual balance of power within the D.C. school system. In 2007, when then-mayor Fenty took charge of D.C.'s failing schools, the D.C. school board was eliminated and replaced by a state board of education with little power. Fenty won the right to name the chancellor of D.C. public schools and in mid-2007 appointed Rhee.
From the start, Rhee emphasized a need to raise scores, restore calm to chaotic schools and close those with lagging scores and small enrollments. She paid bonuses to principals and teachers who produced big gains on scores. She let go dozens of principals and fired at least 600 teachers. Others retired or quit.
Turnover was brisk. Richard Whitmire, author of The Bee Eater, a biography of Rhee, reported that Rhee hired 1,918 teachers during her three years in office –– about 45% of those on the payroll last October. Only 2,318 current teachers had been hired before Rhee took charge.
The pressure on principals was unrelenting, says Aona Jefferson, a former D.C. principal who is now president of the Council of School Officers, representing principals and other administrators. Every year, Jefferson says, Rhee met with each principal and asked what kind of test score gains he would post in the coming school year. Jefferson says principals told her that Rhee expected them to increase scores by 10 percentile points or more every year. "What do you do when your chancellor asks, 'How many points can you guarantee this year?' " Jefferson says. "How is a principal supposed to do that?"
Rhee churned through principals. TheWashington Post reported that Rhee appointed 91 principals in her three years as chancellor, 39 of whom no longer held those jobs in August 2010. Some left on their own, either resigning or retiring; other principals, on one-year contracts, were let go for not producing quickly enough.
Union officials say the pressure for high test scores may have tempted educators to cheat.
"This is like an education Ponzi scam," says Nathan Saunders, head of the Washington Teachers' Union. "If your test scores improve, you make more money. If not, you get fired. That's incredibly dangerous."
When D.C. administrators resisted investigating the 2008 scores, there was no counterweight to force the issue. The state board is empowered only to advise OSSE. Mary Lord, a board member with a teenager who attends a D.C. high school, is critical of the decision not to investigate the 2008 scores. "If you are going to add all this weight" to testing, "hanging the principals' reputations ... and the teachers' pay on it, you have to make sure it is totally accurate," Lord says.
Board members say that, like parents, they have been kept in the dark about testing irregularities. The state board wasn't aware, Lord says, of the dispute between the superintendent's office and Rhee until its members saw reports in TheWashington Post in the fall of 2009. She says she did not see the erasure analysis or the lists of schools flagged by McGraw-Hill until USA TODAY shared its copies.
After Rhee gave bonuses to educators in some schools that posted big gains in test scores in 2007 and 2008, there was little incentive to examine those scores, Lord says. "You've handed out these big bonuses. What are you going to do? Take them back?" she says. "It's a bombshell. It's embarrassing."

'A total disconnect'

Questions were raised about high test scores at Noyes well before 2008.
A former Noyes parent, Marvin Tucker, says he suspected something was wrong in 2003, when the test scores his daughter, Marlana, brought home from school showed she was proficient in math.
Tucker says he was skeptical because the third-grader was getting daily instruction from a private tutor yet struggled with addition and subtraction. "She was nowhere near where they said she was on the test," he says. "I thought something was wrong with the test."
He questioned Ryan, the principal, and teachers about his daughter's scores but no one could explain how she had scored so high, Tucker recalls. Ultimately, Ryan barred him from the school for a year, saying he had threatened staff members, Tucker says. Tucker denies that.
Tucker also points out that if his daughter was proficient as a third-grader, that didn't last. When Marlana moved on to middle school elsewhere in D.C., her test scores fell and she no longer was considered proficient in math, he says.
Tucker shared his concerns about testing and other issues at Noyes with other parents. A small group went to the school board. "We tried to go through the chain of command," says Debbie Smith-Steiner, a neighborhood activist who worked with Tucker.
Parents even staged a small protest at the school board's offices, she says. Nothing changed and the group eventually let it go, Smith-Steiner says. "There wasn't anything we could do. You are fighting these battles and nobody is listening. Nobody is saying, 'How are these test scores going up so much?' "
Councilman Tommy Wells, then a school board member, says he relayed the parents' concerns to school officials. He says those officials assured him the allegations were checked out and nothing was confirmed. "There were parents and community members who did not like the principal," Wells says. "But we took their concerns seriously."
Several teachers at Noyes also were dubious about the legitimacy of test scores, describing what one called "a disconnect" between the high scores and how their students performed in class.
Ernestine Allen, a former teacher who taught pre-K as well as second- and fourth-grades for five years at Noyes, says it was hard to trust the scores of some students entering her classes. Their scores showed they were doing well when, she says, they were still struggling with reading.
"You wonder, how is it that this student got such a high score?" Allen says. She says teachers talked about the problem among themselves. But, she says, "Who do you tell?"
Allen left Noyes in 2006 after a series of run-ins with Ryan, which included a poor evaluation and an incident in which he called the police on her son, Preston. A police report shows Preston Allen, then 31, went to Ryan's office in October 2005 and asked the principal to stop using profanity when he talked to his mother. Ryan said the situation would be handled "administratively," the report said. No arrests were made.
Another Noyes instructor who taught more recently than Allen agrees with her that test scores were unreliable. "Something doesn't make sense," says the former teacher who spoke on condition of anonymity out of fear of retaliation. "It's a total disconnect between what scores showed and what I could see in the classroom."
The former teacher also says "there was no way" the students themselves could have erased their own answers and changed them to the right ones. "They didn't check their work," the teacher says.

A limited investigation

The tests administered in April 2009 produced another round of score improvements for D.C. schools. The proficiency rate districtwide in reading for elementary schools rose 3 percentage points over 2008; the math rate jumped 7 points.
Data obtained by USA TODAY show that, after those tests, 46 D.C. public schools were flagged by McGraw-Hill for having classrooms with high rates of wrong answers changed to right ones. Last October, five of those schools won TEAM awards — and bonuses for teachers and principals — for their high scores. It was the second win for Noyes' staff and the first for J.O. Wilson Elementary, another school that regularly has had more than 80% of its classrooms flagged for high erasure rates.
OSSE chose eight D.C. public schools plus four charter schools for investigation. District officials would not identify the eight D.C. public schools, but USA TODAY was told by a former official that Noyes was one of them.
Fremer, president of Caveon Consulting Services, the Utah company hired by D.C., acknowledges the investigations were limited and focused mainly on process. "Did everyone who should have received training (on how to give tests) receive training? Was there a mechanism in place for checking out the test booklets? How were they stored?" he says in describing the questions.
When Caveon interviewed individual teachers, Fremer says, an official from the school district was always present and occasionally a principal sat in. Teachers were asked about why erasure rates were so high, Fremer says, but he adds: "We didn't ask if teachers cheated."
D.C. school officials did not ask Caveon to do its own analysis of the test data, Fremer says. For other investigations, he says, Caveon has gone to the testing company to examine the tapes of the scanning machines that detected which wrong answers were erased and changed to right. It is helpful, he says, to examine each student's answers to determine, for example, whether students got hard questions right but missed easy ones. That unlikely outcome can indicate tampering.
After Caveon's investigation, D.C. school district officials cleared all but one of the eight public schools originally on the list. OSSE approved those findings, according to documents USA TODAY obtained.
At Burrville Elementary, where half of the school's classrooms had been flagged for high wrong-to-right erasure rates by McGraw-Hill, the conclusion was that one teacher had wrongly cleaned up stray pencil marks on student answer sheets. That was not allowed, OSSE said in a letter to Rhee. In that classroom, students' math and reading scores were invalidated.
At another school, Stanton Elementary, where wrong-to-right erasures in one fourth-grade class were about 10 times the district average, no violation was found. But an unidentified teacher was banned from administering future tests. The letter sent to Rhee by OSSE did not explain why.
Ted Trabue, president of the State Board of Education, agrees the 2009 investigation was limited. But he credits OSSE, which sets test security policy, for tightening the rules since 2009. For this year's testing season, which starts April 4, OSSE added a security seal to the outside of the test booklets that can be broken only by students.
Acting chancellor Henderson said "stricter protocols for receiving, storing and returning test materials" are now in place, and each school has been assigned an independent observer from the central office to monitor test administration.
There are people here — parents, politicians and some educators — who also want D.C. to be more aggressive and open about irregularities.
Jefferson, the head of the Council of School Officers, says that if questions remain about the legitimacy of test scores at schools such as Noyes, the school district should not just conduct a thorough investigation, it should also tell the community about it. "You don't want this cloud hanging over them," she says. "You don't want their achievement tainted. ... They did all this work to be a Blue Ribbon School. When you don't say anything, you leave a lot of questions."
Lord, the state board member, says it's hard to fix a problem when there has been no open discussion about it. Without a public debate, Lord says, "it begs the question: Was this a strong school because they were cheating all along?"
Posted by at

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)