State Guts Its Test of Reading: Union Study Sees Inflated Scores

BY ELIZABETH GREEN - Staff Reporter of the Sun

September 7, 2007
URL: http://www.nysun.com/article/62089

The difficulty of a reading test used to judge students across New York State dropped by as many as six grade levels between 2004 and 2005, according to an internal study by the New York City teachers union obtained by The New York Sun.


The study, written in March 2006, found that passages in the 2005 test hovered around third- and fourth-grade reading levels, down from a ninth-grade level in 2004. It also found that the 2004 test was characterized by longer passages, smaller print, crammed text, and more complex questions, such as asking a student to make an inference versus asking the main idea. Despite this apparent drop in difficulty, however, the number of correct answers needed to pass — known as the "cut score" — was just slightly higher in 2005 than in 2004.


Schools across the state had reported fourth-grade reading gains in some cases of more than 40 points in 2005, and New York City pupils registered a 10-point gain on average.

In New York City, low scores on state tests can prevent students from advancing to the next grade level or lead to a school being shut down.


Coming during Mayor Bloomberg's re-election campaign, the touted surge raised many eyebrows, including those of the then-chairwoman of the City Council's Education Committee, Eva Moskowitz of Manhattan, who held a six-hour hearing on the test scores, and the city's public advocate, Betsy Gotbaum, who handed copies of the 2004 and 2005 tests to researchers at the United Federation of Teachers for examination.


Shown the UFT study yesterday afternoon, a state education department spokesman, Tom Dunn, called its conclusion "completely wrong."


"The passages combined with the questions are at the same grade level for both years," he said.

The UFT study was done in consultation with at least three researchers, according to a March 1, 2006, memo to the union president, Randi Weingarten.


In an interview yesterday, Ms. Weingarten said she chose not to publicize the study out of concerns that doing so would make her appear "anti-test." She also said the study could not be considered comprehensive because her researchers are not psychometricians and lack access to some specific data about the test.


She said the study did concern her.


"It's part of why I keep saying, be careful about data. Standardized test scores can't be used for these high-stakes measures for kids or for teachers," she said.


The director of New York University's Center for Research on Teaching and Learning, Robert Tobias, said the UFT study added empirical confirmation to the concerns he had raised in 2005. But he said the study's grade-level measure, called the Fry Readability test, which uses two indicators to determine "readability" — the average number of syllables per 100 words and the average number of sentences — was too simplistic. A test tabulating more nuanced characteristics — "vocabulary, the complexity of the sentence structure, the number of subordinate clauses, that type of thing" — would be more conclusive, he said.


Mr. Tobias praised the part of the UFT study that describes more subtle differences between the tests, such as differences in the length of passages and question types.

"To me, that's some pretty powerful stuff," he said.


A freelance data analyst who has worked for the Department of Education, Frederick Smith, said a readability study he has conducted using the Fry method — but testing every word on the tests rather than a sample — is more precise than the UFT study. His results, which he showed to The New York Sun, found a drop of five reading levels, to third-grade in 2005 from eighth-grade in 2004.



Mr. Smith said the results confirm his longstanding calls for an independent review board to oversee state testing. Since as long ago as a 1982 Newsday article, Mr. Smith has called for a testing ombudsman, citing the importance of an independent reviewer to temper politicians' inclination to boast of gains under their watch.


Michael Cohen, a former education adviser to President Clinton who now works with 30 states to improve their annual assessments through his nonprofit group, Achieve, said he believes that some states have "dumbed down" tests in response to pressures to get better academic results — with deleterious results.


"The tests, generally speaking, are not all that rigorous to begin with. So almost any dumbing-down is moving the expectations in the wrong direction," he said.


Mr. Cohen said that he would be very surprised, however, if New York was one of those states. "I do know Rick Mills," he said, referring to the state education commissioner. "This hardly sounds like what he would do."


A vice president of a Washington, D.C.-based think tank, the Thomas Fordham Foundation, Michael Petrilli, said the federal No Child Left Behind law has caused several states to make their tests easier — some publicly and some without announcement, a conclusion that will be issued in a Fordham report due later this month, Mr. Petrilli said.


Responding to a Daily News story questioning the reliability of state math tests published on the first day of the new school year Tuesday, Mr. Bloomberg pointed out that New York City can use state tests to compare its schools to schools in other districts across the state. Between 1999 and 2007, the portion of New York City students who passed the state reading test has risen by more than 15 percentage points, to 51% from 35%, while cities such as Buffalo have had single-digit increases.