Ravitch Responds to Tweed's Jennifer Bell-Elwanger

The DOE’s attempt to refute Diane Ravitch's recent oped, was referenced in the latest Principals Weekly.

As Klein wrote:

“As you know, the law granting the Mayor control of the schools expires at the end of June, and discussions about our governance structure are taking place throughout the City and the State. This is the right time to think about the past seven years—and to focus on the governance law and how it can best assure that New York City schools and students excel going forward. In that regard, a number of you have asked me about a recent Op-Ed by Diane Ravitch, which did not fully or accurately portray our record under mayoral control. I have asked the executive director of our Research and Policy Support Group, Jennifer Bell-Ellwanger, to respond to the specifics in Dr. Ravitch’s presentation, and you can read her response HYPERLINK ; http://schools. nyc.gov/NR/ rdonlyres/ 360A5553- BEA6-4156- BA3A-4EC0B0A3665 E/0/090428_ jbe_response. pdf here. It is important to correct the record so that our discussions may proceed on a firm foundation of fact.”



Jennifer Bell-Elwanger is Klein’s statistical adviser who announced to the PEP and at the Assembly hearings on mayoral control that the lack of any statistically significant increase in NAEP scores for NYC in three out of the four areas tested – 4th and 8th grade reading and 8th grade math since 2003 -- was not important, as long as there was “directionality”.



Diane’s response to her memo is below.


Jennifer Bell-Ellwanger’s analysis does not refute the data that I offered in my article in The New York Times. The editor at the Times required documentation for every single fact in the article, and I supplied it.

Her biggest error is that she uses the wrong baseline data in presenting state test scores. She chooses 2002 as her starting point, which is wrong. The baseline begins when Chancellor Klein’s reforms begin, and not sooner. The Legislature approved mayoral control legislation in June 2002, and Chancellor Klein assumed office in August 2002. He spent the next six months conferring with consultants. He and the mayor announced their plans and their pedagogical choices in January 2003. At that very time, students in the city and state were taking the state tests. The chancellor had done nothing in the schools prior to January 2003 to raise student achievement. His reforms were introduced into the schools in September 2003. Thus, it is inappropriate for Dr. Bell-Ellwanger to take credit for any gains registered on the state tests that were administered in January 2003.

The reason that Dr. Bell-Ellwanger wants to claim credit for that particular year is that there were huge gains across the city in both reading and mathematics. But at the time, Chancellor Klein did not claim credit for the gains of that year. It was widely understood that his program would not start until September 2003.

David Herszenhorn wrote in the Times on May 21, 2003, “New York City public school students posted sharp gains on the state’s standardized reading and writing test this year, with striking double-digit jumps in some of the city’s poorest and historically lowest-performing school districts…In both the city and the state, black and Hispanic fourth graders significantly narrowed the gap with white and Asian students…City officials, who might otherwise have been jubilant about yesterday’s results, offered a muted reaction, saying that the gains were not broad enough and that the school system as a whole was failing at least half the city’s children.”

A few months later, Elissa Gootman wrote in the Times on October 22, 2003: “Fourth graders across the state made stunning gains in their math scores last spring, with even sharper increases in New York City…In the city, news of the gains, which were particularly pronounced in the Bronx and in some of the poorest-performing districts, elicited cheers among teachers and principals. But not everyone greeted the news so enthusiastically:



“The suggestion that city schools were on the upswing put Chancellor Joel I. Klein, who is overhauling them, in a tricky position. While the chancellor’s critics pounced upon the higher scores as evidence that the school system did not need such an overhaul, some of his allies acknowledged that he would now be under even more pressure to show gains next spring.

“Mr. Klein’s reaction to the good news was muted, as it was to news of higher reading scores last spring.”



If Chancellor Klein thought he had produced those sharp gains, why was his reaction “muted”? Obviously he did not take credit for the scores because the scores were announced before his reforms had been introduced. Why does he take credit for them now? Quite simply, the gains in that one year were so large that they became irresistible. Fourth grade reading scores from 2002-2003 were up by 6 points, a larger gain than in any subsequent year. Fourth grade math scores jumped by an astonishing 14.7 points, a larger gain than in any subsequent year under Chancellor Klein.



In the interests of honesty, Ms. Bell-Ellwanger must deduct the scores of 2002-2003 from her claims for this administration and revise the city’s gains on state tests accordingly.



Yes, I agree that the massive investment in test prep, interim assessments, and testing, as well as bonuses for principals and some teachers tied solely to test scores has certainly produced increases in state test scores. No question about it. But it is worth remembering that the city’s education budget has grown from $12.5 billion to $21 billion, and one would certainly expect that expenditures of that size should buy some improvements.



I don’t agree with Ms. Bell-Ellwanger that the state test scores are somehow more valid than the federal NAEP scores. Quite to the contrary. The NAEP test is a far superior test to the state tests; Congress has invested tens of millions of dollars in making NAEP the best testing program in the nation. As for the state tests, they are not reliable because the Regents and the State Education Department have repeatedly changed the testing pool by manipulating which groups of students are tested and which are not. Just this past September, the Regents decided that students who are LEP and who have tested proficient may be excluded from state testing for up to two years after they passed the English proficiency test. So, once again, the testing pool has changed, and some potentially low-scoring students may be excluded. I feel that is safe to predict another big increase in state scores in a few weeks or months because of the removal of these students from the testing pool.



The claim that the state tests are more valid because students practice for them is silly. I recommend that Dr. Bell-Ellwanger read Harvard Professor Dan Koretz’s recent book Measuring Up, in which he demonstrates that when students practice repeatedly for a test, the test becomes less valid. I suspect she already knows this.



If students do well on the state test, but not on other tests because they didn’t practice for those specific tests, then they really haven’t mastered the skills. If all they learn in a NYC public school is to take state tests, then their skills will not be transferable to reading in college or in the workplace. I have never heard anyone claim—aside from the NYC Department of Education—that NAEP is less valid because students don’t practice for it. The point of NAEP is that it is an audit test. As an audit test, it is far more consistent and meaningful than the state tests.



Dr. Bell-Ellwanger gets the NAEP results wrong. As I said in the article, NYC showed significant progress in fourth-grade math, but those gains were suspect because of the extraordinary number of students who were given accommodations (extra time, extra help). I served on the National Assessment Governing Board for seven years, and I know that a sharp change in the accommodations rate raises questions. In fourth grade math, 25% of NYC students got accommodations; this compares to only 12% who received them in 2003. Why the sharp increase? NYC’s accommodations rate far exceeded that of any other city. Los Angeles, with many more limited-English- proficient students gave accommodations in this subject and grade to only 8%.



On the NAEP fourth-grade reading test, she again errs by giving the Klein regime credit for the big gains of 2002-03, before he had introduced Balanced Literacy as the standard, mandated reading curriculum in the elementary schools. The correct baseline year is not 2002 but 2003. New York City fourth-graders made no significant gains in reading from 2003 to 2007. I have often wondered why the Chancellor did not replace Balanced Literacy after he saw these unimpressive results. NAEP showed no significant gains in fourth grade reading for black students, white students, Hispanic students, Asian students or lower-income students. NAEP found no narrowing of the gap between the city and state from 2003 to 2007.



On the NAEP eighth-grade reading test, NAEP showed no significant gains for any racial or ethnic group from 2003 to 2007. Fifty percent of black students and 49 percent of Hispanic students in eighth grade were “below basic,” which is the lowest classification for a score (only 20 percent of whites and 21 percent of Asians scored so low on the reading test). And these were students who had spent four years in the Children First reforms.



There were some shifts in the racial gaps. But once again, Dr. Bell-Ellwanger mistakenly uses 2002 data for reading, which do not belong to the Klein regime. In addition, her data for the fourth-grade math scores are just plain wrong. I have the NAEP report in front of me (it is also online and anyone can check: google NAEP TUDA 2007 mathematics, p. 62).



From 2003 to 2007, these were the changes in the gaps, as reported by NAEP:

*On fourth grade reading, the black-white gap narrowed by four points, from 30 points to 26 points.

*On fourth grade reading, the Hispanic-white gap increased by two points, from 26 points to 28 points.

*On eighth grade reading, the black-white gap increased by five points, from 25 points to 30 points.

*On eighth grade reading, the Hispanic-white gap increased by six points, from 23 points to 29 points.

*On fourth grade math, the black-white gap decreased by 3 points, from 25 to 22 points.

*On fourth grade math, the Hispanic-white gap decreased by 6 points, from 24 to 18 points.

*On eighth grade math, the black-white gap decreased by six points, form 36 to 30 points.

*On eighth grade math, the Hispanic-white gap decreased by 3 points, from 29 to 26 points.



So, yes, there were some small improvements up and down, but not the large gains which she erroneously claims. And NAEP does not say that any of these are statistically significant changes.



As for the graduation rate, I do believe that it has been increased by dubious means, including “credit recovery” and discharges. Elissa Gootman wrote an article in the Times on April 11, 2008 (“Lacking Credits, Some Students Learn a Shortcut”) about students who failed a course or never attended a single class, who used “credit recovery” to get credits and graduate. This is social promotion. Regarding the discharge rate, a report is coming out this week showing that it has increased during the past few years, and that more students are now being pushed out. This is another way of pumping up the graduation rate. If social promotion had been ended, surely we would not see three-quarters of the New York City high school graduates who enter CUNY community colleges requiring remediation in basic skills. The rate of remediation may have been higher in the past, but at least no one then claimed to have ended social promotion.



Also, though I did not mention it in my article, the city’s SAT scores were down this past year for the first time since 2003. The city’s SAT reading average is 438, a full 50 points behind the state average of 488. This is appalling when one considers that these are our college-bound, if not college-ready, students.



To the question of independent verification of New York City’s improvement: Dr. Bell-Ellwanger says that New York City won the Broad foundation’s annual prize in 2007; this is true, but it was based on state scores alone. The NAEP scores did not appear until November, several weeks later. I was not impressed that the city won a prize based on state scores, since the state scores are as inflated as the city’s. Bear in mind that the state has been reporting dramatic improvements in test scores for the past several years, yet the state scores on NAEP were as flat from 2003-2007 as the city’s.



The Brookings report (on which I was an advisor) does not provide independent verification of the city’s claims because it covered the span from 2000 to 2007. That means that three of the seven years were unaffected by mayoral control, and one of those years was 2002-2003, the year of sharp gains preceding the Klein reforms.



The report “Closing the Graduation Gap” by Christopher B. Swanson for Editorial Projects in Education shows that New York City had a graduation rate of 50.5 percent in 2005. This placed us at #33 of 50 cities, behind the District of Columbia (57.5 percent) and even Chicago (51 percent). The report showed that NYC had gained by 12.8 percent from 1995 to 2005, well behind the much-maligned city of Philadelphia, which posted a gain of 23.2 percent during the same period.



Every expert on graduation rates acknowledges that the problem of tallying the rate is a huge mess because different districts use different methodology. Any report at the national level must rely on those at the local level to give them the data, and the data must be consistent from city to city. I am hopeful that Secretary Duncan will follow through and require all states and cities to produce consistent data, using the same metrics from district to district.



But so long as the city tolerates the misuse of credit recovery (social promotion), and so long as it continues to discharge students in the ninth and tenth grades who have not moved out of the district and are legally entitled to stay in school, then the graduation rate figures are meaningless.



I know this is a vain hope, but I wish that once in a while the Department of Education would admit error or at least show some humility. It might make them human.



Diane Ravitch