Distorting Data: Diane Ravitch on the DOE Spin

Excerpt from post by Diane Ravitch in dialogue with Deborah Meier at Bridging Differences (Education Week)

So the small schools graduation rate is impressive, right? Wrong. Dee Alpert, who publishes SpecialEducationMuckraker.com is a relentless devourer of data from the city and state education departments, and she sent out a bulletin describing her inability to decipher the real graduation rates at the "new small schools." She says:

1. NYCDOE gives no list of the 'new, small schools' included in its calculations for the spinrelease, so...
2. They give percentages, not raw numbers, for their graduation rates: you can't even try to work backwards to see what was included.
3. Most importantly, they gave no numbers for 'still enrolled,' nor for 'discharged.' ...NYCDOE is notorious for mis-reporting dropouts as having enrolled elsewhere, i.e., discharged from a NYCDOE school's rolls.
4. The numbers for graduates who earned local v. Regents diplomas is also critical, and missing. In prior years, local hs diplomas predominated. According to the NYS Court of Appeals' CFE [Campaign for Fiscal Equity] decision, a local diploma resulting from passing RCT [low-level competency examinations] put a kid-depending on the subject—at between the 6th and 9th grade level. This isn't exactly college prep.

Dee Alpert concludes: "It's all obviously part of the Bloomberg presidential campaign spin." And she strongly recommends "an editorial moratorium on reporting NYCDOE spin numbers unless the complete data set accompanies the press release."

I hope not to befuddle our national readership with too much news and talk about New York City, but there is an important point here. Alpert's analysis reveals how easily education data are distorted for political purposes. We have seen this done before, but seldom so blatantly or so cleverly. It is likely the case that every school superintendent wants to release test scores and graduation rates that show what a success he or she has been. But what we have seen over the past five years is a determined political campaign—not just Bloomberg for President, although that may yet happen—but rather a political campaign to "prove" that mayoral control without any checks or balances is the absolute best way to manage a school system. The Department is incapable of impartial research. Its press releases are filled with the kind of P.R. spin that we have come to associate with politicians running for office, not with research departments where someone has his or her professional reputation on the line.

Whenever there is a new release of test score data from the state, I invariably get calls from reporters, asking what I think of the latest numbers. I have learned over the past few years never to answer their questions until I have had a chance to see the complete data set with my own eyes. I know that the Department has massaged the data and sought out every glimmer of good news while creating a narrative that distracts the reporters' attention from anything unfavorable.

The full interchange below:

Historical amnesia

Dear Diane,

I was talking with a young man yesterday who is working at a new NYC high schools for students who have dropped out or are about to. He's very enthusiastic about the work and the school. He thinks Bloomberg and Company invented such schools, and that his is the first.

Historical amnesia is, alas, widespread. In a piece on Bloomberg’s ambitions for the Presidency and another on the High School of International Business and Finance, NY Times reporter Sara Rimer suggests that Bloomberg/Klein are the first to worry about how to educate the kids at the bottom, the first to develop small schools, the first to be enamored of test score accountability, the first to eliminate “social promotion” and so on. How does she explain that half of NYC's kids were starting high school at least one year over-age in the late 80's. Shouldn’t a fact-checker catch such historical untruths? My young friend’s ignorance is forgivable, the NY Times reporter’s is less so.

Small, personalized schools for kids who are floundering, or for kids in general, is not a new idea. It was thriving 30 years go. Steve Phillips, now teaching at Brooklyn College, ran a district that served thousands of NYC’s most needy kids in small, personalized and relatively successful schools. He operated a "system" of schools for twenty years that was threatened by each new Chancellor, and disappeared finally under Bloomberg/Klein. With it NYC’s school history was rewritten. Phillips' "domain" eventually included many small schools that weren't designed just for dropouts—like CPESS, Vanguard, Landmark, International etc. By the time his work was dismantled, Phillips had created a system that was larger than Boston’s. How sad that this new recruit to NYC’s schools thinks his school has no history to draw upon.

Amnesia is dangerous, and that's why I agree with you, Diane, on the importance of history. But "teaching" history and getting in the habit of wondering aloud about its uses and abuses do not always go together: The reason you had “no problem” deciding what to include in your California curriculum proposal was that you included everything. (Leaving, as Sizer used to point out, the hard question of what to leave out to classroom teachers.) Getting into the habit of asking questions such as "has this ever been tried before?" "what happened?" "is there a pattern here?" “based on what evidence?” and “does it matter?” is what takes time and patience to work into the classroom’s life where decades and centuries whiz by so fast. Thoughtful questioning starts in kindergarten and is built on every year thereafter if—mind you, “if” that’s what we want young people to carry away from 13 years of schooling.

It’s not unconnected with what we’d both like to see in the reporters who cover the education beat, who probably aced school history courses but are missing the know-how and perhaps the guts to speak truth to power. I’d love to see a real debate based on Lynnell Hancock’s wonderful Nation piece (subscription required). It deserves some heated words in response as we separate mythology from history and dicover that uncertainty has its place in history—and science—too.

The “new” dogma that teachers (and kids!) will work harder or smarter if they are paid for high test scores also enters the debate as though it has no history, no research on its risks, etc. I'm sometimes chided by proponents of “merit” pay for assuming that good teachers will be motivated by such crass incentives to do things they know are not educationally sound. We can't have it both ways. Either it's a waste of money or an effort to undermine professional norms by punishing anyone who doesn’t fall in line. It’s a simple-minded form of current economic orthodoxy gone wild. Meanwhile our critics blather on about character education—as we systematically undermine what is at the root of character: some courage.

Schools that lack the courage to teach in ways that honor what they believe is most important about the “liberal arts”, and agree to dumb it down to fit into a multiple-choice format are a danger to liberal society. But a Mayor and Chancellor who have done their best to encourage such cowardice are at least equally at fault.

Deb

p.s. The reason I love Mike Rose’s work is that he, like Gerald Graff, is prepared to liberate “the liberal arts” from its narrowed “academic” definition.

Posted by Deborah Meier on June 29, 2007 10:25 AM | Permalink

Distorting data

Dear Deb,

Now that our mayor, Michael Bloomberg, is readying himself as a potential candidate for the Presidency, it is clear that education will be one of his signature issues. Sadly, he knows no more about education today than he did when he became mayor in 2001, based on his latest plan to pay poor kids to get higher test scores. That strategy seems to me to be an abject admission of cluelessness: When you don't know anything about teaching or curriculum, then just pay for results.

I understand your frustration about the historical amnesia that you encountered. It seems to be the policy of our New York City Department of Education to wipe out all historical memory and at that they have been quite successful. Apparently, this administration wants the world to believe that whatever it is doing is historic, unprecedented, and of course dazzlingly successful. Part of their strategy is to launch one initiative after another, to persuade the public that they are on the move, when in fact they are merely lurching from one confused plan to another.

Last Friday, the Department released dazzling statistics about the graduation rates of its new small schools. The Department has created about 200 of them, of which 47 have a graduating class this year. The rate for these 47 schools is 73 percent, compared to a citywide rate of.... This is where it gets complicated. The city says it has a graduation rate of 60 percent. The state says the city's graduation rate is 50%. Education Week's latest Diplomas Count report says the city's graduation rate is 45 percent.

So the small schools graduation rate is impressive, right? Wrong. Dee Alpert, who publishes SpecialEducationMuckraker.com is a relentless devourer of data from the city and state education departments, and she sent out a bulletin describing her inability to decipher the real graduation rates at the "new small schools." She says:

1. NYCDOE gives no list of the 'new, small schools' included in its calculations for the spinrelease, so...
2. They give percentages, not raw numbers, for their graduation rates: you can't even try to work backwards to see what was included.
3. Most importantly, they gave no numbers for 'still enrolled,' nor for 'discharged.' ...NYCDOE is notorious for mis-reporting dropouts as having enrolled elsewhere, i.e., discharged from a NYCDOE school's rolls.
4. The numbers for graduates who earned local v. Regents diplomas is also critical, and missing. In prior years, local hs diplomas predominated. According to the NYS Court of Appeals' CFE [Campaign for Fiscal Equity] decision, a local diploma resulting from passing RCT [low-level competency examinations] put a kid-depending on the subject—at between the 6th and 9th grade level. This isn't exactly college prep.

Dee Alpert concludes: "It's all obviously part of the Bloomberg presidential campaign spin." And she strongly recommends "an editorial moratorium on reporting NYCDOE spin numbers unless the complete data set accompanies the press release."

I hope not to befuddle our national readership with too much news and talk about New York City, but there is an important point here. Alpert's analysis reveals how easily education data are distorted for political purposes. We have seen this done before, but seldom so blatantly or so cleverly. It is likely the case that every school superintendent wants to release test scores and graduation rates that show what a success he or she has been. But what we have seen over the past five years is a determined political campaign—not just Bloomberg for President, although that may yet happen—but rather a political campaign to "prove" that mayoral control without any checks or balances is the absolute best way to manage a school system. The Department is incapable of impartial research. Its press releases are filled with the kind of P.R. spin that we have come to associate with politicians running for office, not with research departments where someone has his or her professional reputation on the line.

Whenever there is a new release of test score data from the state, I invariably get calls from reporters, asking what I think of the latest numbers. I have learned over the past few years never to answer their questions until I have had a chance to see the complete data set with my own eyes. I know that the Department has massaged the data and sought out every glimmer of good news while creating a narrative that distracts the reporters' attention from anything unfavorable.

Forgive me, Deborah, but all this media manipulation persuades me that we need national tests (with no stakes), so that states and cities and districts can't play games with the numbers. Failing that, I think that every state should have an independent agency to administer tests and report their results, and that these agencies should be run by professional psychometricians who can neither take credit nor blame when the scores rise or fall. In New York State, when the math scores went down last year, the State Education Department said that the test was harder; when the math scores went up this year, the State Education Department was congratulating itself and the Regents (our state board) for its wise policies. How about an agency that dispenses the facts without fear or favor or self-praise?

Diane

Posted by Diane Ravitch on July 2, 2007 11:56 AM |