The Loneliness of the Long-Distance Test Scorer

The Loneliness of the Long-Distance Test Scorer


Dan DiMaggio


Monthy Review


December 2010


http://www.monthlyreview.org/101201dimaggio.php





Standardized testing has become central to education


policy in the United States. After dramatically


expanding in the wake of the No Child Left Behind Act,


testing has been further enshrined by the Obama


administration's $3.4 billion "Race to the Top" grants.


Given the ongoing debate over these policies, it might


be useful to hear about the experiences of a hidden


sector of the education workforce: those of us who make


our living scoring these tests. Our viewpoint is


instructive, as it reveals the many contradictions and


absurdities built into a test-scoring system run by for-


profit companies and beholden to school administrators


and government officials with a stake in producing


inflated numbers. Our experiences also provide insight


into how the testing mania is stunting the development


of millions of young minds.





I recently spent four months working for two test-


scoring companies, scoring tens of thousands of papers,


while routinely clocking up to seventy hours a week.


This was my third straight year doing this job. While


the reality of life as a test scorer has recently been


chronicled by Todd Farley in his book Making the Grades:


My Misadventures in the Standardized Testing Industry, a


scathing insider's account of his fourteen years in the


industry, I want to tell my story to affirm that


Farley's indictment is rooted in experiences common


throughout the test-scoring world.1





"Wait, someone scores standardized tests? I thought


those were all done by machines." This is usually the


first response I get when I tell people I've been eking


out a living as a test-scoring temp. The companies


responsible for scoring standardized tests have not yet


figured out a way to electronically process the varied


handwriting and creative flourishes of millions of third


to twelfth graders. Nor, to my knowledge, have they


begun to outsource this work to India. Instead, every


year, the written-response portions of innumerable


standardized tests given across the country are scored


by human beings-tens of thousands of us, a veritable


army of temporary workers.





I often wonder who students (or teachers and parents,


for that matter) picture scoring their papers. When I


was a student, I envisioned my tests being graded by


qualified teachers in another part of the country, who


taught the grade level and subject corresponding to the


tests. This idea, it turns out, is as much a fantasy as


imagining all the tests are being scored by machines.





Test scoring is a huge business, dominated by a few


multinational corporations, which arrange the work in


order to extract maximum profit. I was shocked when I


found out that Pearson, the first company I worked for,


also owned the Financial Times, The Economist, Penguin


Books, and leading textbook publisher Prentice Hall. The


CEO of Pearson, Marjorie Scardino, ranked seventeenth on


the Forbes list of the one hundred most powerful women


in the world in 2007.





Test-scoring companies make their money by hiring a


temporary workforce each spring, people willing to work


for low wages (generally $11 to $13 an hour), no


benefits, and no hope of long-term employment-not


exactly the most attractive conditions for trained and


licensed educators. So all it takes to become a test


scorer is a bachelor's degree, a lack of a steady job,


and a willingness to throw independent thinking out the


window and follow the absurd and ever-changing


guidelines set by the test-scoring companies. Some of us


scorers are retired teachers, but most are former office


workers, former security guards, or former holders of


any of the diverse array of jobs previously done by the


currently unemployed. When I began working in test


scoring three years ago, my first "team leader" was


qualified to supervise, not because of his credentials


in the field of education, but because he had been a


low-level manager at a local Target.





In the test-scoring centers in which I have worked,


located in downtown St. Paul and a Minneapolis suburb,


the workforce has been overwhelmingly white-upwards of


90 percent. Meanwhile, in many of the school districts


for which these scores matter the most-where officials


will determine whether schools will be shut down, or


kids will be held back, or teachers fired-the vast


majority are students of color. As of 2005, 80 percent


of students in the nation's twenty largest school


districts were youth of color. The idea that these


cultural barriers do not matter, since we are supposed


to be grading all students by the same standard, seems


far-fetched, to say the least. Perhaps it would be


better to outsource the jobs to India, where the


cultural gap might, in some ways, be smaller.





Many test scorers have been doing this job for years-


sometimes a decade or more. Yet these are the ultimate


in temporary, seasonal jobs. The Human Resources people


who interview and hire you are temps, as are most of the


supervisors. In one test-scoring center, even the office


space and computers were leased temporarily. Whenever I


complained about these things, some coworker would


inevitably say, "Hey, it beats working at Subway or


McDonald's."





True, but does it inspire confidence to know that, for


the people scoring the tests at the center of this


nation's education policy, the alternative is working in


fast food? Or to know that, because of our low wages and


lack of benefits, many test scorers have to work two


jobs-delivering newspapers in the morning, hustling off


to cashier or waitress at night, or, if you're me (and


plenty of others like me) heading home to start a second


shift of test scoring for another company?





Company communications with test-scoring employees often


feel like they have been lifted from a Kafka novel.


Scorers working from home almost never talk to an actual


human being. Pearson sends all its communications to


home scorers via e-mail, now supplemented by automated


phone calls telling you to check your inbox. After the


start of a project, even these e-mails cease, and


scorers are forced to check the project homepage on


their own initiative to find out any important changes.


Remarkably, for a company entrusted with assessing


students' educational performance, messages from Pearson


contain a disturbing number of misspellings, incorrect


dates, typos, and missing information. Pearson's online


video orientation, for example, warns scorers that they


may face "civil lawshits" from sexual harassment. Error-


free communications are rare. I was considering whether


this was a fair assessment, when I received a message


from Pearson with the subject "Pearson Fall 2010." The


link in the e-mail took me to a survey to find out my


availability-for the spring of 2011.





Communications at scoring centers are hardly better. For


example, test-scoring jobs never have a guaranteed end


date. If you ask a supervisor when a job is going to be


completed, you will get a puzzling response that "we


don't know how many papers are in the system, so we


can't say when we'll be done." This response persists,


even though it's pretty easy to calculate how many


fifth-graders there are in Pennsylvania and how long it


will take to grade their papers, given our scoring rate.


If we are lucky, we get twenty-four-hours notice before


being told that a project is about to end and we should


seek other work. Two hours notice is more common. In


general, scorers are given no information beyond what is


absolutely necessary to do the job.





What is the work itself like? In test-scoring centers,


dozens of scorers sit in rows, staring at computer


screens where students' papers appear (after the papers


have undergone some mysterious scanning process). I


imagine that most students think their papers are being


graded as if they are the most important thing in the


world. Yet every day, each scorer is expected to read


hundreds of papers. So for all the months of preparation


and the dozens of hours of class time spent writing


practice essays, a student's writing probably will be


processed and scored in about a minute.





Scoring is particularly rushed when scorers are paid by


piece-rate, as is the case when you are scoring from


home, where a growing part of the industry's work is


done. At 30 to 70 cents per paper, depending on the


test, the incentive, especially for a home worker, is to


score as quickly as possible in order to earn any money:


at 30 cents per paper, you have to score forty papers an


hour to make $12 an hour, and test scoring requires a


lot of mental breaks. Presumably, the score-from-home


model is more profitable for testing companies than


setting up an office, especially since it avoids the


prospect of overtime pay, the bane of existence for


companies operating on tight deadlines. But overtime pay


is a gift from heaven for impoverished test scorers; on


one project, I worked in an office for twenty-three days


straight, including numerous nine-hour days operating on


four to five hours sleep-such was my excitement about


overtime.





Yet scoring from home also brings with it an entirely


new level of alienation. You may work on a month-long


project without ever speaking to another human being,


never mind seeing the children who actually wrote the


papers. If you do speak to another person, it's at your


own expense, since calling the supervisors at the test-


scoring center takes time, and might cut into the


precious moments you spend scoring (especially when you


have to wait fifteen minutes for someone to answer, as


happens routinely on some projects).





The piece-rate system also leads to some sinister math;


I have often wondered how much money I lose for every


trip to the bathroom, and debated taking my laptop there


with me. And since you are only guaranteed employment


until the papers run out, you are in a race against all


your phantom coworkers to score as many papers as you


can, as fast as possible. This cannot be good for


quality, but as long as the statistics match up and the


project finishes on time, the companies are happy. I did


receive some automated warnings from Pearson that I was


scoring too fast, while simultaneously receiving


messages on the Pearson website to the effect that,


"We're way behind! Log in as many hours as you can and


score as much as possible!"





No matter at what pace scorers work, however, tests are


not always scored with the utmost attentiveness. The


work is mind numbing, so scorers have to invent ways to


entertain themselves. The most common method seems to be


staring blankly at the wall or into space for minutes at


a time. But at work this year, I discovered that no one


would notice if I just read news articles while scoring


tests. So every night, while scoring from home, I would


surf the Internet and cut and paste loads of articles-


reports on Indian Maoists, scientific speculation on


whether animals can be gay, critiques of standardized


testing-into what typically came to be an eighty-page,


single-spaced Word document. Then I would print it out


and read it the next day while I was working at the


scoring center. This was the only way to avoid going


insane. I still managed to score at the average rate for


the room and perform according to "quality" standards.


While scoring from home, I routinely carry on three or


four intense conversations on Gchat. This is the reality


of test scoring.





There is a common fantasy that test scorers have some


control over the grades they are giving. I laugh


whenever someone tells me, "Make sure you go easy and


give the kids good grades!" We are entirely beholden to


and constrained by the standards set by the states and


(supposedly) enforced by the test-scoring companies. To


ensure that test scorers are administering the "correct"


score, we receive several hours of training per test,


and are monitored through varying quality control


measures, such as random "validity" papers that are pre-


scored and that we must score correctly. This all seems


logical and necessary to ensure impartiality-these are,


after all, "standardized" tests. Unfortunately, after


scoring tests for at least five states over the past


three years, the only truly standardized elements I have


found are a mystifying training process, supervisors who


are often more confused than the scorers themselves, and


a pervasive inability of these tests to foster


creativity and competent writing.





Scorers often emerge from training more confused than


when they started. Usually, within a day or two, when


the scores we are giving are inevitably too low (as we


attempt to follow the standards laid out in training),


we are told to start giving higher scores, or, in the


enigmatic language of scoring directors, to "learn to


see more papers as a 4." For some mysterious reason,


unbeknownst to test scorers, the scores we are giving


are supposed to closely match those given in previous


years. So if 40 percent of papers received 3s the


previous year (on a scale of 1 to 6), then a similar


percentage should receive 3s this year. Lest you think


this is an isolated experience, Farley cites similar


stories from his fourteen-year test-scoring career in


his book, reporting instances where project managers


announced that scoring would have to be changed because


"our numbers don't match up with what the


psychometricians [the stats people] predicted." Farley


reports the disbelief of one employee that the stats


people "know what the scores will be without reading the


essays."2





I also question how these scores can possibly measure


whether students or schools are improving. Are we just


trying to match the scores from last year, or are we


part of an elaborate game of "juking the stats," as it's


called on HBO's The Wire, when agents alter statistics


to please superiors? For these companies, the ultimate


goal is to present acceptable numbers to the state


education departments as quickly as possible, beating


their deadlines (there are, we are told, $1 million


fines if they miss a deadline). Proving their


reliability so they will continue to get more contracts.





As Farley writes, "Too often in my career the test


results we returned had to be viewed not as exemplars of


educational progress, but rather as numbers produced in


a mad rush to get things done, statistics best viewed


solely through the prism of profit."3 It seems to me


that what the companies would tell us, if they were


honest, would be something like, "Hey guys, your scoring


doesn't really matter. We just want to give the same


scores as last year, so that there's no controversy with


the state and we get more contracts and make more


profits-so no matter what you learned in training, just


try to forget it." States and local governments,


meanwhile, play their own version of this game, because


it looks good for them when politicians can claim that


test scores are going up. Witness the recent controversy


in New York City, where the percentage of students


passing the math exam rose from 57 percent in 2006 to 82


percent in 2009, before plummeting back down to 54


percent in 2010 (along with a 43 percent passing rate in


English) after the standards were reviewed.4





As test scorers, we never know what the numbers we are


assigning to papers mean, or where we fit in this


elaborate game. We are only responsible for assigning


one score, on one small part of a test, and we do not


even know whether the score we assign is passing or


failing-that information is never divulged in training.


We never hear how the students fared. Whether Marissa


will be prevented from going to seventh grade with her


friends because one of us, before our first cup of


coffee kicked in, decided that her paper was "a little


more like a 3 than a 4," we will never know. Whether


Marissa's school will be closed or her teachers fired


(to be reborn as test scorers next spring?) remain


mysteries to the test scorers. And yet these scores can


be of life-and-death importance, as seen in the recent


suicide of beloved Los Angeles middle school teacher


Rigoberto Ruelas, Jr. Upon learning that he ranked as


"less effective" on the LA Times teacher performance


rating scale-based solely on test scores-Ruelas took his


own life.5





Even if the scoring were a more exact science, this


would in no way make up for the atrocious effect on


creativity wrought by the mania for standardized


testing. This impact has now been documented. According


to one study, creativity among U.S. children has been in


decline since 1990, with a particularly severe drop


among those currently between kindergarten and sixth


grade.6





While test scorers and students might be separated by


age, geography, race, and culture, we share one bond:


standardized testing puts us to sleep. In the face of


the crushing monotony of the hundreds of rote responses


fostered by these tests, scorers are left to fight their


own individual battles to stay awake. In any test-


scoring center, by far the most essential job is done by


the person whose sole responsibility consists of making


coffee for hundreds of workers, many of whom will


consume four to six cups a day to survive. In my mind, I


see a hideous symmetry between test scorers' desperate


attempts to avoid dozing off, and the sleepy, zombie-


like faces of the students as they prepare for and take


these tests.





Of course, these students only exist in my imagination.


Just as test scorers are never allowed to know the


effects of our scores on students, we never get a chance


to meet them, to see how they have developed as writers,


thinkers, or human beings, or to know what life in their


communities or families is like. All we see is a paper


on a screen. And after reading hundreds of monotonous


papers each day, it's not uncommon to start to feel a


bitter distaste for the undoubtedly beautiful youth of


America and the seeming poverty of their creative


thought.





I remember reading, for twenty-three straight days, the


responses of thousands of middle-schoolers to the


question, "What is a goal of yours in life?" A plurality


devoted several paragraphs to explain that their life's


goal was to talk less in class, listen to their teacher,


and stop fooling around so much. It's asking too much to


hope for great literature on a standardized test. But,


given that this is the process through which so many


students are learning to write and to think, one would


hope for more. These rote responses, in themselves, are


a testament to the failure of our education system, its


failure to actually connect with kids' lives, to help


them develop their humanity and their critical thinking


skills, to do more than discipline them and prepare them


to be obedient workers-or troops.





While we test scorers might be prone to blame these


children for the monotony of their thoughts, it's not


their fault that their imaginations and inspirations are


being sucked out of them. No points are given for


creativity on these tests, although some scorers have


told me that, until recently, a number of states did


factor creativity into their scores. Ironically, scorers


are often delighted to see papers that show


individuality and speak in their own voice, and often


reward them with higher scores, though, judging by the


papers I've read, it appears as if students often


explicitly are told not to be creative. Yet even if


creativity were considered, it would not likely do much


to change the overall character of the writing-and


education-engendered by an emphasis on standardized


testing. As Einstein put it, "It is a miracle that


curiosity survives formal education."





An entire education policy that thrives on repetition,


monotony, and discipline is being enacted, stunting


creativity and curiosity under the guise of the false


idol of accountability. What is more, this policy has a


differential impact, depending on students' race and


class. As Jonathan Kozol explains,





In most suburban schools, teachers know their kids are


going to pass the required tests anyway-so No Child Left


Behind is an irritant in a good school system, but it


doesn't distort the curriculum. It doesn't transform the


nature of the school day. But in inner-city schools,


testing anxiety not only consumes about a third of the


year, but it also requires every minute of the school


day in many of these inner-city schools to be directed


to a specifically stated test-related skill. Very little


art is allowed into these classrooms. Little social


studies, really none of the humanities.7





Seeing the results of this process is demoralizing to


test scorers, and you can feel it in the scoring


centers. Even though you can move about freely, use the


bathroom when you need, and talk to one another, the


room I was in this spring was almost always completely


silent. On every project, as the weeks go by, the health


of many scorers deteriorates, making me curious as to


whether the relentless, soul-crushing monotony of the


papers has an actual physical impact on those forced to


read them.





To be fair, these papers aren't a total wash. There is


often wisdom in them, even on standardized tests. The


chasm between rich and poor is at times felt in the


writing itself, as some students come from unimaginable


privilege, while many more endure heartbreaking


experiences in foster homes. The papers are also a


testament to the persistence of racism, describing


teenagers kicked out of stores or denied service or jobs


because of the color of their skin. And it would be


wrong to think of test scorers as a down-and-out bunch-


many of us do this job in order to avoid having to get


other ones that would keep us from our creative


endeavors, or from traveling or pursuing other life-


enriching possibilities. A number of test scorers I've


met over the past three years are authors, artists,


photographers, or independent scholars, and it's common


to see postings for book releases and other events


featuring the work of test scorers on bulletin boards in


the break room.





In the error-filled Pearson training video, Marjorie


Scardino says, "Most of the people who work at Pearson


work with a passion and an intensity, because they think


know are doing something important." But I've never


gotten the sense from my coworkers that they "think


know" what they're doing is helping kids or the


education process. If the Obama administration asked


test scorers whether the solution to this country's


education system would be more standardized testing, I


think most of them would laugh. I've never gotten the


sense from my coworkers that they feel that what they're


doing is helping kids or the education process.


Unfortunately, the joke is on us, as the Obama


administration pushes for even more high-stakes


standardized testing. I didn't know whether to laugh or


cry back in April, when all workers at my test-scoring


center were asked to fill out a form allowing the


company we were working for to get a tax break for


hiring us. This tax break came via the Obama


administration's HIRE Act, which was supposed to provide


subsidies for companies "creating jobs." Never mind that


we were all going to be hired anyway, because this is


seasonal employment. Or that this money was subsidizing


temporary jobs with no health care and no hope for


transitioning into long-term employment-jobs which, in a


better world, would not exist.





While these companies brazenly collect what can only be


described as corporate welfare checks, hundreds of


thousands of teachers are being laid off, as governments


cut funding to education. Maybe next year, some of them


will get paid $12 an hour (or $10, if they flood the


market) to score tests taken by students stuffed into


even bigger classes, and help "impartially" decide which


schools will be shut down, and which of their colleagues


will be laid off. Equally bad, the fanaticism


surrounding accountability via testing, which claims it


will result in higher-quality teachers, is doing nothing


of the sort. Referring to the test-intensive No Child


Left Behind Act, Kozol says, "By measuring the success


of teachers almost exclusively by the test scores of


their pupils, it has rewarded the most robotic teachers,


and it's driving out precisely those contagiously


exciting teachers who are capable of critical thinking


who urban districts have tried so hard to recruit."8





As a friend of mine was saying his goodbyes to the


coworkers in his room at the end of this year's scoring


season, his seventy-year-old supervisor, a veteran test-


scoring warrior, uttered the words I imagine many test


scorers hope to hear: "I hope I never see you here


again." This is a measure of the cynicism with which


many test scorers approach the industry, recognizing


that it is fundamentally a game, which too many people


are forced to play-but "hey, it beats working at


McDonald's or Subway!" Yet amid all the hopes of


escaping the industry, these test-scoring companies are


successfully expanding and are now hoping to get their


hands on billions in "school turnaround" money handed


out by the Obama administration and state governments.


Pearson, for example, has "formed the K-12 Solutions


Group, and.is seeking school-turnaround contracts in at


least eight states.[claiming it] could draw on its


testing, technology and other products to carry out a


coherent school-improvement effort."9





The big test-scoring companies will undoubtedly be


called on to furnish their supposed "expertise" in


developing and scoring the new generation of more


complex tests envisioned by Secretary of Education Arne


Duncan. The Obama administration just gave two groups of


states $330 million in grants to develop these new


national tests, with the stated aim of assessing more


critical thinking skills and providing better feedback


to students and teachers. But rather than addressing the


problems outlined above, it seems more likely that this


move will only transfer the absurdities in current state


tests to a national level, with the danger that they


will take on an even greater legitimacy. In fact, given


that Duncan's proposal involves even more tests, it is


likely to make matters worse.





If scoring is any indication, everyone should be worried


about the logic of putting more of our education system


in the hands of these for-profit companies, which would


love to grow even deeper roots for the commodification


of students' minds. Why would people in their right


minds want to leave educational assessment in the hands


of poorly trained, overworked, low-paid temps, working


for companies interested only in cranking out acceptable


numbers and improving their bottom line? Though the odds


might seem slim, our collective goal, as students,


teachers, parents-and even test scorers-should be to


liberate education from this farcical numbers game.





Notes





   1.  Todd Farley, Making the Grades: My Misadventures


   in the Standardized Testing Industry (San Francisco:


   Polipoint, 2009).





   2.  Todd Farley, "A Test Scorer's Lament," Rethinking


   Schools (Winter 2008/2009).





   3.  Todd Farley, "Standardized Tests Are Not the


   Answer: I Know, I Graded Them," Christian Science


   Monitor, October 28, 2009.





   4.  Sharon Otterman, "Confusion on Where City


   Students Stand," New York Times, August 28, 2010.





   5.  Alexandra Zavis and Tony Barboza, "Teacher's


   suicide shocks school," Los Angeles Times, September


   28, 2010, http://articles.latimes.com.





   6.  Po Bronson and Ashley Merryman, "The Creativity


   Crisis," Newsweek, August 10, 2010.





   7.  Matthew Fishbane, "Teachers: Be subversive


   (Interview with Jonathan Kozol)," Salon.com, August


   30, 2007.





   8.  Ibid.





   9.  Sam Dillon, "Inexperienced Companies Chase School


   Reform Funds," New York Times, August 9, 2010.





___________________________________________





Portside aims to provide material of interest to people


on the left that will help them to interpret the world


and to change it.





Submit via email: portside@portside.org





Submit via the Web: http://portside.org/submittous3