Here's a graphic from an article in the New York Times (login to a free account is required):
This is interesting to me. How do you measure and assign a value to "student characteristics"? Does a poor black kid living with his grandmother who makes him do his homework and be respectful to his elders count higher or lower than a kid who has two professional parents who will always side with their son because he, like them, is entitled to the best in life? How do you measure "true total school effect"? What is the "student error term"? Subscripts, superscripts, Greek letters as variables and summations. I don't buy it.
Here's how the NFL rates passers (per Wikipedia):
where mm(x) = max(0,min(x,2.375))
It's not the subscripts, superscripts, Greek letters, and mathematical operations that make these two formulas fundamentally different, it's the nature of the "data" and the interrelationships between the values that make one formula comprehensible and the other not for me.
In a 16 game regular season, an NFL quarterback may have a few passes where there is a difference of opinion as to whether it was caught or incomplete or intercepted. A few touchdowns may be called back because of a penalty or some other factor. But, by and large, there is wide acceptance to what it means to complete a pass, to attempt a pass, to intercept a pass, or to score a touch down. There is an accepted method for measuring the distance gained when a pass is completed. The commisioner, owners, players, coaches, referees, announcers, and fans share in this body of knowledge and pretty much accept the outcome when the final decision comes down.
Teaching and learning is not this calculable. What works for one student may not work for another. Not all kids have the same resources at home or at school. Not all teachers have the same level of support from their principals. Not all principals have access to a supportive central office or others who can help improve learning in their school. Not all PTOs can raise the same dollar amounts. Not all school libraries are powerful centers for meaning making.
Another thing that I find amazing in all of this is the notion of winning or losing a game regardless of how well the quarterback rated. Having a perfect quarterback rating for a game does not guarantee victory, which is what really counts in the end. Just ask Chad Pennington and Bobby Hebert - they both had perfect games according to the formula but walked away losers. And, the highest rated quarterback doesn't always make it to the playoffs. How will this play out for our teacher rating formulas?
What would you do if you could invent the teacher ranking system?
And the US Airforce Academy did an extremely valuable experiment in some of their classes (physics?) where they randomly assigned studednts to professors, for years, so the students couldn't pick and choose the way i used to do. They found, very consistently,t hat professors who got the highest evaluations were the ones who were able to earn the highest grades for their students -- grades largely being assigned by department-wide final exams and tests and so on -- but those high-ranking professors by those short-term measures were, quite surprisingly to many folks -- the absolute worst in promoting long-term understanding. And yes, this LTU was measured!!!
So what do you want? Long-term understanding and poor grades and low student evalua5tions, or high grades and good testr scores and positive s5tudent evaluaions, coupled with a serious lack of thoughtful insight and long-5term understanding of the basic fundamental concepts of the discipline?