While it has not been talked about much and never mentioned by name, one of the more insidious parts of the proposed RAISE Act is using something called VAM (Value Added Modeling) to project performance levels of students and pay teachers based on whether their students meet their targets. VAM is extremely controversial. Enough so that its use has been (or is being) litigated in at least seven states.
There are several VAM models. Basically they are designed to answer the question of whether the influence of a particular teacher causes students’ achievement to grow more or less than the “average” or “expected” growth. A white paper by Pearson says, “Overall there is little consensus within the field about which model specification produces the most accurate value-added measure.” It further states. “The research base is currently insufficient to support the use of VAM as the sole basis for high-stakes decisions.”
Dr. Bill Mathis at the National Education Policy Center at the University of Colorado called VAM “basically psychometric junk as a technique” in a recent email.
A New Mexico judge ruled in early December that the state department of education there may not continue using VAM until “a trial on the merits where the count can be assured of the statistical validly of the data collection and reporting that feeds into the system.”
Dr. Audrey Beardsley at Arizona State University is considered by many to be the “guru” when it comes to VAM. Her blog, VAMboozled!, is widely read. After looking over a draft of the proposed RAISE Act, here is what she posted.
Obviously she is not impressed by what is being proposed. And the title of her post, Is Alabama the New, New Mexico? leads me to think that if this bill is passed, we will soon be in court spending money that could buy textbooks.
Here are some more of her comments.
In Alabama, the Grand Old Party (GOP) has put forth a draft bill to be entitled as an act and ultimately called the Rewarding Advancement in Instruction and Student Excellence (RAISE) Act of 2016. The purpose of the act will be to…wait for it…use test scores to grade and pay teachers annual bonuses (i.e., “supplements”) as per their performance. More specifically, the bill is to “provide a procedure for observing and evaluating teachers” to help make “significant differentiation[s] in pay, retention, promotion, dismissals, and other staffing decisions, including transfers, placements, and preferences in the event of reductions in force, [as] primarily [based] on evaluation results.” Related, Alabama districts may no longer use teachers’ “seniority, degrees, or credentials as a basis for determining pay or making the retention, promotion, dismissal, and staffing decisions.” Genius!
“For grade levels and subjects for which student standardized assessment data is not available and for teachers for whom student standardized assessment data is not available, the [state’s] department [of education] shall establish a list of preapproved options for governing boards to utilize to measure student growth.” This is precisely what has gotten the whole state of New Mexico wrapped up in, and currently losing their ongoing lawsuit (see my most recent post on this here). While providing districts with menus of preapproved assessment options might make sense to policymakers, any self respecting researcher or even assessment commoner should know why this is entirely inappropriate. To read more about this, the best research study explaining why doing just this will set any state up for lawsuits comes from Brown University’s John Papay in his highly esteemed and highly cited “Different tests, different answers: The stability of teacher value-added estimates across outcome measures” article. The title of this research article alone should explain enough why simply positioning and offering up such tests in such casual (and quite careless) ways makes way for legal recourse.
Most importantly, though, for those of you who have access to such leaders in Alabama, do send them this post so they might be a bit more proactive, and appropriately more careful and cautious, before going down this poor educational policy path. While I do embrace my professional responsibility as a public scholar to be called to court to testify about all of this when such high-stakes consequences are ultimately, yet inappropriately based upon invalid inferences, I’d much rather be proactive in this regard and save states and states’ taxpayers their time and money, respectively.