For months and months someone advising Senator Del Marsh about his RAISE/PREP Act insisted that using student test scores to evaluate teacher effectiveness is a grand idea.  Usually referred to as VAM, research has shown time after time that this methodology is suspect at best.  Some education researchers refer to it as “junk science.”

As time went along in all the back and forth about the legislation, VAM stuck in the craw of many educators who bothered to look at research.  But RAISE/PREP proponents (most of whom had very limited education experience) were dug in and unyielding.  The result was Senator Marsh finally announcing that he would not bring the bill to the Senate floor this session.

Here is info from yet another research project showing that VAM is unreliable.

The National Center for Education Evaluation and Regional Assistance (NCEE) conducts unbiased large-scale evaluations of education programs and practices supported by federal funds; provides research-based technical assistance to educators and policymakers; and supports the synthesis and the widespread dissemination of the results of research and evaluation throughout the United States.

This NCEE Technical Methods paper addresses likely error rates for measuring teacher and school performance in the upper elementary grades using student test score gain data and value-added models.

Type I and II error rates for comparing a teacher’s performance to the average are likely to be about 25 percent with three years of data and 35 percent with one year of data. 

This means that in a typical performance measurement system, 1 in 4 teachers who are truly average in performance will be erroneously identified for special treatment, and 1 in 4 teachers who differ from average performance by 3 to 4 months of student learning will be overlooked.

These results strongly support the notion that policymakers must carefully consider system error rates in designing and implementing teacher performance measurement systems based on value-added models, especially when using these estimates to make high-stakes decisions regarding teachers (such as tenure and firing decisions).

The moral of this story.  The next time Senator Marsh tells his staff to write a cookbook, he should ask if any of them know how to cook.