A retired college professor friend has carefully reviewed the A-F school report cards mess and basically called it “worthless.” And this is someone who had minors in both research and statistics when she obtained her doctorate.
Take a look:
In research and statistics, the “conclusions” section of the research report answers questions such as this: “Okay, here are the numbers (or letters). Now what does that mean?” In the case of the A-F report card grades for schools, the assignment of that letter grade is supposed to mean the same thing that is customarily understood: A = Great; B = Good; C = Average; D = Fair; F = Poor. Yes, there are other descriptors that equally describe the A-F letter assignments; however, most people get the fact that D’s and F’s will get a kid in trouble at home, and, they are not something to brag about.
As far as the “implications/limitations” section of the research report is concerned, the researchers answer such questions as these: “Was our research sample of participants too small to be able to generalize the results beyond the sample that was used?” “What would limit the interpretation of the results?” For example, concerning the A-F report card grades for schools, “Was a limitation that this score (A-F) based upon valid and reliable data?” The answer that they would have to present is NO! Why? The A-F grades were based on ONE score; AND, the data collection itself was flawed because the instrument that was used (ACT Aspire) had no aspirations at all – in fact, it was removed by those who put it in place, and deemed to NOT be able to have any validity or reliability.
Put simply, the data collection instrument has to be demonstrated to be valid and reliable. Validity for the data collection instrument (ACT Aspire, for example), has to answer this question (among others): “Will this instrument accurately contain content that will allow us to collect data to answer our research question?” In the case of the A-F report card grades for schools, the research study falls flat with validity. They wanted to know and report how well our schools are doing. However, they used an instrument that had already been deemed to be flawed. And, that was ALL they used.
To demonstrate how ridiculous it is for our State Department of Education to do this, how would students and parents feel about this? An Algebra teacher gives ONE test during a grading period, and that’s the only grade used to determine the student’s report card grade. Well, that’s flawed in itself. Furthermore, the test that was given had been pulled from the textbook company because there were questions in the test that didn’t match the chapters they were supposed to represent (e.g. the content that was studied and taught). Additionally, the textbook company had pulled this test because there were questions that were so confusing, some answers that were supposed to be wrong actually could be right, depending on the alternate (and acceptable) interpretation of the flawed questions.
In determining reliability of the data collection information, one questions is: “Do we get the same realm of scores that we expect to get each time we administer this test?” For example, “Do schools that have traditionally demonstrated excellence in a variety of other ways, also have ‘excellent’ category ratings from the data collection measure we are using? If not, why?” Again, the assignment of the A-F report card grades fall flat.
In summary, if the A-F report card grade scenario were to be used as a university graduate school student project, the student’s committee would recommend a wider selection of data collection instruments. Furthermore, it would most surely disallow the ONE data collection instrument (the ACT Aspire) being used.
As a member of editorial publications in education, I can safely say that if this were to be a research study for publication, it would be rejected. In fact, the only reason it might ever be published is to demonstrate what NOT to do, as it is one of the examples of the worst kind of data collection and reporting one might ever read.
I completely reject these grades, not based on an emotional response due to my feelings of rejection resulting from this school or that school receiving a D or an F. That behavior would be as inaccurate and unprofessional as those who assigned these grades. In fact, due to the flawed nature of how the State Department of Education arrived at the letter grades, I am also unable to confidently celebrate with the schools that received an A.
Granted, some of this is pretty “deep” for a redneck from a south Alabama farm like me. But I can figure out what the professor is saying, which is “Not no–but Hell no.”
And the politicians who schemed this up still refuse to believe the truth.