Sunday, November 6, 2011

When testing only ACTs like it's measuring student achievement


During last week's AuthorSpeak 2011 sponsored by Solution Tree, one of the lunch panel sessions included Dr. Thomas Guskey, renowned guru on educational assessment. I've attended sessions in the past that featured Dr. Guskey but I guess I never really heard him make a case against using the ACT (and SAT) in NCLB-forced high school assessments. As he claims, the assessments used by states such as Michigan are designed to measure academic achievement against state-adopted curriculum standards, however, the ACT and other tests like it are intended solely to compare and select potential college applicants. These two purposes are at odds with each other according to Dr. Guskey.

In the current issue of ASCD's Educational Leadership, Dr. Guskey summarizes his main objection to using college entrance exams as academic achievement measures:

Assessments used for selection purposes, such as college entrance examinations like the ACT and SAT, are designed to be instructionally insensitive (Popham, 2007). That is, if a particular concept is taught well and, as a result, most students answer an assessment item related to that concept correctly, it no longer discriminates among students and is therefore eliminated from the assessment. These type of assessments maximize differences among students, thus facilitating the selection process.

Guskey, Thomas R. Five Obstacles to Grading Reform, Educational Leadership. ASCD Nov 2011 pp. 17-21 

Naturally, I was more than curious to learn what else Dr. Guskey might have to say about college entrance exams as the basis for state testing. In a web search, I came across an interview for the Lexington Herald-Leader presented on March 16, 2008:

...College entrance exams such as the ACT and SAT help colleges and universities decide whom to admit [but they] do not reflect any particular level of knowledge... rather where each student ranks in relation to others. Ranking makes the selection process easier.

Problems arise when a test designed for one purpose is used for another. ... tests like the ACT and SAT are labeled "instructionally insensitive." If instruction helps most students answer a question correctly, then that question is removed from the test, for it no longer serves its purpose. Even if the question asks about a vitally important concept, it no longer differentiates students and is eliminated.

This is why scores on selection tests are more strongly related to social and economic factors than are scores on competence tests. Aspects other than those influenced by instruction often account for the differences among students. It is also why it makes little sense to use a selection test like the ACT or SAT as a measure of the quality of instructional programs. Doing so would be analogous to using a ruler to measure a person's weight.

Having all students take a selection test such as the ACT or SAT may help some realize that they rank high enough to get into a college or university. That would be a good thing, especially for non-traditional students and those who come from disadvantaged backgrounds.

But to use the results of an "instructionally insensitive" selection test to assess the quality of instructional programs is educational sacrilege. No testing expert would agree to it -- and neither should any legislator or policy-maker.

So I guess my question is rather obvious: Given the argument presented by one of America's most noted researchers on educational assessments, why have states such as Michigan simply ignored the fact that using the ACT as a significant factor in ranking schools and students in academic achievement is akin to assessment malpractice?

Or does it even matter what makes sense when political agendas take over the reins of public education?