Are We Comparing Apples and Oranges?

As I'm reading through Education Week's article on KIPP attrition rates (Vol. 30, No. 27, April 30, 2011) I can't help but ponder the simplistic methods we've latched onto for comparing school achievement. For the most part, schools in each state and across the nation are simply compared by reading and math scores (state, national and international tests) and graduation rates. Occasionally, science scores are thrown in for good measure but are not consistently used in the same manner as reading and math.

The article on KIPP reviews a recent study by researchers at Western Michigan University that concludes while the schools have a great reputation for high achieving minority students, they have an exceptionally high rate of attrition for black males in grades 6-8, over forty percent, than their neighboring schools in the same districts. Therefore, it's difficult to compare whether the KIPP schools are being any more successful overall than their neighbors.

While some folks would read this and cheer, because they merely want to see anything that's not a traditional public school fail, I look at it as more of an indicator of the inherent problems with simplistic approaches to comparing schools and school districts, something that NCLB – the current incarnation of the Elementary and Secondary Education Act of 1965 – has turned into a cottage industry that has done more to enrich testing companies and so-called education reform activists, than it has to improve overall student achievement.

Here are a number of questions I have about the validity of comparing schools with each other:

• Are comparisons valid between two or more schools that receive different levels of funding from either public or private sources? For instance, in the same Education Week article, the researchers found that KIPP schools receive far greater funding than surrounding districts, about $6,500 more per student than the average for the other districts. Even traditional public school districts will differ in their per-pupil revenues and shouldn't that be a factor?

• What about schools that have differing numbers of students who are have validated limited English proficiency skills?

• Can you compare schools in lower income urban settings with those in upper income suburban areas? Recent reports seem to indicate that achievement results on high-stakes tests do correlate well with urban versus suburban settings. And what about rural schools compared to urban or suburban?

• How about schools that differ in the number of minority students. This can be looked at from two different perspectives: schools with higher percentages of students from minority groups that have traditionally struggled in school, and those with higher percentages of minorities from groups that have traditionally excelled in schools. Either way, would the comparisons be skewed?

• And what about schools that are co-located in towns with major universities or multiple institutions of higher learning? Could it be expected that these schools might have a greater percentage of students with parents who have higher levels of post-secondary learning? Is that a factor?

• What about schools in cities or towns with exceptionally low property values compared to more affluent districts or districts in vibrant business communities?

• Would student transiency effect achievement comparisons between schools? What about schools with a higher percentage of students who are their by choice, not by circumstance?

• And lastly (because I can't think of any more right now), what about the difference in demographics of the school staff? What if there are more males than females, or vice versa, on the staff? Or differences in what universities they earned their degrees? Or marital status and family statistics? Or even whether they lived within the school district or commute from many miles away?

My point is that comparing schools – traditional public, charters, or private – is often just an exercise in trying to make an apple look, feel and taste like an orange. If we only compare the two by their general shapes, than we can easily argue whether one is better than the other and we need to work harder to make the lesser one better. But when you peel back the outer surface of both, you'll see that there are many, many variables making valid comparisons difficult if not next to impossible.

But if the real motives for comparing schools are political, it's not likely the question of validity will stop us.