The gooder great

Sometimes I read the news. I read it even though I know it is one of the major sources of my anger. Apparently, knowing stuff bothers me but I can't break myself of the habit. I read the news today.

Oh boy.

It seems that Newsweek has released its list fo the top 500 high schools in the country. Hey, i figured, I work in a high school. maybe I'll know some of the words in that article! So off I went.

First, I read a summary and then I clicked through to the actual full rankings and may i just say...what a bunch of crap. I'll walk you through the stupidity and you can judge for yourself, as long as your judgment confirms mine.

Step one: Which schools? According to Newsweek, "NEWSWEEK reached out to administrators, principals, guidance counselors, and Advanced Placement/International Baccalaureate coordinators at more than 10,000 public high schools across the country"

OK, let's start the problems.

To be good one must be a public school? Apparently this article, though it is called "America's Best High Schools" is "America's Best PUBLIC High Schools". Now I'm not complaining about public schools, but about misleading headlines. The article is not judging all high schools so the headline should not pretend that what is being decided is from the larger group.

Step two: How did you collect your data? According to Newsweek, this data was self reported by the schools. So lying, while not encouraged, is certainly possible. Of course, no one would fudge numbers in an attempt to be listed as a top school...right?

Step three: What data did you collect?

As reported by Newsnet5.com from Ohio, "graduation rate (25%), college matriculation rate (25%), AP tests taken per graduate (25%), average SAT/ACT scores (10%), average AP/IB/AICE scores (10%), and AP courses offered (5%)."

Ah, more junk statistics.

* Students who go early admission don't graduate in many schools.
* Students who study abroad for a year after high school don't matriculate at college (and why are we deciding that college matriculation is so desirable or necessary that it defines a good high school?).
* SAT/ACT scores are often the result of test prep classes and tutors and not the schools, and are not very good indicators of intelligence or performance anyway. And using averages ignores the diversity of a school's population -- a school which has remedial classes and does well by those students though they will still do poorly on a standardized test should not be considered "less top" because it is serving its population effectively!
* AP scores (I can't speak for IB programs) often reflect the school's policy of allowing students to take AP classes. The college board is constantly pushing to allow more students to take AP classes (though their agenda might be driven by the money they collect for every AP test taker). More students allows borderline learners to be exposed to "college level" material. But more students will lower average scores. So either you turn the AP into an elitist program to be a top school, or an open enrollment school to make the college board happy. Brilliant.
* The number of AP courses offered is a reflection on budget and student interest. To say that a school is less of a top school because only 2 students want to take the Music Theory course and it doesn't make sense to hire a skilled teacher to serve only 2, is a silly conclusion to draw. Should we encourage schools to offer more AP courses taught by unskilled teachers simply to bump of the rankings?

So I'm already annoyed because a magazine is trying to rate quality schools while not asking parents and students about their actual educational experience, but asking the school about its and its students participation in external standardized programs.

But then I take a look at some numbers. Of the 10,000 schools Newsweek "reached out" to, 1,100 responded. Suddenly the sample has shrunk. Did schools not respond because they self evaluated and found themselves to be "not top"? Or because the process was cumbersome and they are trying to run a school? Maybe it was because they didn't want to play the little game so they removed themselves, knowing that no news magazine has the ability to evaluate a complex set of curricula and instructors and compare it to that of another school dealing with a different popualtion and set of dynamic needs.

And how many made the "top" list out of the 1,100? 500.

Yes, almost half of the schools which took the time to respond got listed on the final list. I don't know whom I feel sorrieest for, the schools which worked hard to be placed on a not-very exclusive list, the schools which filled out the forms and found that they were not in the top 46%, or one of the almost 9,000 schools which didn't fill in the forms and, had they known that they only had to make the top 500 in order to get that PR bump, would have put in something.

A school's quality is hard to quanitfy. It changes from day to day, moment to moment and year to year. Lists like this are bad ideas, made worse by horrible methodology and execution.