November 09, 2004

Maclean's University Rankings

The latest issue of Maclean's magazine is their annual university ranking issue. This is the first year where they asked graduates what they thought of their Universities. While I think that this is a good thing to do, I think that Maclean's could have done this better.

It appears (I can find a complete list) that graduates were asked seven questions regarding their experience at University. The sampling plan is described well and appears to be well thought out. However, there was only a 17% response rate, which while high for most surveys of this type, suggests that there might be some bias in who responded in who didn't. Maclean's acknowledges this fact and says that the survey data was statistically weighted to to attempt to minimize this bias but doesn't expand on how this weighting was done.

Another concern I have is how they decided to present the results of the survey. Instead of doing something reasonbly sensical like presenting histograms of the responses to each question they decided to present the percentage of 'very good' or 'definitely yes' responses to each question. How often when you do a survey with a five point Lickert scale (Very Bad, Bad, Neutral, Good, Very Good) do you answer Very Good? what if you were waffling between Good and Very Good? I typically would only answer Very Good if there was little room for improvement. How were survey participants instructed to interpret their responses? These instructions, which I'm assuming existed, are known to influence how people respond but Maclean's did not include them in the graduate response section. Maclean's tries to address the issue of only reporting 'Very Good' responses by including a table of percentage of all responses for each University that were Very Good or Good (pg. 33). When looking at this you see that these responses made up approximately 90% of the responses for each University.

I can kind of understand why they wouldn't have histograms for the responses for every university for every question, it would take up too much space. But why not at least have a weighted score based on all the response categories? They do that for their rankings and go into great detail explaining how they weight the ranking score.

While I'm glad to see that Maclean's is finally asking graduates for their opinion about their University experience, I think that some more thought needs to go into the design and reporting mechanism for next year's issue.

Posted by bforsyth at November 9, 2004 09:48 PM