Skip to main content

Deceptive Per Captia Rankings for Brain Cancer

We're working on a project to produce a report on the ratings that students give to their professors at the end of the semester. There is a big concern by the math people on the committee that we will give some professors an unfair bad (or good) rap because of the variability in these rankings. They don't want to report the ratings as a mean (average). Instead, they want to plot an uncertainty range. 

I was reading a book this week (How Not to Be Wrong by Jordan Ellenberg) that provided a great example of the risk of ranking things when there is uncertainty. It can lead to erroneous conclusions. Here is a summary of his argument that appeared in an NPR interview. Perhaps this sort of example will be helpful for the committee to share when teaching the general faculty about the new instrument.

If you take a rare disease like brain cancer and you look at its rate of incidents in different states, there are very big differences. And so you might say, "Well, I should go where this form of cancer is the rarest. Clearly something's going on in that state that is preventative against that disease." But when you look at the numbers, they're rather strange because at the very top of the list you see South Dakota with an extremely elevated rate of brain cancer, but if you look at the bottom, you see North Dakota with almost none. So that's very strange because South Dakota and North Dakota are not actually all that different.

But when you look at those numbers a little more closely, what you notice is that the states at the top of the list [South Dakota, Nebraska, Alaska, Delaware, Maine] and the states at the bottom of the list [Wyoming, Vermont, North Dakota and Hawaii, and the District of Columbia] have something in common, which is that they are very small. ... So basically hardly anybody lives in those states; that's what they have in common. And a sort of fundamental principle is that when you compute rates, the smaller the state, or ... the smaller the sample size, the more variation is going to be created just by random chance.

This seems analogous to the problem of small class sizes for ratings which cause us to draw a longer uncertainty bar on the report. One disgruntled student in a small class can cause a disproportionate movement on the class average. 

Comments

Popular posts from this blog

Sitting or Standing at Work

I've read some stuff about the negative effects of sitting for prolonged periods of time. As a consequence, I've been experimenting with standing more at work when the task will permit it. I've been enjoying it. I think the next step to make that work even better would be to get a desk that can quickly raise or lower. (The glacial speed movements of some electric raise/lower desks would discourage much use.) Here are some resources I've found interesting.

NY Times article is a good summaryThis piece from Exercise and Sport Science Reviews is co-sponsored by Steelcase, the furniture makers, so I must take it with a grain of salt, but it recommends standing as an activity that should be categorized as superior to sitting, whereas they have been both lumped together as "sedentary" in previous literature. An abstract with says standing or sitting on a therapy ball are about the same, and both more active than sitting in a chair. This survey of papers says the resu…

How much will you remember?

There is a commonly passed around "stat" that, according to a blog post I recently read, isn't true. You've probably seen it or heard it.

It is said that people remember:10% of what they read20% of what they hear30% of what they see50% of what they see and hear70% of what they write and say90% of what they say as they do
The blog author says:
Quite where these numbers come from is a mystery to many, and indeed it is difficult to understand what 90% retention actually means… 90% of what for how long? As a model it looks and on first thought appears to be credible, however as many of us will know some people have almost 100% retention for a considerable period of time if they read something, others teach others from a structure or procedure which they themselves do not understand!Thanks, RapidBI, for pointing out this common misconception!

Mike Rowe Says Don't Follow Your Passion

With a zingy catch line, Mike Rowe explodes some myths about "working people" in this talk on Fora.tv, "Mike Rowe on Discovery, Realization and Lamb Castration." Some surprising insights.
Lamb castration the way the Humane Society recommends (with rubber bands) is more painful for the lamb than the way the shepherds actually do it (with a knife)."Following your passion" is not a mandatory ingredient for a happy work life.People with "dirty jobs" are very well balanced and happy.Safety first is a slogan that doesn't always make sense for the people doing the job. His example about the crab fishing rig was a good example. The captain said, "OSHA? We've got Ocean. I'm not here to make you safe. I'm here you make you rich. If you want to get home alive, that's your job."