Skip to main content

Deceptive Per Captia Rankings for Brain Cancer

We're working on a project to produce a report on the ratings that students give to their professors at the end of the semester. There is a big concern by the math people on the committee that we will give some professors an unfair bad (or good) rap because of the variability in these rankings. They don't want to report the ratings as a mean (average). Instead, they want to plot an uncertainty range. 

I was reading a book this week (How Not to Be Wrong by Jordan Ellenberg) that provided a great example of the risk of ranking things when there is uncertainty. It can lead to erroneous conclusions. Here is a summary of his argument that appeared in an NPR interview. Perhaps this sort of example will be helpful for the committee to share when teaching the general faculty about the new instrument.

If you take a rare disease like brain cancer and you look at its rate of incidents in different states, there are very big differences. And so you might say, "Well, I should go where this form of cancer is the rarest. Clearly something's going on in that state that is preventative against that disease." But when you look at the numbers, they're rather strange because at the very top of the list you see South Dakota with an extremely elevated rate of brain cancer, but if you look at the bottom, you see North Dakota with almost none. So that's very strange because South Dakota and North Dakota are not actually all that different.

But when you look at those numbers a little more closely, what you notice is that the states at the top of the list [South Dakota, Nebraska, Alaska, Delaware, Maine] and the states at the bottom of the list [Wyoming, Vermont, North Dakota and Hawaii, and the District of Columbia] have something in common, which is that they are very small. ... So basically hardly anybody lives in those states; that's what they have in common. And a sort of fundamental principle is that when you compute rates, the smaller the state, or ... the smaller the sample size, the more variation is going to be created just by random chance.

This seems analogous to the problem of small class sizes for ratings which cause us to draw a longer uncertainty bar on the report. One disgruntled student in a small class can cause a disproportionate movement on the class average. 


Popular posts from this blog

The "True Cost" Phantom in Project Cost Tracking

I want to know if I should wear a coat when I go out the door on an unpredictable spring morning. The temperature from the weather app on my phone isn't going to match exactly the temperature just outside my front door, but it will be close enough to make the decision. If the question at hand is whether or not I should wear a coat, it would be a costly mistake to set up an elaborate system of thermometers around my property to try increase the precision of my measurement of the temperature. The added expense wouldn't yield any better decisions about whether to wear a coat, so why bother?

We are interested in tracking the relative costs of the various projects we undertake in our organization. We must admit up front that we will never be able to measure the precise cost of each of our projects. Consider the following things you could include in the "true cost" measurement of a project. Where will you draw the line?

An engineer reads an article on the bus that gives her…

The UX of Windows 8

Jensen Harris, the Director of Program Management for the Windows User Experience Team, gave a really compelling talk where he explained the work his team had done in crafting the UX for Windows 8. They knew they needed to shake things up and re-imagine some concepts that had become stale. They threw out the Start menu and built a cool new tiled interface.
He talked about how important it was to sweat the small stuff. For example, he talked about the herculean effort it was to eliminate a little blue flash in the boot sequence the first time you power on a device straight from the manufacturer. His point was that the seemingly small stuff REALLY matters when we're creating an experience for a user. A little bit of sand in my hamburger is going to spoil the whole experience. 
I had to compare my own experience as a user of Windows 8 with the description I was hearing from Harris. I thought it still had a lot of rough edges and it was an uncomfortable melding of two paradigms that m…