Skip to main content

Deceptive Per Captia Rankings for Brain Cancer

We're working on a project to produce a report on the ratings that students give to their professors at the end of the semester. There is a big concern by the math people on the committee that we will give some professors an unfair bad (or good) rap because of the variability in these rankings. They don't want to report the ratings as a mean (average). Instead, they want to plot an uncertainty range. 

I was reading a book this week (How Not to Be Wrong by Jordan Ellenberg) that provided a great example of the risk of ranking things when there is uncertainty. It can lead to erroneous conclusions. Here is a summary of his argument that appeared in an NPR interview. Perhaps this sort of example will be helpful for the committee to share when teaching the general faculty about the new instrument.

If you take a rare disease like brain cancer and you look at its rate of incidents in different states, there are very big differences. And so you might say, "Well, I should go where this form of cancer is the rarest. Clearly something's going on in that state that is preventative against that disease." But when you look at the numbers, they're rather strange because at the very top of the list you see South Dakota with an extremely elevated rate of brain cancer, but if you look at the bottom, you see North Dakota with almost none. So that's very strange because South Dakota and North Dakota are not actually all that different.

But when you look at those numbers a little more closely, what you notice is that the states at the top of the list [South Dakota, Nebraska, Alaska, Delaware, Maine] and the states at the bottom of the list [Wyoming, Vermont, North Dakota and Hawaii, and the District of Columbia] have something in common, which is that they are very small. ... So basically hardly anybody lives in those states; that's what they have in common. And a sort of fundamental principle is that when you compute rates, the smaller the state, or ... the smaller the sample size, the more variation is going to be created just by random chance.

This seems analogous to the problem of small class sizes for ratings which cause us to draw a longer uncertainty bar on the report. One disgruntled student in a small class can cause a disproportionate movement on the class average. 

Comments

Post a Comment

Popular posts from this blog

Becoming a SharePoint Convert

SharePoint has always been in my peripheral vision, but never a tool that I actually used. We're in the middle of an investigation project to look at putting up an enterprise instance of it. I watched the Lynda.com intro training for SharePoint 2010 and I have to say that I'm quite impressed with how far SharePoint has come since my first introduction to it. It is sporting the ribbon interface from the Office applications that I've come to like very much. The ability to keep calendars, task lists, and even documents synchronized with Outlook is awesome. I wish it was a little more straightforward to save a document from Microsoft Word or Microsoft Excel in to your SharePoint work space for the first time. Right now, the interface feels a little broken to me. You click the button to browse for a SharePoint location to save, and word does absolutely nothing. You click the button again, and it still does nothing. It turns out that you have to click the SharePoint button, a...
Post a Comment
Read more

Hammers and Nails: Technology Push Design

"We need to refine our requirements first, before we look at tools." This is a common phrase that I hear. While I sympathize with the sentiment, I think it is frequently wasteful. I suspect that we'd get to the right requirements faster by looking at tools already available in a given problem space. Pushing the concept further, is it foolish to find a cool technology and then look for ways that that technology can apply to current problem spaces?  What if you don't even recognize you have a problem space? Without a constant search and openness, we'll miss many serendipitous opportunities. Here is BYU professor Larry Howell discussing this issue. I often enjoy doing something ... that is sometimes controversial. In this approach, rather than starting with a need, you start with a new technology and you search to identify a need that it can fulfill. This second more controversial approach is called "technology push design."   You can imagine t...
Post a Comment
Read more

Broader Process Sharing - Service Request Fulfillment

While at a BMC UserWorld conference a few years ago, I wrote the following. It still applies and I wanted to put it down in a permanent spot where I could find it later. ========== We're perfecting the Service Desk call center. What would it look like if we invited other campus support units to share our processes and tools, if not our Service Desk? My recent experience moving into a new building has sharpened my focus on this issue. I'd like to share a taste of my experience by way of introduction and illustration of my implementation recommendation. In my new office, I need to obtain keys, request door locks to be installed, deal with furniture alterations, move network and power outlets that were obstructed by furniture, request appropriate signage for our department, order new furniture, resolve electrical wiring problems involving and interface with the furniture, request building card swipe access, request thermostat adjustments, and request permanent placement of...
Post a Comment
Read more