Pondering the South African Memesphere – Looking for the Good in Everything header image 2

Selection Bias

August 22nd, 2010 · Posted by Hugo · 7 Comments

There are many mistakes one can make when setting up a scientific study. Awareness of such mistakes and biases is very important so that we can avoid drawing false conclusions from bad data. Today I’m briefly introducing selection bias.

Suppose we are doing a survey in which we need to talk to a large number of people, we want to collect statistically representative opinions on privacy concerns. We pick up the phone book, pick random names from the list and start dialling. That ought to be quick and practical, right?

No it won’t. It would be a bad idea, and for more reasons than just the fact that we’re making unsolicited calls (“cold calling”): people more obsessed over privacy are less likely to be listed in the phone book, thus giving our survey a sampling bias — the opinions of the privacy conscious would not be adequately represented in our results. Combined with that there will be the subset of people we phone that will refuse to talk to us. This subset might also correlate in some way with opinions relevant to the survey, resulting in further bias in our results.

Let’s consider another cause of sampling bias: self-selection bias. Suppose you were to openly look for volunteers to take part in some kind of study of human sexuality. You could end up with a disproportionately large number of liberals and people with exhibitionist tendencies, and too few conservative, shy or repressed participants. Unless you can somehow accurately compensate for this effect, your conclusions would not generalise to the population at large. They would be biased by self-selection, your study would be under the impression that humanity is more liberal and exhibitionist than it really is.

If you are ever going to rely on volunteering for a study, you have to be sure that the variables being studied is completely unrelated and uncorrelated to the criteria by which people end up self-selecting. Easier said than done.

There are many more types of selection bias. Wikipedia lists a bunch.

Categories: Science

7 responses so far ↓

  • 1 Kenneth Oberlander // Aug 25, 2010 at 5:21 pm

    Nice post Hugo!

  • 2 Hugo // Aug 25, 2010 at 10:56 pm

    Thanks Kenneth, thought you’d like it, though I’m unable to tell if it’s just because of the topic / subject matter or because it’s actually well written: I bet you’d like all pro-critical-thinking posts. 😛

  • 3 Kenneth Oberlander // Aug 26, 2010 at 7:34 pm

    Well…it is rather less wordy than usual!

    I bet you’d like all pro-critical-thinking posts.

    Depends how they’re written 😛

  • 4 Confirmation Bias // Aug 30, 2010 at 12:32 am

    […] Blog | Comments ← Selection Bias […]

  • 5 Hugo // Aug 30, 2010 at 12:37 am

    I was also wondering if you had some examples of selection bias that you had to watch out for in some of the studies you did? Or read about? (If there’s something off the top of your head, don’t waste time on it!) Real world examples are way cooler than possibly dubious examples I suck out of my thumb. 😉

  • 6 Diet Coke Makes You Fat — On Correlation and Causality // Nov 2, 2010 at 2:39 am

    […] get to voluntarily choose between smoking and not smoking, you will have a problem ruling out self-selection bias: some unknown factor (C), be it a genetic predisposition or something else, could influence what […]

  • 7 UHU Tv Forum // Apr 2, 2011 at 11:02 pm

    we need to study the cognitive sciences, figure out the way our intuitions work and how we might correct for mistakes. Above all, we need to learn to always question the workings of our minds, for we need to understand that they are not magical.

Leave a Comment

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>