Questioning the implicit association test

7 Jul

New York Magazine has published a devastating critique of the widely cited implicit association test (IAT). The creators of the IAT claim that it can ferret unconscious biases. From the point of view of psychometrics, the first important question we must ask about any instrument is its reliability, that is, does the measure yield consistent results if the measured phenomenon has not changed. It is a basic law of measurement that an unreliable measure can not be valid.

Here is what the article says about reliability:

What constitutes an acceptable level of test-retest reliability? It depends a lot on context, but, generally speaking, researchers are comfortable if a given instrument hits r = .8 or so. The IAT’s architects have reported that overall, when you lump together the IAT’s many different varieties, from race to disability to gender, it has a test-retest reliability of about r = .55. By the normal standards of psychology, this puts these IATs well below the threshold of being useful in most practical, real-world settings.


5 Responses to “Questioning the implicit association test”

  1. Enrique Guerra-Pujol July 7, 2017 at 1:38 pm #

    to the extent all these “tests” are based on subjective self reports, they are all inherently flawed

  2. Enrique Guerra-Pujol July 8, 2017 at 6:43 am #

    Quick follow up: this post rekindled (in me) the old problem of making “interpersonal comparisons of utility.” More here:

    • jecgenovese July 8, 2017 at 2:03 pm #

      Thanks for the comment and the link!

      I don’t completely dismiss self report data, if it is reliable and if it has predictive validity, that is, if responses can predict some real world outcome that we care about. For example, there are a number of clinical self report instruments in psychology that are helpful in diagnosing conditions such as depression.

      It occurs to me that utility is a much more slippery concept than depression. Depression, on the other hand, has a number fairly well defined behavioral correlates.

      • Enrique Guerra-Pujol July 8, 2017 at 7:55 pm #

        Yes, I have always found the concept of “utility” in economics to be very problematic, as you cannot measure what you cannot define (to paraphrase Lord Kelvin!)


  1. True or false: can we make interpersonal comparisons of utility? | prior probability - July 8, 2017

    […] science studies relying on surveys and other subjective tests are totally bogus. By the way, it was this post regarding the Implicit Association Test that motivated us to pose this question about interpersonal […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: