The deception of “Studies say”

In my recent talk on Calling BS on Social Media Gurus, I pointed out how studies are frequently misrepresented by by experts, journalists and even the researchers themselves.

My point is not that you should never believe anyone. It’s just that you should ask some basic questions before you do.

Here’s a recent example:

The Washington Post reported on a study published last week by The University of Michigan about college students and empathy. The article is titled College students losing their sensitive side.

Their second paragraph makes this claim:

“A new University of Michigan study has found that since 2000, college students have become less empathetic.”

The problem is this isn’t quite what the research says.

And the Washington Post article is likely based on the press release from the University of Michigan (Non an uncommon thing, but a surprise to those who don’t know this).

The actual research is just a survey, much like the ones you avoid filling out if you can. The survey asks a bunch of questions about student’s own perceptions of their empathy. This is not the same as their actual empathy, which is hard to measure, unless during the study you can throw some wounded puppies their way and see how the students behave, an experiment hard to do for 14000 students.

Some headline stories, such as this one from The Globe and Mail, lead with the claim today’s students are 40% less empathetic than previous generations. What does that even mean? That they’d let 40% more puppies die? No. It means on a survey 40% of them answered one radio box to the left, instead of to the right, or something similarly abstract and detached from a specific situation where empathy might apply.

So what the study actually says is this: students perceptions of their own empathy, as framed by a survey, is lower than previous generations.

Measuring someone’s perception is not the same as measuring their behavior. You might think you’re cool, but really you’re a jackass. Or vice-versa. But this article suggests perception of yourself and reality are the same thing (There is one referenced study that claims a correlation, but it’s behind a paywall, which I perceive as annoying). It is entirely possible there are other reasons for the change in data. Perhaps students are more self-aware and honest in 2010 about their selfishness than previous generations? Seems possible. It’s certainly a question worth asking.

Konrath, one of the researchers is quoted as saying “We found the biggest drop in empathy after the year 2000… college kids today are about 40 percent lower in empathy than their counterparts of 20 or 30 years ago, as measured by standard tests of this personality trait.”

The key phrase is “standard tests of this personality trait”. By standard, she probably means these surveys, and all surveys have known problems, biases and limitations (The specific personality test is apparently the Davis Inter-personality Reactivity Index). Standard tests are limited – very limited. They might be the best tools we have, but when a limited tool is used to make a general claim, it’s less based on science than opinion, especially when a scientist is asked to explain “why”. The studies are rarely designed to explain why, but that doesn’t stop many of these articles and experts from theorizing on why.

And to be completely fair – even if there is good reason to make these claims, it doesn’t seem the journalists and reporters writing about these claims have done much work to verify that’s true. Also, everyone is entitled to an opinion. But quoting a scientist’s opinion is typically framed as science, rather than as opinion.

If nothing else, take the actual survey yourself – you’ll see what the students in the survey saw, and when you’re done you’ll be scored against the actual data. Cool. And I suspect you’ll feel more aware of what studies and claims might really mean.

For reference, the same basic story that appeared in the Washington Post also appeared on:

MSNBC
USnews (they did better ‘Today’s College Students More Likely to Lack Empathy’)
Psychology Today
LA Times (actually talks about the study)
SfGate – This opinion piece suggests our critical view of youth is perennial

But none provide a link to the actual paper presented last week. I dug around for a half-hour and found a PDF of their summary from Sara Konrath’s site. I’d really love to see a rule where any article referring to any study or research must include a link to the actual research. It’s rarer than you’d think.