Tag Archives: Statistics

Just how big is that effect size?

24 Jan

On Monday The Washington Post ran a story with this headline: “Teens who spend less time in front of screens are happier — up to a point, new research shows.”

What the article does not tell us (but the abstract does) is the that the study had 1.1 million participants. Well that seems like a good thing, doesn’t it?

The problem is that with a sample that large almost any correlation will be statistically significant. For example, according the Post account, the correlation between texting and happiness was r = -.05. Typically a correlation of the this magnitude would be described as “none or very weak.” 

 

Testing a hypothesis: Over 21 edition

24 Apr

Tonight, I am going to lecture about hypothesis testing. I plan to show my students this video I found on BoingBoing:

The question for the students: was this a fair test?

Becoming a better consumer of research

5 Apr

This article, published by the Association for Psychological Science, is aimed at improving the skills of university students, but I think everyone could benefit from a better understanding of research.

Consistent with my experience, author Beth Morling tells us that her students “almost always think larger samples are more representative.” When evaluating research, she advises us to pay attention to four types of validity:

“external validity (the extent to which a study’s findings can generalize to other populations and settings

internal validity (the ability of a study to rule out alternative explanations and support a causal claim);

construct validity (the quality of the study’s measures and manipulations); and
statistical validity (the appropriateness of the study’s conclusions based on statistical analyses).
To evaluate any study they read, students can ask questions in these four categories:
“Can we generalize?” (External);
“Was it an experiment? If so, was it a good one?” (Internal);
“How well did they operationalize that variable?” (Construct); and
“Did they have enough people to detect an effect? How big was the effect? Is it significant?” (Statistical).”

Enhanced by Zemanta

On self experimentation, soy, and memory

24 Mar

When I ask my students to read a research paper, they often argue that “the sample size was too small.” Typically, students who are new to statistical analysis do not understand the mathematics of sample size, and this becomes a reflexive criticism. If they don’t like the result of a study they argue that a larger sample was needed.

This is why students are often shocked when I talk about the value of single subject research and argue that it is possible to gain information when the sample size is N = 1. One of the advantages of single subject research is that you can control for background characteristics, such as genetics, because these traits remain constant over the individual.

One variant of single subject design is self experimentation, as advocated by the always interesting Seth Roberts. Roberts recently ran an experiment on himself where he found, in his words, that tofu made him “stupid.” To his credit Roberts recently linked to a post by  Alex Chernjavsky which reached the opposite conclusion:

 “The results were not consistent with the hypothesis that eating soy is harmful to brain function.  Surprisingly, my scores became significantly faster during the study.”

One possibility for the different results is that self-experimental trials are often not blinded, that is the subjects frequently know which treatment they are receiving and unconscious bias might play a role in the results. For example,  Roberts is an advocate of a meat based diet while Chernjavsky is a vegan. Chernjavsky, himself suggests:

“I don’t know why my results are inconsistent with prior work.  Perhaps people differ in their sensitivity to soy.  Or perhaps I’ve been eating so much soy for so long that I’ve made myself resistant to any changes that might result from relatively short-term fluctuations in level of soy consumption.”

I have three suggestions:

1. While obviously difficult to implement, self experimentation would be more persuasive if the the experimenter was blinded to the different treatments.

2. We should develop better methods for combining the results of many single subject and self experimental designs. Perhaps some form of Bayesian analysis.

3. The results of single subject and self experimental designs should be replicated with larger sample sizes (although they don’t have to be very large) using within subject designs. This will help guarantee the reliability, validity, and  generalizability of the results.

Here is Dr. Greger on the brain effects of tofu:

Enhanced by Zemanta
%d bloggers like this: