On self experimentation, soy, and memory

24 Mar

When I ask my students to read a research paper, they often argue that “the sample size was too small.” Typically, students who are new to statistical analysis do not understand the mathematics of sample size, and this becomes a reflexive criticism. If they don’t like the result of a study they argue that a larger sample was needed.

This is why students are often shocked when I talk about the value of single subject research and argue that it is possible to gain information when the sample size is N = 1. One of the advantages of single subject research is that you can control for background characteristics, such as genetics, because these traits remain constant over the individual.

One variant of single subject design is self experimentation, as advocated by the always interesting Seth Roberts. Roberts recently ran an experiment on himself where he found, in his words, that tofu made him “stupid.” To his credit Roberts recently linked to a post by  Alex Chernjavsky which reached the opposite conclusion:

 “The results were not consistent with the hypothesis that eating soy is harmful to brain function.  Surprisingly, my scores became significantly faster during the study.”

One possibility for the different results is that self-experimental trials are often not blinded, that is the subjects frequently know which treatment they are receiving and unconscious bias might play a role in the results. For example,  Roberts is an advocate of a meat based diet while Chernjavsky is a vegan. Chernjavsky, himself suggests:

“I don’t know why my results are inconsistent with prior work.  Perhaps people differ in their sensitivity to soy.  Or perhaps I’ve been eating so much soy for so long that I’ve made myself resistant to any changes that might result from relatively short-term fluctuations in level of soy consumption.”

I have three suggestions:

1. While obviously difficult to implement, self experimentation would be more persuasive if the the experimenter was blinded to the different treatments.

2. We should develop better methods for combining the results of many single subject and self experimental designs. Perhaps some form of Bayesian analysis.

3. The results of single subject and self experimental designs should be replicated with larger sample sizes (although they don’t have to be very large) using within subject designs. This will help guarantee the reliability, validity, and  generalizability of the results.

Here is Dr. Greger on the brain effects of tofu:

Enhanced by Zemanta

4 Responses to “On self experimentation, soy, and memory”

  1. locksleyu March 24, 2014 at 4:57 pm #

    While I agree its possible to gain “some” information from a N=1 study, in general there seem so many problems that the information is close to nil. As one of the quotes above says, perhaps different people are differently sensitive.

    Given this, why are you focusing on N=1 studies, what can they offer than a larger study cannot? I would prefer a 10 person (or 100 person) study over a N=1 one in pretty much all cases, and never would I make any important decisions on a N=1 study.

    • jecgenovese March 24, 2014 at 5:26 pm #

      Thank you for your comment.
      Actually there is a lot of good research done with single subject designs, especially in the field of behavior analysis. Between subject designs can be misleading because they may not fully control for differences between the control and experimental groups. See the excellent book Single-Case Designs for Educational Research by Craig Kennedy

      • locksleyu March 24, 2014 at 6:24 pm #

        Thanks for the reference. By the way I just looked at this post again and I see that “… and memory” is part of the title. Which part of the post is referring to memory? Did the studies with Tofu degrade memory?

      • jecgenovese March 25, 2014 at 1:03 pm #

        While memory improvement is the main focus of this blog and my upcoming book, I do blog about other topics including psychology, self quantification, and realizing human potential.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: