On Wednesday, I blogged about a new study of the effects of yoga on cognitive impairment. Thinking it over, I realized that some of the study’s results rest on a serious methodological flaw.
The study compares measures before the intervention to measures after the intervention within in each group. For example, it looks at the Geriatric Depression Scale scores for the yoga group before and after the intervention and says that there is a statistically significant difference. But this is not the correct analysis, we want to compare the changes between the yoga group and the control group. An appropriate procedure would have been a gain score analysis. The authors could have subtracted the after treatment scores from the before treatment scores and then compared those two values using an appropriate statistical test.
In the other words, the study had the possibility of comparing the control and the experimental group but failed to so. All it really says it that the scores improved in the treatment group. That is an interesting finding, but it should be considered only exploratory and suggestive. I have no objection to publishing exploratory findings, I have done so myself. But the authors had the opportunity to make a better test and they failed to do so.