Autism, Mice, and the Dangers of Scientific Hype

Another day, another PR barrage centered on a scientific paper and a real danger that people will be tempted to treat a serious human disease with untried, ineffective, and potentially harmful “cures.” The paper, published in the prestigious magazine Cell, reports that some autism-like symptoms in mice, generated by treating pregnant animals with a compound that mimics viral infection, can be ameliorated by treating their offspring with the bacteria Bacterioides fragilis (1).  Rob Knight, a major player in the “American Gut” project, claims that, “The broader potential of this research is obviously an analogous probiotic that could treat subsets of individuals with autism spectrum disorder.”(2)  This is a truly remarkable claim given the multiple inherent and substantial limitations of the original study.  

So what are these limitations and why do they lead to significant doubts about whether their “promise” is either misguided or likely to be fulfilled?  The first and most obvious issue is whether mice, no matter how experimentally manipulated, can actually be autistic or, better put, whether the symptoms such manipulated mice display are related in any useful way to developing an understanding of autism or the more general “autism spectrum disorder” (ASD) in humans, yet alone alleviate the symptoms displayed by people diagnosed with ASD.  

Let us begin with what autism is currently defined to be.  According to the National Institute of Neurological Disorders and Stroke, “The hallmark feature of ASD is impaired social interaction.”(3)   Because people are so deeply and inherently social, ASD is a serious condition. That said, it remains unclear whether ASD is one or a number of distinct diseases that produce similar symptoms.

Mice, and in fact the vast majority of animals, are very, very much less social than humans, and in so far as they are social, they are social in dramatically different ways.  Moreover, mice often differ quite dramatically from humans in their responses to various physiological insults and how diseases arise and progress.  Unless carefully taken into account, these differences can make observations in mice more or less irrelevant to humans. This has been revealed most recently in the course of comparative studies on sepsis, the life threatening consequence of a wide-spread bacterial infection, where “… researchers report evidence that the mouse has been totally misleading as a model system to investigate at least three major killers – sepsis, burns, and trauma. As a result, years and billions of dollars have been wasted following false leads, they say.” (Kolata, 2013).

Let us consider the mouse as a model for autism/ASD used in the Cell study.  The investigators used an inbred line called C57BL/6N. To induce autism/ASD-like symptoms, pregnant mice were injected on embryonic day 12.5 (birth occurs between 19-21 days) with the compound poly(I:C); this treatment mimics an acute viral infection.  Administration of this compound hyper-stimulates the mother’s immune system, leading to a condition known as “maternal immune activation” (MIA) associated with elevated levels of inflammatory factors in the maternal blood, placenta, and amniotic fluid.  These changes have dramatic effects on fetal development and the affected offspring often display behaviors seemingly analogous to some of those displayed by people with ASD. Of course, since mice never display the types of social behaviors that normal people do, whether the symptoms mice from MIA mothers display are relevant to individuals diagnosed with ASD is speculative at best. 

What the authors of the Cell paper focused on were changes in the behavior of the gut, that is primarily changes in cellular behavior affecting the gut’s permeability in MIA-derived mice and the possibility of repairing these behaviors through treatment of afflicted mice with bacteria. There have been some reports that humans with autism/ASD have gastrointestinal abnormalities, but the data is unclear. A small study, published in the British Medical Journal, found no relationship between an ASD diagnosis and gastrointestinal symptoms (4) while a larger study by the Autism Microbiome Consortium reported “significant enrichment of bowel symptoms and disorders in patients with ASD (11.74% vs. 4.5%, p<0.0001 by chi-square test)”, perhaps not a completely unexpected result from a project centered on the relationship between autism and gut bacteria, but these authors note that their conclusions “may well be affected by the limitations of our study,” which relate to the specific population of patients examined. Here it is worth noting that only a minority (~12%) of individuals with ASD appeared to display gastrointestinal symptoms.

The authors of the Cell paper pursued the hypothesis that autistic/ASD-like symptoms displayed by mice might well arise directly from the defects in their gastrointestinal tract associated with MIA treatment. They found changes in the relative abundance of intestinal microbes between normal and MIA mice. We know that the interaction between intestinal microbes and gut is critical for the normal “development, maintenance, and repair of the intestinal epithelium.”  Following on they tested whether feeding mice B. fragilis, a microbe also found in humans and previously shown to ameliorate experimental colitis, could influence the effects of MIA. Their observation was a striking yes. This treatment also corrected some, but not all of the autism/ASD-linked behavioral effects found in MIA offspring mice. Most interestingly, however, from the perspective of the defining social aspects of autism and ASD, “they (the B. fragilis treated MIA mice) retain deficits in sociability and social preference” (emphasis added). Now given that autism/ASD is primarily a disease of social interactions, together with the rather dramatic differences between human and mouse immune systems brought to our attention by the sepsis studies, these observations suggest that this study’s results may well not be relevant to ASD. Moreover, from a practical perspective “B. fragilis, which accounts for only 0.5% of the human colonic flora, is the most commonly isolated anaerobic pathogen due, in part, to its potent virulence factors. Species of the genus Bacteroides have the most antibiotic resistance mechanisms and the highest resistance rates of all anaerobic pathogens. Clinically, Bacteroides species have exhibited increasing resistance to many antibiotics.” (6)  This suggests that the use of B. fragilis as a “probiotic” treatment in humans might be actively dangerous.    

Based on the lack of compelling evidence i) that mice really can, in any meaningful sense, be made to be autistic, ii) that there are substantial differences between mice and humans, particularly with respect to their immune systems, which is critical to these studies, and iii) that treatment of mice with B. fragilis fails to reverse the social symptoms of MIA mice, one might well have expected that objective, disinterested scientists would refrain from excessive hyperbole until more relevant and DEFINITIVE data is available. This would not be of great concern if confined to the scientific community but the danger is that such sanguine interpretations in the public press will lead parents or caretakers of people with ASD  to “medicate” affected children with unproven, ineffective, and potentially hazardous  “probiotic” treatments.  This concern is not a theoretical one when one considers how ubiquitous self-medication is in our society.  Any trip to the grocery store or Costco will reveal the widespread availability of dietary supplements including probiotics and other “alternative” medicines that claimed to prevent or treat a wide array of disorders, real or imagined, including gastrointestinal ailments. In the absence of a sober evaluation of the soundness of the observations made in mice and, more importantly, their relevance to humans, the essential question is whether it is socially irresponsible not to include clear and appropriate caveats directed to a scientifically-naïve population who may rightly assume the experiments described represent established fact and therefore, give license to self-directed probiotic treatment of affected individuals, or included in diets of unaffected children as a prophylactic measure.   

1. “Microbiota Modulate Behavioral and Physiological Abnormalities Associated with Neurodevelopmental Disorders,” http://www.ncbi.nlm.nih.gov/pubmed/24315484. 

2.  http://www.dailycamera.com/cu-news/ci_24756593/cu-boulder-professor-says-study-carries-groundbreaking-potential

3. http://www.ninds.nih.gov/disorders/autism/detail_autism.htm 

4. http://www.bmj.com/content/325/7361/419 

5. Kohane et al., 2012. PLoS One DOI: 10.1371/journal.pone.0033224

6. Wexler, 2007: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2176045/ 

Further reading.

Engber, D. 2011. The Mouse Trap. Slate: http://www.slate.com/articles/health_and_science/the_mouse_trap/2011/11/lab_mice_are_they_limiting_our_understanding_of_human_disease_.html

Kolata, G. 2013. Mice Fall Short as Test Subjects for Some of Humans’ Deadly Ills. New York Times. http://www.nytimes.com/2013/02/12/science/testing-of-some-deadly-diseases-on-mice-mislead-report-says.html?_r=1&

Measuring learning: identify effective courses through data analysis.

Higher education faces serious challenges, not the least of which is how to provide compelling evidence that the learning attained is worth the cost, particularly since a number of studies suggest learning is often less impressive than many colleges might wish to portray. There is a growing interest in more effective and efficient educational practices and companies that claim to provide it. Lurking In the background is a “college ranking” industry that rarely uses objective measures of student learning. Altogether, this raises the question, what would evidence-based higher education look like? 

We became interested in how to improve educational outcomes as professors of molecular biology and chemistry, respectively. We, and others, have come to the conclusion that there is a need to change both the way students are taught and what they are taught. Lecturing alone, an anti-Socratic strategy, often presents a fire-hose of information that is, at best, superficially understood. While students can pass exams, the extent of their understanding remains problematic, since it is often unclear what the exams used actually measure.

So how to use evidence-based practices to monitor educational outcomes?  Clearly, what is missing is an objective method to characterize what a particular course claims and what it actually delivers in terms of student learning. At the same time, it is critical to recognize that monitoring educational outcomes is a complex task and easily manipulated for a variety of non-educational purposes. Various “accountability” projects, including both the Bush and Obama administrations’ “No Child Left Behind/Race to the Top” schemes, tend to approach assessment in a ham-fisted, one-size-fits-all manner, with the pernicious effect of eroding support for public education.

Our suggestion builds on the growing ubiquity of the digital collection of student assessment data. With access to the questions students are asked, the answers the students generate, and how these answers were graded, which we refer to collectively as a student’s “ed-data”, it is possible to describe what a particular course aims to teach and what it actually delivers. It enables us to distinguish between memorization-based assessments, that monitor who is paying attention, and assessments that require an accurate working understanding of the materials presented. Using student’s ed-data we can, for the first time, describe courses on their own terms rather than through what may be seen as, and often are, irrelevant or arbitrary assessments. We can focus attention on what the instructor or institution deems important. An ed-data-based analysis would reveal what types of answers are acceptable and so provide direct evidence of the rigor of a class. Evaluating evaluations uncovers what a particular course actually values and delivers.      

At this point, the non-academic might object: don’t all courses with a given title, say genetics or chemistry, teach similar content and don’t all students passing such courses learn more or less the same things, together with the critical thinking skills needed to apply that knowledge productively? Sadly, there is little evidence to support this conclusion. A systematic analysis of student’s ed-data would help identify those courses, curricula, or institutions that value and help foster critical thinking and subject mastery.  

Using student’s ed-data raises a practical question.  Who owns this data–the student, the institution, or the instructor?  We would argue that a student’s ed-data, like medical data, is the property of the student or that, at the very least, they have a right to its fair use. For example, it could be used to establish that courses delivered by less prestigious institutions produced levels of subject mastery similar to more expensive and exclusive alternatives. 

We suspect that instructors and their institutions will robustly defend their claims of exclusive ownership over student’s ed-data. The problem is that analysis of ed-data is the only objective and economical way to establish that a course has, or could have, delivered what it purports to teach, that is, what the student paid for. The issue becomes one of a product claim, a claim that, in the educational arena, must surely be based on what has been learned. 

Assuming ownership and use issues can be resolved, we are left with the practical question, who would perform and assume of cost of such analyses?  There are extant commercial entities, and perhaps a start up or two, that may be willing to provide such services. The cost could even be part of an institution’s accreditation process or third party ranking schemes.

The analysis of student ed-data provides a unique opportunity to distinguish educational excellence from mediocrity and outright malpractice. It would enable low-cost education providers to establish their quality, provide an impetus for traditional institutions to improve on their efforts, and establish a valuable criteria by which to evaluate educational enterprises. 

Mike Klymkowsky is a professor of molecular, cellular and developmental biology at the University of Colorado Boulder and co-director of the CU Teach science and math teacher preparation program.  Melanie Cooper is the Lappan-Phillips Professor of Science Education and a professor of chemistry at Michigan State University. Both are fellows of the American Association for Advancement of Science and recipients of the Outstanding Undergraduate Science Teacher Award from the Society for College Science Teachers/National Science Teachers Association.  Many of the ideas expressed here emerged through a workshop on student education data held at ETH Zurich in June 2013.