[*original source of the estimated number]
By Mark Blaxill
Reading scientific papers about autism can be really frustrating sometimes, especially when smart scientists write silly things. As I’ve discussed in several previous columns here, the autism scientific literature certainly has its fair share of mistakes, errors and downright outrageous comments. The slow response of so many scientists to the crisis facing our children will go down in history as one of the greatest failures in the history of modern medicine
But every once in a while, you run across a piece of brilliance, even deep wisdom, embedded in the dense, technical prose that characterizes most scientific papers. I ran across one of these recently. The title of the paper, “Microparadigms: chains of collective reasoning in publications about molecular interactions” doesn’t exactly trip off the tongue. But the words of the author team, headed up by Columbia Biomedical Informatics Professor Andrey Rzhetsky, jumped off the page when I read them.
Their paper describes a statistical model they developed that is quite complex. But the authors' point is exceedingly simple and written with a clarity often lacking in these kinds of papers. They state their main point directly at the end of their opening paragraph.
We…found that previously published statements, regardless of whether they are subsequently shown to be true or false, can have a profound effect on interpretations of further experiments and the probability that a scientific community would converge to a correct position.
In other words, when a scientist says something wrong, if the statement gets a kind of social momentum among other scientists, the initial error can be perpetuated for an extended period of time without correction. The authors base this conclusion on a model of published sequential statements about molecular interactions. Using a remarkable database of over 3 million such statements (e.g., "A inhibits B" or "C induces D"), the authors observed a particular pattern of influence called an information cascade, i.e., the “special form of collective reasoning chain that degenerates into repetition of the same statement.” The durability of these degenerative information cascades led the authors to suggest a quite pessimistic conclusion regarding the reliability of the much-vaunted scientific method. In blunt terms, they observed:
[Our] finding suggests that the scientific process may not maximize the overall probability that the result published at the end of a chain of reasoning will be correct.
I was reminded of Rzhetsky’s paper last week when I picked up the recent study by Weiss et al in The New England Journal of Medicine on behalf of the Boston-based “Autism Consortium.” This was the study that made headlines all over the world for the consortium's claim to have located a consistent de novo mutation in about 1% of autism cases. Part of the latest autism gene science fad, this paper and an accompanying editorial made a big deal out of a small finding: that a small number of autistic children were missing a small portion of the DNA (a microdeletion) on the short arm of chromosome 16.
I’ll come back to the Autism Consortium’s findings in a later column, but there was something else in the paper by Weiss et al (there were 22 authors in total) that caught my eye. In the paper’s first paragraph, the authors made a quite astounding claim.
In approximately 10% of patients, autism can be explained by genetic syndromes and known chromosomal anomalies (most of which have recognizable features in addition to autism) including the Fragile X syndrome, tuberous sclerosis, the Smith-Lemli-Opitz syndrome, the Potocki-Lupski syndrome, and…the region (15q11-13) that is affected in Prader-Willi and Angelman syndromes
A full 10% of autistic cases have already been fully explained as genetic syndromes? After close to ten years of dealing with autism, I can safely say that I have read well over a hundred papers on the genetics of autism -- genome scans, single gene studies, copy number variant papers, and review articles — written over four decades. In all that time, I had never seen a statement this definitive before and it caught my attention: in part because if autism rates have multiplied tenfold, then embedded in the 10% estimate might be some kind of assumption about a “background” autism rate; but mostly because I didn’t believe it. All of the genetic disorders cited above (and I must confess, I’d never heard of Potocki-Lupski syndrome before) represent a vanishingly small portion of the total autism population. Although I’m sure my sample isn’t representative, I’ve neither met a family with one of these kids nor have I even heard of one. And I’ve certainly never read anything, anywhere that was remotely supportive of this claim of genetic causation: 10% down, only 90% left to go.
So I asked myself, what was the Original Source Of The Estimated Number (for brevity’s sake, let’s call it the OSOTEN)? I wracked my brain, trying to think of where I might have run across it before.
I remembered I had heard the 10% number once before. In a talk at last year’s Autism One conference, Dr. Sue Swedo of the National Institutes of Health made a presentation in which she mentioned the 10% statistic on one of her slides. She wasn’t the source, I knew that. But, I’ve met Sue on a few occasions and like her (anyone who chokes up when showing a video of an autistic child has her heart in the right place), so I sent her an e-mail asking her for the OSOTEN. She never got back to me.
So I was beginning to wonder if this particular OSOTEN might not be one of these “information cascades” in action: a statement made once that then undergoes “degenerative repetition” even though it may have no basis in fact. I resolved to follow the citation trail in the scientific papers to see if I could find the elusive OSOTEN.
As scientific papers are supposed to do, Weiss et al provided a citation to support their statement. Could this be the OSOTEN? The citation was to a 2007 review paper by Dr. Christine Freitag of Hamburg, Germany, entitled “The genetics of autistic disorders and its clinical relevance: a review of the literature.” I got a copy and printed it out, hoping this paper might be the OSOTEN. Reading through Freitag’s review, I came to the section that was probably the source for Weiss et al’s claim. It read as follows:
It is generally agreed that about 10-15% of individuals with AD have a known medical condition that causes the disorder.
“Generally agreed” she says. By whom? I wondered, and found no further evidence in Freitag’s paper, a review that in all other respects was like a dozen others that I’ve read over the years. Clearly, this was not the OSOTEN. But like a good scientist, she also provided a specific citation for this claim. This time the potential OSOTEN was a 2001 review paper by Susan Folstein and Beth Rosen-Sheidley entitled, “Genetics of autism: complex aetiology for a heterogenous disorder.”
So my search for the OSOTEN continued to this new paper. And lo and behold, about halfway into the paper I found the critical passage at the start of a section called “Known medical conditions.” It read as follows:
It is generally agreed that ~10-15% of individuals with autism have an identifiable Mendelian condition or genetic syndrome (including chromosome abnormalities).
The discussion that followed proceeded through a list of genetic disorders often associated with autism, such as Fragile X syndrome, neurofibromatosis, tuberous sclerosis and Rett syndrome (it’s important to note that the first three of these genetic disorders are actually autism susceptibility genes; you can have the disorder and not have autism). There was no further source cited for the 10-15% estimate to which everyone “generally agreed.” Despite the fact that the “chain of collective reasoning” that ends at Weiss et al and starts with Folstein reveals deep inconsistencies at each step in the chain, perhaps, I thought, the Folstein statement was the elusive OSOTEN I had been looking for.
Or perhaps not (if anyone can provide a better original source with a more credible analysis supporting the 10% claim, I’d love to see it).
So although this comment from Folstein--the end of at least this trail of OSOTEN candidates--was clearly upstream from Weiss et al I was struck in particular by two things. First of all, the list of disorders was entirely different in the two papers, with only Fragile X syndrome and tuberous sclerosis common to both lists. But second, and most notably, the source statement for this incredibly important information cascade was completely unsupported. There was no table, no analysis, no method, and—most importantly—no further citation to another possible OSOTEN. Was this was the kind of outcome that substantiated Rzhetsky’s most pessimistic scenario: an incorrect statement that is repeated multiple times until an error is “generally agreed” to be true and never tested again?
In terms of basic logic and science, this sort of error certainly ought to be self-correcting. Increaingly, in the modern world of autism that most of us see, these kinds of authors' claims, upon close examination, just look silly. But we are surrounded by so many of them (from “we don’t know if there’s a real increase” to “we do know that much of the increase come from better diagnosing” to “autism is one of the most highly heritable disorders”) that it’s hard to know how to trace down every incorrect OSOTEN, determine whether or not it should be “generally accepted” or not, and find some rational way to deal with this truly bizarre form of collective scientific irrationality. And when unsupported claims like Folstein’s find their way uncritically into prestigious journals like The New England Journal of Medicine, they become quite serious. Make no mistake; information cascades like these become serious obstacles to the kind of science we need to help our children today. And too many of them are not self-correcting.
I’d like to think that Rzhetsky’s pessimistic model won’t really bear out in the long run. But when you read silliness like this, it just makes you shake your head and wonder, what are all these people thinking?
Mark Blaxill is Editor at Large of Age of Autism He also gratefully acknowledges the original source for the idea behind this essay, a wonderful book by Robert Merton titled On the Shoulders of Giants (affectionately known to its admiring readers as OTSOG).