WHEN MAKING SCIENTIFIC MISTAKES IS A GOOD CAREER MOVE

By Mark Blaxill

A couple of weeks ago, I commented on the good work of Catherine DeSoto and Robert Hitlan who recently exposed a statistical error in a study conducted by researchers at the University of Hong Kong (click here to see the post.)

DeSoto and Hitlan examined a Chinese study that compared mercury in the blood and hair of autistic children with blood and hair mercury levels in typically developing children. And although the first published account reported no significant association between mercury exposure markers in autistic children, DeSoto and Hitlan showed that original authors’ calculations were simply wrong and that their data not only showed evidence of increased mercury exposures in the autistic children, it also showed evidence of impaired mercury excretion, a key element in the theory linking mercury exposure with autism risk.

Roger Brumback, the editor of the Journal of Child Neurology, in an accompanying editorial, commented on the problem of uncovering and exposing scientific mistakes. “One of the philosophical myths associated with science is that invalid information will be exposed by scientists who find that a particular study cannot be replicated. Unfortunately, this is rarely the case”, wrote Brumback, also a specialist in neuropathology at Creighton Medical Center. He theorized that this lack of follow up was partly due to lack of interest, noting that “nearly half of all articles published in the quality biomedical journals are never cited by another publication. Even for those articles that are read, interest in replicating the published results is lacking because there is no glory in being the second person to describe a particular phenomenon and research funding agencies (such as the US National Institutes of Health) do not provide grants to investigators wanting to confirm the results of other investigators.”

But the erroneous findings in the Hong Kong study were not overlooked by other scientists. In fact, as Mary Webster (one of AOA’s more astute readers) pointed out to us, the Hong Kong study has been cited frequently by authors who are hostile to the autism-mercury connection. In a note to Age of Autism editor Dan Olmsted, Webster documented nine citations to the flawed Hong Kong analysis, including two papers by Eric Fombonne denying any link between vaccines and autism and a paper by Paul Shattuck claiming that the rise in autism rates could be explained by “diagnostic substitution” (Note: I submitted a letter to Pediatrics, pointing out clear errors in Shattuck’s analysis; Pediatrics refused to publish it). In each citation, the Hong Kong study was on a short list of studies based on which the citing authors made assertions like this: “Biological studies have also not validated any thimerosal and autism link.”

Lack of interest in the autism-mercury connection? I think not. The consequences of the kind of scientific mistakes by the Hong Kong group are far graver than the more commonplace kinds of errors Brumback described. These are scientific mistakes that add confusion and uncertainty to an ongoing controversy. They become weapons in brutal and aggressive misinformation campaigns designed to nullify worrisome evidence, the scientific equivalent of the “Swift boat” tactics of the radical right.

All of which leads to a fascinating question. What, if any, will be the ripple effects of DeSoto and Hitlan’s discovery? Will Paul Shattuck’s Pediatrics study issue a correction to its reference list? Will Eric Fombonne go back to correct his claims regarding the evidence base on mercury and autism and note that the Hong Kong evidence actually lends support to the theory? Will the publicity and awareness of the error be received in the same way the erroneous result was disseminated? And, perhaps most important of all, will there be any negative consequences for the careers and scientific standing of the authors involved in making and promoting the mistake?

If there’s any evidence of accountability, it’s hard to observe from the outside. As for Virginia Wong, the corresponding author for the flawed study (whose “the dog-ate-my-homework” explanation to Brumback was simply that “quite a few results in the table were typed wrongly”), the consequences appear nonexistent. She remains in her post as the head of the Division of Child Neurology and Developmental Pediatrics at the University of Hong Kong. She remains on the editorial boards of seven pediatric neurology journals. Most surprisingly, she remains listed on the editorial board for the Journal of Child Neurology. And she has never, to my knowledge, publicly apologized for the error.

Should we be surprised at this lack of accountability for scientific error? Sadly, as long as a mistake supports the interests of the institutions that want the erroneous finding to be true, there may even be career benefits to scientists who try to refute discomforting theories. Even thought the study might turn out to be flawed, the authors were at least demonstrating that they were team players. And team players’ career interests are protected.

I was reminded of this career management issue earlier this month when a notable report crossed the news wires. The CDC announced a new $6 million “study to explore early development”, the “largest study ever to investigate risk factors of autism and other developmental disabilities.” As part of the announcement, the health maintenance organization (HMO) Kaiser Permanente announced their plans to begin enrolling families in California.

Buried in the Kaiser press release, however, was an intriguing quote from a Kaiser representative. "’We hope this study will help us learn more about the factors that may lead to autism and other developmental disabilities, and how genes and the environment may affect child development,’ said Lisa A. Croen, PhD, the study's local principal investigator and an epidemiologist with Kaiser Permanente's Division of Research in Oakland, Calif.”

When it comes to the top scientific mistakes in autism, one of Croen’s studies belongs on a very short list.  Croen was among the first of the epidemic deniers in autism, proposing in a 2002 article in the Journal of Autism and Developmental Disorders that the increase in the reported rate of California autism was due to “diagnostic substitution” and not a true increase. She implied that an increase in autism rates between 10 year olds (born in 1987) and 3 year olds (born in 1994) was offset by an equivalent decline in mental retardation rates. Eric Fombonne, the associate editor of the journal commented favorably on Croen’s “careful analysis” and echoed her suggestion that “changes in diagnostic practices are therefore likely to account for” the increase in autism. Like the Hong Kong study that Fombonne later cited, Croen provided a convenient finding for a medical and research establishment eager to reassure themselves that the autism epidemic could be explained away by a friendly epidemiologist.

But like the Hong Kong group’s mercury analysis, there was a problem with Croen’s diagnostic substitution result. It was wrong.

Croen made a simple but stunningly careless mistake. She assumed that registration rates of autism and mental retardation in the California services system would be the same for 10 year olds and 3 year olds. But that’s not simply wrong, it’s obviously wrong. Anyone familiar with autism epidemiology would know it takes a while to get an autism diagnosis (reports for median age at diagnosis range between 4 and 7 years of age). In Croen’s California data, the 3 year olds' autism rates (and mental retardation rates as well), would be severely underreported relative to the rates for 10 year olds, making Croen’s claims of offsetting trends in autism and mental retardation demonstrably false. More importantly, the theory of diagnostic substitution that she averred based on the trend data was clearly not supported by her data, it was instead an artifact of the errors in her interpretation.

Reading her study, I quickly realized the flaws in her assumptions. So I wrote and submitted an analysis in which I pointed out the mistake to the journal. Nearly a year later, and to Croen and the journal’s credit, Croen issued a re-analysis of their data alongside a publication of my critique. Croen’s reanalysis demonstrated quite conclusively that the autism epidemic in California could NOT be explained away by diagnostic substitution. And again to her credit, Croen’s commentary graciously acknowledged the validity of our criticism. But Fombonne never owned up to his own mistakes. And although Croen and Fombonne’s claims of diagnostic substitution have been refuted whenever a well designed study has been done, the specter of this and other epidemic denial hypotheses continues to rear its ugly head, most recently in the flawed Pediatrics study by Shattuck.

I have no particular axe to grind with Lisa Croen. But one would hope that her careless mistakes and epidemic denial biases might make her something less than the ideal candidate for a leadership role in the “largest ever” study designed to get to the bottom of the causes of autism. But five years later, Lisa Croen is back, working for the agency that many parents hold accountable for launching the epidemic to begin with, working under the Kaiser Permanente banner (the HMO that mysteriously “lost” the vaccine safety data that first revealed a correlation between mercury exposure and subtle brain damage in children) and has even assumed a role as the public face of the project.

The behavior of the CDC in the autism controversy continues to make you shake your head. Do they realize how much damage they’ve done to their reputation by their actions in autism? Do they have any idea what kind of changes would be required to make themselves credible in the eyes of the autism community? Do they really think it’s a good idea to make Lisa Croen the spokesperson for their “largest study ever” to find the cause of autism? Or do they simply not care at all?

It seems that in autism, government agencies like the CDC have a forgiving attitude when it comes to serious errors. Indeed, it seems that making scientific mistakes can even be good for your career. It certainly doesn’t seem to have reduced Lisa Croen’s opportunities. If you make mistakes of the right kind, even if you get caught, you can get a free pass back to the grant money machine. That’s what seems to happen when you take one for the team.

Mark Blaxill is editor-at-large of Age of Autism and Vice President of SafeMinds. He has authored or co-authored a number of peer-reviewed studies on autism as well as numerous SafeMinds commentaries. He lives in Cambridge MA with his wife and two children, one of whom was diagnosed with autism and has since made great strides towards a full recovery. In addition to his autism activities, he has had a long, distinguished business career.

Comments

[Managing Editor]

Dear PassionlessDrone, I only wish we had an award for commenter names. I gather you're at work? Thanks for commenting. Couldn't resist commenting myself on your choice of monikers.

Kim

[Kelli Ann Davis]

Mark,

Excellent analysis.

Regarding your points about the CDC – agreed. I brought this up to Dr. Raub a few months ago when I heard that they were in the midst of yet “another” study. I was suggesting that BEFORE they embark on another flawed study, MAYBE they ought to get some stakeholders from the community at the table to help DESIGN them first. After all, none of these studies are ever going to be accepted as “legitimate” considering who’s doing them. Money down the drain, in my opinion.

When I was at the IACC meeting on Friday, I was talking with Marshalyn Yeargin-Allsopp (CDC) about this very issue. She said it wasn’t going to matter anyways because they (CDC) were short of money considering all they got from CAA was a million dollars and this was going to limit their ability to do studies.

I’m like, “Yeah….no kidding. We made SURE the language was written that way for a reason.” I told another parent what I said that and she started laughing. She couldn’t believe I told her the truth.

One glimmer of hope – seems the IACC is going to take the research suggestions from the IOM Environmental Workshop (which occurred in April) and use them as a base for environmental research in regards to the IACC matrix. A “springboard” if you will.

Kelli

[passionlessDrone]

Hi Mark -

I think it is important to acknowledge that the biomedical treatment community has its share of questionable conclusions out there as well.

By way of example, I've read a critique of a paper on which I believe you are a co-author, Reduced levels of mercury in first baby haircuts of autistic children. The abstract states that:

"Hair mercury levels in the autistic group were 0.47 ppm versus 3.63 ppm in controls, a significant difference. "

Which is fine and well, but a very large study seemed to find very, very different values for the normal population. This study, Hair Mercury Levels in U.S. Children and Women of Childbearing Age: Reference Range Data from NHANES 1999–2000 indicates that in a group of 800+ children and 1700+ women, the levels were much, MUCH lower.

"Geometric mean (standard error of the geometric mean) hair mercury was 0.12 μg/g (0.01 μg/g) in children, and 0.20 μg/g (0.02 μg/g) in women"

It would seem, the control group used in your paper had mean mercury levels many, many times higher than what has been found in the past. I'm curious if you have any idea on the source of this apparent discrepancy?

Don't get me wrong, I am an ardent believer in biomedical treatments for autism, mainly due to what I see with my own eyes; but it can get to be a bit discouraging to see papers that seem to describe what I have experienced taken apart so simply by evaluating other studies.

Take care!

- pD