By John Stone
A widely publicized report purporting once again to disprove a link between the measles-mumps-rubella shot and autism has been shown to be irretrievably flawed within hours of publication. Launched two days ago in the British media amid headlines like ‘Autism just as common in adults, so MMR is off the hook’ (Guardian HERE) and ‘Autism rates back MMR jab safety’ (BBC HERE) the study was based on just 19 cases, who were never assessed according to accepted diagnostic criteria for autism, and included adults as young as 16 who would have received MMR vaccine anyway.
The study (HERE) – led by Professor Traolach (Terry) Brugha - was conducted by the University of Leicester and the prestigious Autism Research Centre of the University of Cambridge, director Professor Simon Baron-Cohen. Despite this, neither standardized psychometric testing for autism, or accepted epidemiological methods were employed in reaching its conclusions , and much of its methodology remains obscure. The Department of Health has long been under pressure to show that autism was just as prevalent as today before MMR vaccine was introduced in 1988. But rather than dispel such concerns the new report is more likely to support claims of a cover-up.
Given the seriousness of assessing the needs of an autistic population and what environmental or medical issues might be involved in fluctuations in incidence over a period - along with the fact that these questions have been the subject of intense public concern for more than a decade - the failure is manifest. Earlier this year Baron-Cohen expressed his opinion to Age of Autism that in time autism incidence in the present adult population would be shown to be in line with the 1 in 64 figure he had detected in Cambridgeshire school population in the first part of the current decade (Age of Autism HERE). The figures in the new study as stated are in line with the 1 in I00 figure usually accepted for the school population, but are anomalous in several ways. For instance, while the diagnostic criteria seem loose and even whimsical, all cases are “self-reporting” and therefore presumably rather able – the study contains no lower continuum cases. It is almost as if having found a means to “identify” 1 in 100 cases they stopped counting.
Once again officially approved data leaves us at sea. At the head of the present document we find the cryptic statement:
“This new set of statistics has not been formally assessed for compliance with the Code of Practice for Official Statistics. However, the Statistics Authority has agreed that, in view of the fact that the statistics are the product of secondary analysis of existing National Statistics, they can be designated as National Statistics.The producer body has confirmed that the new statistics are produced to the same standards as the existing ones.”
So, the information in this study has been enshrined as official truth.
Against this I note our local data (Haringey). A decade ago, when diagnostic practices were already very much as they are today we had about 1 ASD case coming through to adulthood every year (although the numbers were exploding in the under 11s and particularly the under 7s). The other day I met a friend who monitors these things who said they now come through at the rate of 20 or 30 a year, but next year it was going to be 53. This is real, and it has happened, moreover, in an era in which government was hostile to the expansion of Special Educational Needs. For all the many acrimonious tribunals they still could not keep the lid on it.
Appended are some expert comments on the new study which have been forwarded:-
Brief Comments/General Points
This is not a prevalence study.
The cornerstone of any ‘prevalence’ or ‘incidence’ study is case-definition. If we are to identify cases with confidence, and accurately differentiate cases from non-cases, the case definition must be clearly laid out, with inclusion and exclusion criteria explicitly stated.
A second major consideration in any study of population frequency is ascertainment; the process by which subjects are brought into the study. It is vital that the sample selected is representative of the population of interest and that those identified as cases within the sample are likely to represent the range of cases in the population with the disorder of interest.
A third consideration is the choice of measurement tools and methods by which information is gathered. Measurement tools must have known and well referenced psychometric properties in respect of the disorder of interest, and standardization rules with respect to the measurement tools must be followed. Any departure from the standardized administration or interpretation of instruments invalidates their use.
A fourth consideration is statistical power. If inferences are to be made about the statistical significance of within-sample differences (for example in age, sex) the sample size must be sufficiently large to provide adequate statistical power.
• There is no clearly stated case-definition anywhere in the report
• Inclusion / exclusion criteria are not clear
• Experts have NOT reached broad consensus on what constitutes the <diagnostic> category of ASD. In fact, as the authors point out, ASD does not exist as a diagnostic category
• All identified cases are based initially on self-report, from verbally fluent adults living in ordinary household accommodation: this excludes a large proportion of the population of interest
Derivation of the Instruments – Appendix C
• The authors state that: in order to meet data quality requirements in surveys: validity, reliability, portability, and to minimize respondent burden a self-report questionnaire was needed that could predict ASD caseness. It is by no means clear why such an instrument needs to be self-report.
• The reference given to Baron-Cohen et al. 2001 is to the AG-50. No references are provided for the AQ-20. One can only assume therefore, that no psychometric properties for the instrument are available. With respect to the referenced AQ-50 Baron-Cohen et al. (2001) conclude: The AQ is thus a valuable instrument for rapidly quantifying where any given individual is situated on the continuum from autism to normality. Its potential for screening for autism spectrum conditions in adults of normal intelligence remains to be fully explored.
• The AQ-50 is normed on a very selective population not representative of the range of ASD in the UK. Described thus in Baron-Cohen (2001): In this paper, we report on a new instrument to assess this: the Autism-Spectrum Quotient (AQ). Individuals score in the range 0-50. Four groups of subjects were assessed: Group 1: 58 adults with Asperger syndrome (AS) or high-functioning autism (HFA); Group 2: 174 randomly selected controls. Group 3: 840 students in Cambridge University; and Group 4: 16 winners of the UK Mathematics Olympiad
• The authors claim that ‘…the AQ has been shown in clinical populations to have a good correspondence with ASD diagnosis. ‘ but no reference is provided.
• The ‘modelling’ undertaken to identify AQ-20 items was not General Linear Modelling as stated. Logistic Regression is not GLM. In any case neither logistic Regression nor General Linear Modelling are not the correct procedures to use for item selection.
Identification of Cases
• This aspect of the report is substantially flawed. The authors refer to the fact that gathering data on ‘…both early development and on current day to day functioning over an extended period…’ is not practical. Diagnosis of ‘ASD’ (and therefore accurate estimates of prevalence) are simply not possible in the absence of this information
• As noted earlier, psychometric properties of the AQ-20 in predicting ASD have not been reported elsewhere. Further, it appears that the standardized scoring criteria for the ADOS were not followed in the identification of cases. The authors of the current report refer to a score of 10 or more as being indicative of an ASD. Whilst a score of 10 or more on the Communication and Reciprocal Social Interaction (RSI) sections is necessary for an ‘Autism’ categorisation, it is also necessary to score 3 or over on Communication alone and 6 or over on RSI. Someone scoring a total of 10 points on SRI and 0 on Communication, for example does not meet standardized criteria for an autism categorization.
• Further, the ADOS allows for a ‘spectrum’ categorisation based on a score of 2 or more on Communication, 4 or more on RSI and 7 or more in total. This was taken into account in developing the instrument for use in population studies, but is not taken account of in the main ‘study’.
Several references are either incorrect or misleading:
• On pg 16, the authors refer to the ADOS as an instrument used to collect information on adult functioning’. In fact the ADOS Module 4 was normed on only 70 adults, aged between 10 and 40 years. No standardization data is referenced for 40+ age ranges, which comprise a high proportion of the current sample.
• Again on pg 16 the authors refer to the ADOS ‘…rarely being used with older adults or in a general population’ then refer us to a study in which aspects of the ADOS algorithm for all modules was evaluated and replicated ‘…using data from children ages 18 months to 16 years’. Given that the current report is focused on 16+ years and on Module 4, this is misleading at best. They go on to state that an additional validation and quality assurance study was undertaken and is reported elsewhere, but they do not reference this.
• Additionally, the ADOS should not be used in isolation as a diagnostic instrument. It is not standardized as a stand-alone diagnostic tool and the diagnostic algorithm does not cover all aspects of the ASD triad.
Whilst the authors inform us that no statistically significant differences were generated between smaller range age-groups, they provide no data on this and do not address the issue of power. This is not sufficient justification to make a decision not to adjust for age.
In summary (so far!) the report has substantial flaws in terms of case-Definition, Ascertainment, Instrument Development, Identification of Cases and Statistical Analysis and several references (of those checked so far) are misleading.
General notes to add: the age range includes people who would have received MMR. Anyone born after 1987. We need more information on actual numbers. The total of 19 cases doesn’t appear to fit with the percentages based on unweighted bases.
This may be accounted for by missing data but if so, the issue of missing data should be clearly addressed.
It should surely strike the authors as being a little odd that on the basis of only HFA/AS adults we have a prevalence estimate of 1 in 100. What about all the less able people. Are they suggesting that prevalence has in fact decreased?
What would happen if this method was applied to the recently completed Cambridge prevalence study?
John Stone is UK Contributor for Age of Autism.