By John Stone
Later this week the United Kingdom National Health Service is scheduled to publish a new set of adult psychiatric morbidity statistics, including autism, based on data collected in 2014. Below is the text of my response to Prof Traolach Brugha lead author of the 2009 UK adult autism survey and subsequent connected papers taken down from Pubmed Commons, after a complaint, three days ago. Clearly these are issues which deserve discussion and belong in the public domain. We would still be grateful to Prof Brugha if he would respond here if not on Pubmed. In an earlier reply to me on Pubmed Prof Brugha wrote:
Interested readers will find more information on the basis for the scoring of the diagnostic autism test ADOS-4 and evaluation of its use in a validation study published since (Brugha et al 2012). This additional research was required because the original scoring of the diagnostic autism test ADOS-4, at that time, had only been tested in teen age and young adult patients attending specialist US autism clinics; it had never been validated in the survey population, a random sample of adults and elderly people living in private households throughout England. The resulting validated scoring was different. We plan to publish more details of this.
This discloses the fundamental problem that Brugha and colleagues failed to use a standard method for diagnosing the “cases” in the survey (although they originally claimed they did) and have no real way of validating the method used a whole decade later. Here Brugha also introduces the red-herring that the diagnostic tool was developed in the US (though long in use in the UK as well), although it follows a letter in which they contest that their evidence has shown that autism is not only not rising in the UK but also not rising in the US. The 2009 survey followed on a decade of inflated estimates by the UK National Autistic Society and the Department of Health of the numbers of adult autistic people in the population at large. The political agenda behind this was manifest in 2009 at the National Health Service launch of the survey when Sarah Boseley reported in the Guardian ‘Autism just as common in adults, so MMR is off the hook'.
I am grateful to Prof Brugha for his response of 23 August. The peer reviewers seem not to have noticed the missing the ADOS data? Two of three of the ADOS Module 4 scores are missing from the from dataset provided to the NHS Information Centre for Health and Social Care by NATCEN .
It seems inappropriate to claim a calibrated diagnostic instrument like the ADOS has been applied when it has not been.
The authors reported  that they applied Module 4 of the Autism Diagnostic Observation Schedule [the ADOS]. It seems not to have been followed.
The ADOS Module 4 provides for and requires the use of all three Module 4 scores in classifying each case. If the ADOS had been applied, the three Module 4 scores for each case would be in the dataset. They are not there.
The dataset contains only one of three of the ADOS scores required for each claimed “case”. Omitting two of the three inevitably includes as “cases” individuals whose scores were below the required thresholds of the ADOS.
I have not been able to find a statement by the authors that they abandoned two of the three Module 4 scores, nor have I been able to find an explanation. If there is one, perhaps Professor Brugha might be kind enough please to provide a reference by document, page and paragraphs of a publication by the authors reporting that deviation from the ADOS and the reasons for it?
Module 4 does not apply to lower functioning non verbally fluent cases and is not applicable for use in their diagnoses . Thus the study could not cover all adults across the autistic spectrum in any event.
Professor Brugha appears to proffer a reason for the change of the term “Asperger’s Syndrome” to “ASD” in 2009. He suggests it was done in 2009 because in 2013, [four years after the statistics were published in the UK], the American Psychiatric Association did not include the term in their diagnostic manual for America. The rest of the world however in 2016 appears to still to use “Asperger's Syndrome” which appears in ICD-10-CM Diagnosis Code F84.5.
And what of the 13 missing potential cases omitted without mention by the authors? The authors reported [after the event] that they relied solely upon the >=10 ADOS Module 4 cut-off . Regardless of their reasons the ADOS Module 4 was not applied yet the authors claimed to have applied it.
32 cases in the dataset are above the ADOS Module 4 combined score minimum cut-off. 13 omitted cases have a combined score meeting the cut-off of >=7 and <10. [For a verbally fluent case on the autistic spectrum the minimum combined score cut-off is >=7. For a verbally fluent case of autism the combined score cut-off is >=10.] .
These were not diagnoses using the ADOS and no claim to that effect should have been made.
Additionally what the authors reported was:
“The recommended threshold of a score of 10 or more on the phase two ADOS assesment was used to indicate a case of ASD.” 
The ADOS recommended combined score to qualify as a potential case was and remains >=7 for autistic spectrum disorder. The >=10 threshold is used to indicate a case of higher functioning autism.
From what Professor Brugha says here it is clear that the method of diagnosis he and his colleagues employed is unique and was unvalidated at the time [which is obviously the case]. He also thereby confirms it still has not been validated 10 years later. Aside from obvious potential author bias, in science researchers do not validate their own work. The lack of independent validation of the authors’ methods is not corrected by publication of a paper by the same authors claiming the method was valid.
Scientific peer review occurs when many other teams of researchers attempt to repeat and reproduce the same work independently thereby either supporting or falsifying it.
Journal “peer review” is not scientific peer review. It does not validate authors’ claims. It is a modern development of questioned value including amongst the ranks of leading journal editors: “the practice of peer review is based on faith in its effects, rather than on facts.” . It is effectively a method of vetting papers for publication, whether valid or not.
This also by implication confirms the results are not now and were not then comparable to the internationally accepted methods of diagnoses employing over two decades diagnostic instruments including the ADOS. As the statistics are not comparable to any other diagnoses under the ADOS, they do not seem to be support for the proposition that “The proportion of true cases of autism is not changing”.
It is unsurprising therefore that the UK Statistics Authority’s May 2016 Assessment Report of these statistics  stated [in a diplomatically British way]:
“…. HSCIC might have helpfully labelled these statistics as ‘experimental’. …. HSCIC might consider ….. whether it is appropriate to label the statistics as ‘experimental’ until the fitness for purpose and robustness of the outcomes can be determined ...”
Even if the ADOS had been followed, which it had not been, Module 4 cannot be used for diagnosis across the full autistic spectrum. Module 4 is for verbally fluent adults and adolescents. It is not for those not verbally fluent. The study by design could not identify the full range of Autistic Spectrum Disorder [ASD] but only higher functioning cases which are primarily Asperger’s Syndrome.
A later study was commissioned, paid for out of public funds and published in 2012 to include lower functioning cases . The outcome was a different prevalence figure. Accordingly, the proposition that “The proportion of true cases of autism is not changing” is confounded as the proportion changed just by carrying out a further study.