By John Stone
Later this week the United Kingdom National Health Service is scheduled to publish a new set of adult psychiatric morbidity statistics, including autism, based on data collected in 2014. Below is the text of my response to Prof Traolach Brugha lead author of the 2009 UK adult autism survey and subsequent connected papers taken down from Pubmed Commons, after a complaint, three days ago. Clearly these are issues which deserve discussion and belong in the public domain. We would still be grateful to Prof Brugha if he would respond here if not on Pubmed. In an earlier reply to me on Pubmed Prof Brugha wrote:
Interested readers will find more information on the basis for the scoring of the diagnostic autism test ADOS-4 and evaluation of its use in a validation study published since (Brugha et al 2012). This additional research was required because the original scoring of the diagnostic autism test ADOS-4, at that time, had only been tested in teen age and young adult patients attending specialist US autism clinics; it had never been validated in the survey population, a random sample of adults and elderly people living in private households throughout England. The resulting validated scoring was different. We plan to publish more details of this.
This discloses the fundamental problem that Brugha and colleagues failed to use a standard method for diagnosing the “cases” in the survey (although they originally claimed they did) and have no real way of validating the method used a whole decade later. Here Brugha also introduces the red-herring that the diagnostic tool was developed in the US (though long in use in the UK as well), although it follows a letter in which they contest that their evidence has shown that autism is not only not rising in the UK but also not rising in the US. The 2009 survey followed on a decade of inflated estimates by the UK National Autistic Society and the Department of Health of the numbers of adult autistic people in the population at large. The political agenda behind this was manifest in 2009 at the National Health Service launch of the survey when Sarah Boseley reported in the Guardian ‘Autism just as common in adults, so MMR is off the hook'.
I am grateful to Prof Brugha for his response of 23 August. The peer reviewers seem not to have noticed the missing the ADOS data? Two of three of the ADOS Module 4 scores are missing from the from dataset provided to the NHS Information Centre for Health and Social Care by NATCEN .
It seems inappropriate to claim a calibrated diagnostic instrument like the ADOS has been applied when it has not been.
The authors reported  that they applied Module 4 of the Autism Diagnostic Observation Schedule [the ADOS]. It seems not to have been followed.
The ADOS Module 4 provides for and requires the use of all three Module 4 scores in classifying each case. If the ADOS had been applied, the three Module 4 scores for each case would be in the dataset. They are not there.
The dataset contains only one of three of the ADOS scores required for each claimed “case”. Omitting two of the three inevitably includes as “cases” individuals whose scores were below the required thresholds of the ADOS.
I have not been able to find a statement by the authors that they abandoned two of the three Module 4 scores, nor have I been able to find an explanation. If there is one, perhaps Professor Brugha might be kind enough please to provide a reference by document, page and paragraphs of a publication by the authors reporting that deviation from the ADOS and the reasons for it?
Module 4 does not apply to lower functioning non verbally fluent cases and is not applicable for use in their diagnoses . Thus the study could not cover all adults across the autistic spectrum in any event.
Professor Brugha appears to proffer a reason for the change of the term “Asperger’s Syndrome” to “ASD” in 2009. He suggests it was done in 2009 because in 2013, [four years after the statistics were published in the UK], the American Psychiatric Association did not include the term in their diagnostic manual for America. The rest of the world however in 2016 appears to still to use “Asperger's Syndrome” which appears in ICD-10-CM Diagnosis Code F84.5.
And what of the 13 missing potential cases omitted without mention by the authors? The authors reported [after the event] that they relied solely upon the >=10 ADOS Module 4 cut-off . Regardless of their reasons the ADOS Module 4 was not applied yet the authors claimed to have applied it.
32 cases in the dataset are above the ADOS Module 4 combined score minimum cut-off. 13 omitted cases have a combined score meeting the cut-off of >=7 and <10. [For a verbally fluent case on the autistic spectrum the minimum combined score cut-off is >=7. For a verbally fluent case of autism the combined score cut-off is >=10.] .
These were not diagnoses using the ADOS and no claim to that effect should have been made.
Additionally what the authors reported was:
“The recommended threshold of a score of 10 or more on the phase two ADOS assesment was used to indicate a case of ASD.” 
The ADOS recommended combined score to qualify as a potential case was and remains >=7 for autistic spectrum disorder. The >=10 threshold is used to indicate a case of higher functioning autism.
From what Professor Brugha says here it is clear that the method of diagnosis he and his colleagues employed is unique and was unvalidated at the time [which is obviously the case]. He also thereby confirms it still has not been validated 10 years later. Aside from obvious potential author bias, in science researchers do not validate their own work. The lack of independent validation of the authors’ methods is not corrected by publication of a paper by the same authors claiming the method was valid.
Scientific peer review occurs when many other teams of researchers attempt to repeat and reproduce the same work independently thereby either supporting or falsifying it.
Journal “peer review” is not scientific peer review. It does not validate authors’ claims. It is a modern development of questioned value including amongst the ranks of leading journal editors: “the practice of peer review is based on faith in its effects, rather than on facts.” . It is effectively a method of vetting papers for publication, whether valid or not.
This also by implication confirms the results are not now and were not then comparable to the internationally accepted methods of diagnoses employing over two decades diagnostic instruments including the ADOS. As the statistics are not comparable to any other diagnoses under the ADOS, they do not seem to be support for the proposition that “The proportion of true cases of autism is not changing”.
It is unsurprising therefore that the UK Statistics Authority’s May 2016 Assessment Report of these statistics  stated [in a diplomatically British way]:
“…. HSCIC might have helpfully labelled these statistics as ‘experimental’. …. HSCIC might consider ….. whether it is appropriate to label the statistics as ‘experimental’ until the fitness for purpose and robustness of the outcomes can be determined ...”
Even if the ADOS had been followed, which it had not been, Module 4 cannot be used for diagnosis across the full autistic spectrum. Module 4 is for verbally fluent adults and adolescents. It is not for those not verbally fluent. The study by design could not identify the full range of Autistic Spectrum Disorder [ASD] but only higher functioning cases which are primarily Asperger’s Syndrome.
A later study was commissioned, paid for out of public funds and published in 2012 to include lower functioning cases . The outcome was a different prevalence figure. Accordingly, the proposition that “The proportion of true cases of autism is not changing” is confounded as the proportion changed just by carrying out a further study.
The ADOS is also not a stand-alone diagnostic instrument so even if it had been followed, it would still not have been sufficient. Diagnosis requires use of the Autism Diagnostic Interview–Revised [ADI-R] and the ADOS algorithm criteria together with the application of clinical judgement . The ADI–R and the ADOS alone are less reliable in detecting more subtle cases of autism-spectrum condition.  The ADI–R provides a cutpoint for classic autism ,  which are predominantly non verbally fluent cases. Those are cases which the ADOS Module 4 does not address. Professor Brugha’s team’s efforts were and could only have been for verbally fluent higher functioning participants and not the full range of ASD.
And then there is the officially published “correction” about the scope of the study  which some may consider misleading:
“The initial Adult Psychiatric Morbidity Survey (APMS2007) report made reference to Asperger syndrome specifically, rather than Autistic Spectrum Disorder (ASD) generally. However, to be precise and as described in the Autism report Autism Spectrum Disorders in adults living in households throughout England - report from the Adult Psychiatric Morbidity Survey 2007, the screening tool used in phase one (the Autism Quotient) and the diagnostic assessment used in phase two (the Autism Diagnostic Observation Schedule) are both designed to pick up on the full range of ASD and were not able to differentiate between subtypes. We apologise if any confusion was caused.”
It is inappropriate for the authors to claim in the “correction” that the Autism Diagnostic Observation Schedule was used. It was not. That is apparent from the evidence already cited above and it even if it had been applied Module 4 is insufficient for the task.
It also seems inappropriate for the authors to claim in the “correction” that the Autism Quotient was used. It was not. The authors created their own short series of 20 questions and called it "AQ-20". It is not the Autism Quotient. It is not accepted nor recognised nor proven as a screening tool. The Autism Quotient in contrast is a set of 50 questions which is part of a series of diagnostic tools and instruments which have been calibrated to achieve consistency in identifying cases and in diagnoses.
In addition the authors confirmed AQ-20 was not fit for its purpose as a screening tool stating "The AQ-20 was only a weak predictor of ADOS-4 cases." . So not only was the Autism Quotient not used, what was used in its place did not work. That also appears to have had implications for weighting the resultant data.
The authors’ study seems also not "designed to pick up on the full range of ASD" as is apparent from the evidence already cited above.
The claim the ADOS is "not able to differentiate between subtypes" is inappropriate. The cut-off are used as an aid for that purpose. Had the ADOS been used appropriately, the 19 cases scoring >=10 would have been indicative of possible higher functioning autism cases. The other 13 cases scoring >=7 and <10 would have been indicative of possible Asperger’s Syndrome cases.
Of course, the ADOS on its own is not a standalone diagnostic tool in any event.
These appears to have been was no ethical approval granted for the original survey and interviews of adults to cover the full autistic spectrum. Ethical approval was sought expressly for and then granted for a study solely of adults with Asperger’s Syndrome .
An application to amend was made for an extension to include people with learning disability.  It appears that was not followed through and was not included in the published results. Accordingly, the published statistics were of adults with higher functioning autistic conditions and not across the ASD spectrum. It seems inappropriate for the authors to have suggested the contrary.
The later 2012 published study [referred to above] included lower functioning cases and the outcome was a different total prevalence figure across the two studies.
As the departures from the standard methods show this still leaves the original  study as not being of verbally fluent adults with Asperger’s Syndrome under the established methods - which methods include the application of the ADOS.
The published statistics appear to be, as Professor Brugha now confirms, an experimental study employing unvalidated methods, save for the authors’ claims after the event to have validated them in the same study in which they were used.
The authors are asked to provide please a description of the weighting scheme. There appears to be none published. The references Professor Brugha cites do not address the issue. If Professor Brugha believes that to be incorrect perhaps he might specify by document, page and paragraphs where such a description can be found. It would assist if the development of the weighting scheme was set out with a chronology of when and how it was developed.
The method for weighting of data should have been determined and set out in advance and not after the data has been collected. The failure of the “AQ-20” as a screening tool to predict the most likely cases will however have resulted in a predetermined weighting scheme being unlikely to be reliable. If the AQ-20 had been expected to identify about 80% of potential cases, it failed to achieve that. Weighting based on such an expectation would correct for the 20% not selected for interview. 618 participants were screened for interview using AQ-20 from approximately 7400. So an overall weighting factor above about 1.25 might be higher than would be expected in such a circumstance. Whilst the mean weighting was approximately 3.8, each case was allocated a specific weight given to a precision of two decimal places, totalling 71.64. No cases have the same weight aside from two and the range of weights is wide from 19.04 to 0.68: 19.04, 11.09, 6.31, 4.75, 3.85, 3.41, 3.33, 3.28, 2.22, 2.22, 2.17, 1.93, 1.55, 1.43, 1.3, 1.18, 1.07, 0.88, 0.68.
It also seems questionable ethically and lawfully to obtain sensitive personal medical data by telling participants that they are taking part in a general “health and wellbeing” survey when in reality they are being assessed for having specific mental health disorders. That appears to lack the necessary informed consent. It seems potentially not compliant also with legal requirements to protect sensitive personal data.
Professor Brugha appears quick to suggest there is no increase in ASDs based on a study of 19 cases – his own – which appears less than robust. And at the same time, whilst presenting the impression a diagnostic instrument, ADOS, was applied in that study, it appears that it was not. What was applied is a substantially different regimen which has not been validated by any other researchers. ADOS in contrast has been in extensive use internationally and has been developed and calibrated over many years. The argument that ADOS can be supplanted with a new unique and substantially untested experimental approach because ADOS has not been tested on UK adults seems unconvincing.
John Stone is UK Editor for Age of Autism.
 Freedom of Information Reference NIC-40939-Y2QCZ 17/3/2010
 eg. Page 16. Autism Spectrum Disorders in adults living in households throughout England Report from the Adult Psychiatric Morbidity Survey 2007 Section 2.2.2 Assessment of ASD. C Phase two assessment: Autism Diagnostic Observation Schedule (ADOS)
 page 208 “Algorithms and Use of the ADOS-G for Classification” of Lord et al  The Autism Diagnostic Observation Schedule–Generic: A Standard Measure of Social and Communication Deficits Associated with the Spectrum of Autism. Journal of Autism and Developmental Disorders, Vol. 30, No. 3, 2000.
 eg. Page 13. Autism Spectrum Disorders in adults living in households throughout England Report from the Adult Psychiatric Morbidity Survey. 2007: Chapter 2 Autism Spectrum Disorders. Summary Diagnostic.
 “Diagnostic Algorithms and Use of the ADOS-G for Classification” of Lord et al  The Autism Diagnostic Observation Schedule–Generic: A Standard Measure of Social and Communication Deficits Associated with the Spectrum of Autism. Journal of Autism and Developmental Disorders, Vol. 30, No. 3, 2000.
 Page 13. Autism Spectrum Disorders in adults living in households throughout England Report from the Adult Psychiatric Morbidity Survey. 2007: Chapter 2 Autism Spectrum Disorders. Summary.
 Dr Richard Smith*. (2006) Peer review: a flawed process at the heart of science and journals. J R Soc Med April 2006 vol. 99 no. 4 178-182 * former Editor-in-Chief BMJ
 Assessment Report 326 - National Study of Health and Wellbeing in England - Adult Psychiatric Morbidity Survey 23 May 2016
Accessed 22 Sept 2016
 Estimating the Prevalence of Autism Spectrum Conditions in Adults - Extending the 2007 Adult Psychiatric Morbidity Survey [NS]
Publication date: January 31, 2012
Accessed 22 Sept 2016
 Baron-Cohen et al (2009) Prevalence of autism-spectrum conditions: UK school-based population study. Page 502 The British Journal of Psychiatry (2009) 194, 500–509. doi: 10.1192/bjp.bp.108.059345
 ibid citing Lord C, Rutter M, Le Couteur A. Autism Diagnostic Interview – Revised: a revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. J Autism Dev Disord 1994; 24: 659–85.
 ibid citing Bishop DV, Norbury CF. Exploring the borderlands of autistic disorder and specific language impairment: a study using standardised diagnostic instruments. J Child Psychol Psychiatry 2002; 43: 917–29.
. NHS Digital - Adult Psychiatric Morbidity in England - 2007, Results of a household survey [NS] Publication date: January 27, 2009
Accessed 22 September 2016
 2012 Brugha et al Validating two survey methods for identifying cases of autism spectrum disorder among adults in the community Psychological Medicine, Cambridge University Press
 21/04/2006 letter NATCEN to Royal Free Hospital and Medical School Local Research Ethics with Research Ethics Application.
 NATCEN Email 04 June 2008 with the 4th Notice of Substantial Amendment and documents Amendment Request Ref 06/Q050 1/71