• Editorial
• Open Access

• Received: 20 November 2018
• Accepted: 23 November 2018
• Published: 22 February 2019

There was some controversial press reporting of an article published in Molecular Autism by Anwar et al. [1]. In addition, at least two organizations dedicated to scientific accuracy in reporting have commented on this article, including www.sciencemediacentre.org and www.healthnewsreview.org. While the authors expressed some of the caveats of the study in their article, the subsequent reporting in the press and non-specialist community omitted critical caveats and we feel compelled to alter our guidelines to better address such issues.

A particular concern is claims about a biological test being diagnostic. Countless publications have made the point that what is observed in a selected sample of cases and controls is very far from being a diagnostic test. Drs. Paul Meehl and Albert Rosen summarized many of the relevant issues as early as 1955 [2], in response to a 1954 white paper by the American Psychological Association [3]. In summary, findings of even very significant differences in a given measure between cases versus controls are not by themselves sufficient to support the use of such a measure for diagnosis. This is especially true for rare events because for “any attempt at prediction of infrequent behavior, a large number of false positives are obtained” [4]. Even with an incidence of around 1%, autism would be considered such a rare event.

Researchers focusing on diagnostics are generally aware of these issues and reviewing such issues is (or should be) part of any good training program for clinicians and for anyone involved in translational research. While there have been many papers over the ensuing 60 years that made these points, we will review them once more in the current context.

The important terms to remember are sensitivity and specificity. Sensitivity, in the context of autism diagnoses, is the proportion of individuals with autism that are correctly identified as having autism. Specificity, in contrast, is the proportion of typical individuals who are correctly identified as typical. The best classifier identified by Anwar et al. had a sensitivity of 0.92 and a specificity of 0.84.

As can be seen from the Table, in a more representative sample, using the same sensitivity and specificity measures but with a base rate of 1%, fully 95% (602 out of 637) of individuals in the population receiving an autism label based on the screen would not have autism. In addition, under both scenarios, 8% of individuals with autism would be misclassified.

There are some scenarios where such a high false-positive rate might be acceptable, for example, where the burden is small, the outcome dire, and immediate treatment with an effective intervention is critical to changing the outcome (e.g., neonatal screening for treatable metabolic disorders). However, it would be hard to argue for such a test in autism where we have reasonable tools for diagnosis and the effective interventions are both costly and can be administered over a larger temporal window.

Moreover, this remains a most optimistic scenario. For example, we have no idea how other conditions may test using this screen and the authors appropriately call for an “assessment of the specificity of the algorithms for autism versus other psychiatric conditions.” And note that for some measures in the Anwar et al. study [1], the numbers of typical and autism subjects were lower (e.g., 21 vs 27 for the metabolite that was associated with the best classifier).

In addition, the sensitivity and specificity were chosen as the best from multiple different algorithms and from multiple different tests, without formal correction for experiment-wide error, and replication of such findings is a sine qua non for advancing any potential screen or even any scientific findings. This is especially true for a condition that is as etiologically and clinically heterogeneous as autism. Again, the authors make this clear in their Conclusions, where they note: “For future studies, we suggest firstly validation of the current findings in an independent clinical study group.” With validation in an independent sample and prospective studies of, for example, high-risk individuals, together with studies in other neurodevelopmental conditions, we may begin to have another clue into the biology of autism. But likely not a diagnostic test.

This nuance and cautious reporting by the authors was not always well reflected in reports in the press. Here are two out of several examples: “Blood test which is 90% accurate could help to diagnose autism in children by detecting early warning signs” (Daily Mail); and, “The most reliable of the tests is better than any other diagnostic method available” (Newsweek).

Many of the relevant news reports do not include any feedback from experts. When they do, they are much more balanced. CNN, for example, noted that “experts caution that the tests are far from becoming available clinically and that more research needs to be done.” And they include opinions from individuals not involved in the study. Dr. James Cusack is quoted as saying that, “This (study) is weakened by a small sample size, possible overfitting of data and a lack of comparison groups,” and that, “This study does not tell us how effectively this measure can differentiate between autism and other neurodevelopmental or mental health conditions such as ADHD and anxiety.”

And CNN goes on to quote Dr. Max Davie, a spokesman for the Royal College of Paediatrics and Child Health: Dr. Davie notes that “This is a promising area; however, this is a very long way indeed from a test for autism,” and that, “It is important that it is not adopted with too much enthusiasm. If applied to a large population, it will produce large number of false positives, causing huge worry and potential harm to children and families.”

As scientists who have worked in the field of autism research for decades and have learned the importance of always putting families first, we need to protect against the risk of scientific inaccuracy by emphasizing limitations and caution, to maximize the accuracy of public summaries of any study. Molecular Autism prides itself on being open access and we hope that our papers are accessible to families and to non-experts. The upshot of this must be that the Editors-in-Chief, Associate Editors, reviewers, and BMC staff do everything we can to give an honest appraisal of impact in every paper, no matter how redundant the cautions may be.

As Editors, in relation to the Anwar et al. study [1], we should have insisted that any mention of diagnostics should have been minimized in the publication and the limitations be explicitly addressed. Although the authors and the expert reviewers were clearly aware of the limitations, we should have emphasized and re-emphasized them in the publication.

As a second link in the chain we note that, in many examples of press surrounding the study by Anwar et al. [1], the text in print and online news was taken from a university press release. There is evidence that a source of exaggeration of research findings in the news can result from exuberant press releases from both universities and journals [5, 6]. Importantly, these same studies show that neither exaggerated claims nor caveats included in press releases overly influence the likelihood of news coverage, so being more cautious does not come at any real cost to the authors, their universities, or to the journal. It is the responsibility of authors and editors to work with institutional and journal press offices and ensure that caveats are clear and exaggerated claims are minimized.

The third link in the chain is that of the news organizations themselves. Ideally, these organizations should carry out critical reporting on news stories, which includes getting feedback from experts not involved in the work. In some cases, this can be done with little effort. For example, the quotes from Drs. Cusack and Davie come directly from the Science Media Centre, a group that solicits expert comments in order to help journalists understand and cover stories accurately (http://www.sciencemediacentre.org/expert-reaction-to-blood-and-urine-biomarkers-for-autism/). The Science Media Centre sends these quotes directly to the press before the embargo is lifted and journalists often use these quotes in their reports.

Additionally, the American Association for the Advancement of Science has developed SciLine (https://www.sciline.org). SciLine connects scientists and journalists to help ensure that scientific principles and evidence are conveyed accurately and in proper perspective (the name SciLine blends “science” and “deadline,” to emphasize the need for rapid, accurate responses from scientist volunteers).

HealthNewsReview.Org dissects some health claims in the popular press and commented on a report in the Guardian press around Anwar et al. as well. They note that “the story lifts quotes and almost entire sentences from a news release with no apparent original reporting.” In addition they remark that: “What would have helped this story considerably would be interviewing sources not affiliated with the study to clarify the limitations of these biomarkers, explain how autism is currently diagnosed, and provide some much-needed context about the novelty, availability, and clinical relevance of the biomarkers studied” (https://www.healthnewsreview.org/review/guardian-trumpets-test-diagnose-autism-wheres-evidence/).

To their credit, in response to this criticism, the Guardian posted a few days later a note with a statement from the UK autism research charity, Autistica: “This is a small, early-stage study which may explain one biological difference in autism. At this stage, the results presented are not strong enough to suggest that this method could be used for the diagnosis of autism. For example, we don’t know whether this technique can distinguish autism from ADHD, anxiety or other similar conditions. There have been many previous attempts to develop a biological test for autism. Still, the best way to diagnose autism is through clinical interview and observation, which takes into account the many features of autism.”

We are therefore instituting some additional steps in our publication process to help ensure that we minimize the gap between what is presented in our journal and what is reported in the larger community.

With these steps (i.e., a section on Limitations, appropriate sample sizes, appropriate P value interpretation, effect size and sensitivity power analysis, replication and validation in an independent cohort, and attention to coincident reporting by author institutions), applied to all studies in Molecular Autism, we will ensure that we remain at the vanguard of important and impactful research in autism, while ensuring that reported findings and their potential impact are understandable to the broadest audience.