Substantive findings
In order to learn about selection bias on ID in autism research, we sampled all papers published in the most cited specialist autism journals in 2016. First, the review found considerable evidence of selection bias on ID: participants with ID were under-recruited to all sub-fields of autism research. Second, the reporting of participant characteristics was generally poor, with information about intellectual ability and/or ID often absent. There was a limited discussion of this, and forward citation chasing showed that citing authors did not typically acknowledge bias.
Our findings are consistent with the review that focused on brain imaging studies of autism, which found people with ID were under-represented in neuroimaging studies of autism [2]. It also tallies with individual reports: for example, the US National Database of Autism Research has 47,400 participants, but only 11% have borderline ID or ID (IQ < 85) [24]. Our finding is important because the validity of findings in individual studies is grounded on the assumption that the recruitment of study participants is not overly influenced by selection bias.
For studies using samples without any ID participants, forward citation chasing showed later papers typically cited them to describe knowledge about the whole autism spectrum, without taking sample profiles into account. Selection bias in participant characteristics might mean much of the information we have about autism may not be generalisable to all those with autism. Forward citation chasing illustrated how ‘facts’ travel based on titles and abstracts alone. Only a minority of studies displaying bias mentioned lack of generalisability as a limitation, none referred to sub-samples in their titles. Lack of generalisability was dealt with by some studies in their limitations sections (e.g. [25,26,27,28]) but sometimes in a cursory way, although others were examples of best practice. We recommend that if a No ID sample is used, this should be clearly stated in the title and abstract.
Barriers to participation, and solutions
Selection bias on intellectual disability may occur in autism research because children and adults with more severe difficulties are harder to recruit and retain in research studies and, similarly, that families of children with ID are harder to recruit as they may have less time and resources [29]. There is also an issue around mental capacity to consent to take part in research for some severely affected individuals with ID, so for ethical reasons, severely impaired individuals are excluded. ID is also a spectrum, however, and the lack of capacity to consent to research does not apply to everyone with ID. Another reason for bias is that the field is lacking good instruments to study people with intellectual disabilities [30].
Many research studies in the field of ID have examined strategies to make research more inclusive [7, 31,32,33,34,35], and there are reports and guides full of practical strategies and references to other work [36,37,38]. Two useful systematic reviews [39, 40] describe the barriers to cognitively disabled individual participation in trials and public health, and another review summarises the last 20 years of research in this area [7]. Such work concludes that strategies to counter selection bias on ID require more time, effort and funding to be put into recruitment. Strategies include one-to-one meetings with participants to explain study aims, enrolment and protocols [31]; piloting and adaptation of measures and working closely with gatekeepers (like service providers) to recruit potential participants. Recruitment phases for populations with ID should be longer than other populations, with researchers prepared to make home visits and visit after hours to allow for time constraints of participants and carers [32]. Being kept up-to-date with the progress of the research projects was found to be an important part of engendering a sense of personal benefit for participants [38].
Many carers experience overwhelming workloads and exhaustion [41]. Carers (most often women [42]) often have little time available for research or resources to support participation in research activities. Respite care for participants to give carers time off could be a strong motivation to enable participation. Participants and intermediaries need motivators to participate in research. A small payment to cover time and travel can help, although can be viewed as coercive [33]. People with ID living independently may also be recruited through peer networks or advocacy movements (e.g. the ‘Academic Autism Spectrum Partnership in Research and Education’ involves self-advocates). A strong self-advocacy movement has been identified as one of the conditions necessary for inclusive research to flourish [43].
Obtaining consent for participants with ID can be challenging and may require another person to give consent on their behalf. This means going through gatekeepers at services organisations, which should start at the highest level [32]. Identification of key workers within the organisations who may become allies to assist researchers in recruitment is a helpful strategy [32].
There is a sub-group of autistic individuals who may not be fluent verbally but are very able with technology to express their needs and wants. Measures to allow them access to research are developing rapidly as technology improves, for example measuring diet through participants’ mobile photos of their food [44]. There are many more people on the spectrum with ID who do not speak but can use such devices and who can, therefore, make their consent and needs for participating known. Although many measures are not currently designed for the population with very severe ID (such as standard IQ tests), there are creative ways to adapt measures and replicate similar constructs but using non-verbal tasks, such as tests of spatial intelligence or Adaptive Behaviour Scales [45].
Impact of bias by ID
One consequence of the bias encountered is the lack of an evidence base for effectiveness or otherwise of interventions, which have not specifically been trialled in autistic individuals with intellectual disability. This has implications for the introduction of a medication or a specific behavioural therapy or new service. Forward citation chasing indicated that cognitive, psychological and neuroscientific models of autism may be misapplied to the ID group in ongoing research, when in reality, there is no strong evidence base to support their application. Similarly, a diagnostic test that has not been adequately tested on the full population may under-identify the autistic population with ID.
Not only does selection bias lead to inappropriate generalisation, but it may shift the boundaries around who is included for an autism diagnosis. The evidence base about a particular diagnostic category feeds back into what we understand about it via revisions to diagnostic criteria. Thus, the parameters of the category itself are shifted if the research base predominantly contains persons of one type. Autism is a case in point of such ‘looping’, as it was once understood to be a condition affecting very severely impaired children, but is now increasingly associated with non-ID individuals, including adults. Conceptually, autism can be thought of as a collection of multi-dimensional traits that interact with each other and the environment, and these traits may alter with development. They are bounded together as a diagnostic category based on the best current evidence but also bounded for historical, pragmatic and political reasons [46, 47]. If autism research is predominantly drawn from studies of individuals without ID, this will feedback into the evidence base and remodel how we understand ‘autism’. This argument is applicable to all psychiatric diagnoses, with our findings specifically drawing attention where intellectual disability is comorbid [48]. ADHD, psychotic disorder, anxiety disorder and depression all co-occur with ID. Future research could examine if such a bias exists more widely across other psychiatric classes.
Strengths and limitations
Our review has a number of limitations. First, not all autism research was sampled; we are aware that the studies that include individuals with autism might also appear in a variety of other types of journals that are not specific to autism. These include the highest impact journals (e.g. Nature). A systematic review was impractical as an initial scoping review threw up 30,000 hits. Therefore, we simply did not have the capacity to mount a systematic review that was an exhaustive study of all papers relating to autism. This would have included the journals that are specifically about ID (which we would expect to be more balanced for ID and perhaps more transparent in reporting).
A number of studies in our review excluded participants with ID, even though this was not apparent from title screening. Morett et al. [49], for example, operated an exclusion cut-off at 1 SD below mean IQ (IQ < 85). However, even when we removed all studies that deliberately excluded participants with ID, our sensitivity analysis still revealed considerable bias, estimating a large preponderance of autistic participants without ID.
In 2013, the Diagnostic and Statistical Manual Volume 5 (DSM-5) unified the diagnoses of autistic disorder, Asperger’s disorder and Pervasive Developmental Disorder Not Otherwise Specified (PDD-NOS) into ‘autism spectrum disorder’. Our findings may partly reflect the response of the research community to this change. It could be that selection bias towards autistic individuals with an IQ in the normal range is a relatively recent phenomenon, as historically, estimates of population of autistic people with ID were higher. Here, we were interested in sampling the up-to-date (post-DSM-5) picture. Further, it was not possible to report more detail on bias by IQ or by severity of ID as a continuous measure as not enough studies provided data. Nevertheless, this review is the first (we know of) to comprehensively examine a cross-section of autism research for bias by ID in a large sample.
Recommendations
Our recommendation is to address barriers to research participation for the ID group. We agree with Mulhall et al. that studies that have inclusive strategies for recruitment should become a priority for research funders along with that development of inclusive measures [40]. Researchers and academics can help by recommending grants for funding only where broad inclusion strategies to include participants with ID have been considered or are in place. We suggest that exclusionary recruitment strategies are acceptable, and may be necessary for pragmatic reasons, but should be reported in a transparent way in abstracts and titles and justified carefully in funding applications, explaining why the protocol cannot be modified to include those with ID.
Conclusion
Our study throws attention on a gap in the research literature for autistic people with ID, but the point could be extended to other hard-to-reach populations, such as people with complex comorbid mental health conditions, severe ID or minimally verbal participants. Future research might investigate whether such groups may be largely absent from the evidence base for autism but also neurodevelopmental conditions more widely, suggesting what we know about a condition may largely reflect groups who are easier to access.
The reporting evidenced in our study would be unproblematic if all autistic people shared the same neurological profiles, genetic predisposition, aetiological pathways, cognitive and perceptual differences and responses to intervention, but recent evidence suggests this may not be the case [3, 5, 50]: it is difficult to find a ‘signal’ that is specific to one ‘autism’. There are real-world implications if selection bias on ID occurs when participants’ intellectual ability is associated with the phenomena under study. This may be the case for studies of intervention, aetiology, service use, diagnosis, neuroscience and psychology/cognitive functioning: all the sub-fields examined in this review.
