Selection bias on intellectual ability in autism research: a cross-sectional review and meta-analysis

Sampling strategy

Figure 1 illustrates the sampling strategy for studies. To sample a representative cross-section of strong and influential autism research studies, we reviewed all studies published in autism-specific journals with an impact factor higher than 3 between 1 January 2016 and 31 December 2016 (dates for printed, as opposed to online publication). The journals that met these criteria were Molecular Autism, Autism Research, Journal of Autism and Developmental Disorders, and Autism: International Journal of Research and Practice. All studies published in these journals were eligible for our sample. The methodology was adapted from Lee and colleagues [13]: like them, we developed a sampling protocol that is easily replicable. We wanted to sample the literature at one time point as estimates of the proportion of autistic individuals with ID have shifted over time. A cross-sectional method of review (over 1 year) was therefore best placed to test our hypothesis.

It is important to emphasise that we did not set out to undertake a full systematic review, although we utilised systematic review methodology in terms of screening titles, data extraction techniques and meta-analysis to investigate the presence and extent of selection bias among research studies involving participants with autism.

Inclusion and exclusion criteria

To be eligible, papers needed to report empirical research, rather than commentaries, letters, editorials, reviews and errata. We included data-based short reports and original research articles that covered children or adults identified as having ASD. We aimed to provide a cross-section of research studies across the field where titles of published studies stated the population of interest was people on the autism spectrum. In order to test for selection bias, we defined population of interest according to that stated in article titles. Articles were therefore included where the titles referred to ‘autism’, ‘autism spectrum disorders’ or ‘autism spectrum conditions’, i.e. stated population of interest was the entire autism spectrum, and findings were applicable to this. We therefore excluded research studies whose titles stated the population of interest were sub-groups of the autism spectrum in terms of lower/higher cognitive functioning. Examples that were excluded had titles referring to ‘high-functioning’ or ‘minimally verbal’ groups as their findings only applied to these groups. Included articles were those where at least a proportion of the study sample was identified as having ASD, regardless of the study design. We excluded studies where part of the sample or the entire sample was in infancy (age below 2) because this is before autism can be reliably identified [14]. Also, IQ measurement is not possible in this group.

The process of screening for inclusion was carried out by two researchers (DE, RW), who independently screened titles and abstracts. Disagreements and uncertainties were resolved by a discussion at weekly meetings (with GR and TP) to resolve any uncertainties. For those records deemed relevant after the title and abstract screening, full-text articles were retrieved. Articles were stored as full-text references using Zotero software.

Data extraction

Data were extracted from 20 included studies by 4 researchers to develop a pilot data extraction form. The remaining articles were divided between 3 researchers (RW, DE and TP) who extracted data into the finalised bespoke data extraction form. Over 20% of studies were rated by 2 or more reviewers (n = 61). Inter-rater reliability was good to perfect: agreement values for Cohen’s kappa for primary extractor (DE) vs secondary extractor (RW) ranged from 0.72 to 1. The lower range was due to the lack of consensus over sub-field, as some studies belonged to multiple fields, data extractors sometimes coded studies under different sub-fields.

Articles were classified into six sub-fields based on the categories provided by 2008 US funding portfolio analysis (epidemiology; intervention studies; basic biology including genetics, psychology and neuroscience; social and services research; diagnostic scales/identification and screening). After calculating kappa, studies were classified into sub-field by consensus within the weekly team meetings. Data extracted from each published study included the total number of participants and number of participants on the autism spectrum, for the latter their mean age and age range (which we classified as covering children (age 2–10), adolescents, (age 11-18) adults (18+)), gender ratios, whether participants were minimally or non-verbal, exclusion criteria, sub-field of study, recruitment setting and method of case ascertainment.

Data recorded for each included study are presented in Additional file 1: Table S1. Additional file 2 gives a complete citation list.

Primary objective: assessing selection bias by ID

Secondary objective: assessing quality of reporting within and between studies

We wanted to find out about the quality of reporting as well as selection bias. We recorded the number of included studies that did not report any data on participants’ intellectual dis/ability according to the definition above. We also recorded where studies that showed selection bias on ID did not report a lack of generalisability as a limitation or mention sub-populations in their abstracts.

For included studies that did not contain any participants with intellectual disability (100% No ID, N = 78), forward citation chasing was carried out to establish whether later publications cited the studies as a source of knowledge about all people with autism or took the lack of generalisability into account. ‘Forward citation chasing’ refers to the method of finding articles that have cited a previously published work. Cited reference searching was carried out in Google Scholar, which indexes citing studies for each paper. The first three published citing articles were selected where more than three were available for one study. This method of tracking citations is simply another way of searching databases to find relevant sources and articles and has been used in multiple reviews (e.g. [19, 20]).

Selection bias on ID

The majority (82%) of the 165 studies that provided IQ/ID data showed selection bias against individuals with ID (Table 2) despite declaring the population of interest to be, and applying their findings to, the entire autism spectrum. Column 5 of Table 2 shows that an estimated 94% of all ASD participants from 165 studies that provided data did not have ID: the random pooled effect was 0.94 (95% CI 0.94–0.97). Pooling data from all studies where IQ was provided, the mean IQ was 93.8 (n = 136). Figure 2 shows the forest plot for studies in epidemiology, which was the field with the lowest bias. The random pooled effect (given as the effect size in Stata output) is illustrated by the dotted line in Fig. 2.

Field	Total N	Studies where over 75% of the sample had No ID		Studies where the entire sample had No ID		Estimated % of ASD participants without IDa (95% CI)
		N	%	N	%
Intervention	12	8	67	5	42	87 (68, 99)
Biology	24	22	92	14	58	97 (92, 1.00)
Epidemiology	8	4	50	2	25	84 (43, 100)
Psychology	78	68	87	45	57	97 (94, 99)
Social	23	8	80	9	39	87 (80, 93)
Diagnosis	20	19	83	4	20	84 (82, 94)
Overall	165	136	82	79	48	94 (91, 97)

^aCalculated using meta-analysis (i.e. weighted by N per study)

We also estimated the percentage of participants who were non- or minimally verbal using meta-analysis of the 50 studies that reported data: the random pooled effect was 0.98 (95% CI 0.93–1.0). Thus, we estimated that 94% of autistic participants in the included studies did not have ID (an estimated 6% had ID) and only 2% of participants were non- or minimally verbal. The random pooled effect size in sensitivity analysis removing studies that had deliberately excluded participants with ID was 0.89 (95% CI 0.84–0.94).

Reporting quality within and between studies

Out of 301 included studies, most (55%, n = 165) did provide ID data. A quarter, 25% (n = 77), of studies specifically excluded autistic participants with intellectual disability as defined by our hierarchy. Over a third, 38% (n = 114), did not report any data on ID status according to our measures or exclude participants with ID, although some of these did report on rarer measures of developmental delay, especially studies of younger children. Only a small proportion of included studies reported data on the proportion of ASD participants who were verbal or non-verbal/minimally verbal: just 50 (17%) of the 301 studies.

One hundred and thirty-six of the 165 included studies that did provide ID data showed selection bias against participants with ID (Table 2). The majority of these studies (52%) did not report sub-populations in their abstracts or lack of generalisability (n = 71). Only 31% mentioned a lack of generalisability or selection bias as a limitation to the study. There was a wide variation in quality in the way the lack of generalisability was presented for those that did refer to it, ranging from simply stating the participant characteristics, e.g. ‘we only looked at higher functioning children’ [22], to meticulous consideration of the potential for biases, ‘we acknowledge several limitations to this study. Data were collected through a specialist clinical diagnostic service for ASD, where adults are expected to have IQ in the normal range. Thus the sample does not reflect the ASD community as a whole’ [23].

Substantive findings

In order to learn about selection bias on ID in autism research, we sampled all papers published in the most cited specialist autism journals in 2016. First, the review found considerable evidence of selection bias on ID: participants with ID were under-recruited to all sub-fields of autism research. Second, the reporting of participant characteristics was generally poor, with information about intellectual ability and/or ID often absent. There was a limited discussion of this, and forward citation chasing showed that citing authors did not typically acknowledge bias.

Our findings are consistent with the review that focused on brain imaging studies of autism, which found people with ID were under-represented in neuroimaging studies of autism [2]. It also tallies with individual reports: for example, the US National Database of Autism Research has 47,400 participants, but only 11% have borderline ID or ID (IQ < 85) [24]. Our finding is important because the validity of findings in individual studies is grounded on the assumption that the recruitment of study participants is not overly influenced by selection bias.

For studies using samples without any ID participants, forward citation chasing showed later papers typically cited them to describe knowledge about the whole autism spectrum, without taking sample profiles into account. Selection bias in participant characteristics might mean much of the information we have about autism may not be generalisable to all those with autism. Forward citation chasing illustrated how ‘facts’ travel based on titles and abstracts alone. Only a minority of studies displaying bias mentioned lack of generalisability as a limitation, none referred to sub-samples in their titles. Lack of generalisability was dealt with by some studies in their limitations sections (e.g. [25–28]) but sometimes in a cursory way, although others were examples of best practice. We recommend that if a No ID sample is used, this should be clearly stated in the title and abstract.

Barriers to participation, and solutions

Selection bias on intellectual disability may occur in autism research because children and adults with more severe difficulties are harder to recruit and retain in research studies and, similarly, that families of children with ID are harder to recruit as they may have less time and resources [29]. There is also an issue around mental capacity to consent to take part in research for some severely affected individuals with ID, so for ethical reasons, severely impaired individuals are excluded. ID is also a spectrum, however, and the lack of capacity to consent to research does not apply to everyone with ID. Another reason for bias is that the field is lacking good instruments to study people with intellectual disabilities [30].

Many research studies in the field of ID have examined strategies to make research more inclusive [7, 31–35], and there are reports and guides full of practical strategies and references to other work [36–38]. Two useful systematic reviews [39, 40] describe the barriers to cognitively disabled individual participation in trials and public health, and another review summarises the last 20 years of research in this area [7]. Such work concludes that strategies to counter selection bias on ID require more time, effort and funding to be put into recruitment. Strategies include one-to-one meetings with participants to explain study aims, enrolment and protocols [31]; piloting and adaptation of measures and working closely with gatekeepers (like service providers) to recruit potential participants. Recruitment phases for populations with ID should be longer than other populations, with researchers prepared to make home visits and visit after hours to allow for time constraints of participants and carers [32]. Being kept up-to-date with the progress of the research projects was found to be an important part of engendering a sense of personal benefit for participants [38].

Many carers experience overwhelming workloads and exhaustion [41]. Carers (most often women [42]) often have little time available for research or resources to support participation in research activities. Respite care for participants to give carers time off could be a strong motivation to enable participation. Participants and intermediaries need motivators to participate in research. A small payment to cover time and travel can help, although can be viewed as coercive [33]. People with ID living independently may also be recruited through peer networks or advocacy movements (e.g. the ‘Academic Autism Spectrum Partnership in Research and Education’ involves self-advocates). A strong self-advocacy movement has been identified as one of the conditions necessary for inclusive research to flourish [43].

Obtaining consent for participants with ID can be challenging and may require another person to give consent on their behalf. This means going through gatekeepers at services organisations, which should start at the highest level [32]. Identification of key workers within the organisations who may become allies to assist researchers in recruitment is a helpful strategy [32].

There is a sub-group of autistic individuals who may not be fluent verbally but are very able with technology to express their needs and wants. Measures to allow them access to research are developing rapidly as technology improves, for example measuring diet through participants’ mobile photos of their food [44]. There are many more people on the spectrum with ID who do not speak but can use such devices and who can, therefore, make their consent and needs for participating known. Although many measures are not currently designed for the population with very severe ID (such as standard IQ tests), there are creative ways to adapt measures and replicate similar constructs but using non-verbal tasks, such as tests of spatial intelligence or Adaptive Behaviour Scales [45].

Impact of bias by ID

One consequence of the bias encountered is the lack of an evidence base for effectiveness or otherwise of interventions, which have not specifically been trialled in autistic individuals with intellectual disability. This has implications for the introduction of a medication or a specific behavioural therapy or new service. Forward citation chasing indicated that cognitive, psychological and neuroscientific models of autism may be misapplied to the ID group in ongoing research, when in reality, there is no strong evidence base to support their application. Similarly, a diagnostic test that has not been adequately tested on the full population may under-identify the autistic population with ID.

Not only does selection bias lead to inappropriate generalisation, but it may shift the boundaries around who is included for an autism diagnosis. The evidence base about a particular diagnostic category feeds back into what we understand about it via revisions to diagnostic criteria. Thus, the parameters of the category itself are shifted if the research base predominantly contains persons of one type. Autism is a case in point of such ‘looping’, as it was once understood to be a condition affecting very severely impaired children, but is now increasingly associated with non-ID individuals, including adults. Conceptually, autism can be thought of as a collection of multi-dimensional traits that interact with each other and the environment, and these traits may alter with development. They are bounded together as a diagnostic category based on the best current evidence but also bounded for historical, pragmatic and political reasons [46, 47]. If autism research is predominantly drawn from studies of individuals without ID, this will feedback into the evidence base and remodel how we understand ‘autism’. This argument is applicable to all psychiatric diagnoses, with our findings specifically drawing attention where intellectual disability is comorbid [48]. ADHD, psychotic disorder, anxiety disorder and depression all co-occur with ID. Future research could examine if such a bias exists more widely across other psychiatric classes.

Strengths and limitations

Our review has a number of limitations. First, not all autism research was sampled; we are aware that the studies that include individuals with autism might also appear in a variety of other types of journals that are not specific to autism. These include the highest impact journals (e.g. Nature). A systematic review was impractical as an initial scoping review threw up 30,000 hits. Therefore, we simply did not have the capacity to mount a systematic review that was an exhaustive study of all papers relating to autism. This would have included the journals that are specifically about ID (which we would expect to be more balanced for ID and perhaps more transparent in reporting).

A number of studies in our review excluded participants with ID, even though this was not apparent from title screening. Morett et al. [49], for example, operated an exclusion cut-off at 1 SD below mean IQ (IQ < 85). However, even when we removed all studies that deliberately excluded participants with ID, our sensitivity analysis still revealed considerable bias, estimating a large preponderance of autistic participants without ID.

In 2013, the Diagnostic and Statistical Manual Volume 5 (DSM-5) unified the diagnoses of autistic disorder, Asperger’s disorder and Pervasive Developmental Disorder Not Otherwise Specified (PDD-NOS) into ‘autism spectrum disorder’. Our findings may partly reflect the response of the research community to this change. It could be that selection bias towards autistic individuals with an IQ in the normal range is a relatively recent phenomenon, as historically, estimates of population of autistic people with ID were higher. Here, we were interested in sampling the up-to-date (post-DSM-5) picture. Further, it was not possible to report more detail on bias by IQ or by severity of ID as a continuous measure as not enough studies provided data. Nevertheless, this review is the first (we know of) to comprehensively examine a cross-section of autism research for bias by ID in a large sample.

Recommendations

Our recommendation is to address barriers to research participation for the ID group. We agree with Mulhall et al. that studies that have inclusive strategies for recruitment should become a priority for research funders along with that development of inclusive measures [40]. Researchers and academics can help by recommending grants for funding only where broad inclusion strategies to include participants with ID have been considered or are in place. We suggest that exclusionary recruitment strategies are acceptable, and may be necessary for pragmatic reasons, but should be reported in a transparent way in abstracts and titles and justified carefully in funding applications, explaining why the protocol cannot be modified to include those with ID.

Conclusion

Our study throws attention on a gap in the research literature for autistic people with ID, but the point could be extended to other hard-to-reach populations, such as people with complex comorbid mental health conditions, severe ID or minimally verbal participants. Future research might investigate whether such groups may be largely absent from the evidence base for autism but also neurodevelopmental conditions more widely, suggesting what we know about a condition may largely reflect groups who are easier to access.

The reporting evidenced in our study would be unproblematic if all autistic people shared the same neurological profiles, genetic predisposition, aetiological pathways, cognitive and perceptual differences and responses to intervention, but recent evidence suggests this may not be the case [3, 5, 50]: it is difficult to find a ‘signal’ that is specific to one ‘autism’. There are real-world implications if selection bias on ID occurs when participants’ intellectual ability is associated with the phenomena under study. This may be the case for studies of intervention, aetiology, service use, diagnosis, neuroscience and psychology/cognitive functioning: all the sub-fields examined in this review.