- Open Access
Genome-wide linkage analyses of two repetitive behavior phenotypes in Utah pedigrees with autism spectrum disorders
Molecular Autismvolume 1, Article number: 3 (2010)
It has been suggested that efforts to identify genetic risk markers of autism spectrum disorder (ASD) would benefit from the analysis of more narrowly defined ASD phenotypes. Previous research indicates that 'insistence on sameness' (IS) and 'repetitive sensory-motor actions' (RSMA) are two factors within the ASD 'repetitive and stereotyped behavior' domain. The primary aim of this study was to identify genetic risk markers of both factors to allow comparison of those markers with one another and with markers found in the same set of pedigrees using ASD diagnosis as the phenotype. Thus, we empirically addresses the possibilities that more narrowly defined phenotypes improve linkage analysis signals and that different narrowly defined phenotypes are associated with different loci. Secondary aims were to examine the correlates of IS and RSMA and to assess the heritability of both scales.
A genome-wide linkage analysis was conducted with a sample of 70 multiplex ASD pedigrees using IS and RSMA as phenotypes. Genotyping services were provided by the Center for Inherited Disease Research using the 6 K single nucleotide polymorphism linkage panel. Analysis was done using the multipoint linkage software program MCLINK, a Markov chain Monte Carlo (MCMC) method that allows for multilocus linkage analysis on large extended pedigrees.
Genome-wide significance was observed for IS at 2q37.1-q37.3 (dominant model heterogeneity lod score (hlod) 3.42) and for RSMA at 15q13.1-q14 (recessive model hlod 3.93). We found some linkage signals that overlapped and others that were not observed in our previous linkage analysis of the ASD phenotype in the same pedigrees, and regions varied in the range of phenotypes with which they were linked. A new finding with respect to IS was that it is positively associated with IQ if the IS-RSMA correlation is statistically controlled.
The finding that IS and RSMA are linked to different regions that only partially overlap regions previously identified with ASD as the phenotype supports the value of including multiple, narrowly defined phenotypes in ASD genetic research. Further, we replicated previous reports indicating that RSMA is more strongly associated than IS with measures of ASD severity.
Although it is generally accepted that genetic factors play a major role in the etiology of autism spectrum disorders (ASDs), identification of specific genetic risk markers is complicated by the phenotypic complexity of clinical diagnoses. For example, the Diagnostic and Statistical Manual of Mental Disorders 4thed. (DSM-IV) diagnostic criteria for autistic disorder (AD) require impairments in three domains: social interaction, communication and repetitive and stereotyped behavior. Each of these three domains has been shown to be heritable, but their covariation in the general population is modest, and genetic modeling suggests distinct genetic influences for each [3–5]. Thus, it has been argued that the ability to identify susceptibility loci for ASD would be increased if specific ASD/AD traits were used as phenotypes [3, 6].
Specific ASD/AD traits have been employed in genetic studies most often either to stratify pedigrees for linkage analysis or as the dependent variable in association tests for specific alleles. For example, the first approach has found stronger ASD linkage signals in pedigrees with more abnormal levels of phrased speech delay [7, 8], repetitive behavior [9–11] and savant skills , but there have been failures in replication . The second approach has resulted in significant genotype associations with repetitive behavior [14–16]. A third, less common approach has been to use the specific trait as a quantitative or qualitative phenotype in linkage analyses. For example, we used the Social Reciprocity Responsiveness Scale (SRS)  score as the phenotype in linkage analyses of multiplex ASD pedigrees (Coon et al., Genome-wide linkage using the Social Responsiveness Scale (SRS) in Utah autism pedigrees, submitted). Although each of these methods has merit, it should be noted that the first method attempts to reduce heterogeneity of the diagnostic phenotype by stratification on a specific trait, whereas the second and third approaches seek to identify risk markers for the trait itself.
Repetitive and stereotyped behavior is a promising candidate for further genetic study because it probably comprises at least two even more specific phenotypes that differ in their behavioral correlates, familiality, and relation to genetic linkage with ASD. The 'restricted and repetitive stereotyped behavior' (RRSB) domain of the Autism Diagnostic Interview--Revised (ADI-R) [18, 19] is a well-accepted measure of the repetitive behavior phenotype. To uncover the factor structure of RRSB, a variety of factor analytic techniques have been used with different subsets of RRSB items and with study populations that differ in ASD severity and ethnicity [11, 20–25]. Remarkably, in spite of their methodological differences, these analyses converge on a two-factor solution comprising 'repetitive sensory-motor actions' (RSMA) and 'insistence on sameness' (IS). RSMA items investigate repetitive physical mannerisms and unusual sensory interests, whereas IS items investigate compulsive behaviors. There are two exceptions to the common two-factor solution. First, an exploratory factor analysis of RRSB items  recovered essentially the same RSMA and IS factors but also found a third factor ('circumscribed interests'). This finding does not detract from the conclusion that RRSB comprises RSMA and IS, but rather suggests that RRSB may measure additional factors as well. Second, a principal components analysis of all ADI-R items identified six factors, including a 'compulsions' factor that contained some items from both the IS and RSMA factors, and a 'social intent' factor that combined social interaction items with the RSMA item of 'hand and finger mannerisms' . Despite this, the preponderance of statistical evidence indicates that RSMA and IS are distinct factors within the RRSB domain.
It is well established that IS and RSMA have different patterns of relationship with other ASD traits. Specifically, RSMA, but not IS, has been reported to be associated with lower IQ, less adaptive behavior, and later age of appearance of first words and phrases [6, 20, 21], which suggests that RSMA may be more correlated with ASD severity . These findings support the validity of treating IS and RSMA as different phenotypes.
There is more empirical support for a genetic effect on IS than on RSMA. Whereas modest evidence of familial concordance occurs for IS, no reported concordance occurs for RSMA [21, 25]. Thus, the IS factor may account for earlier findings that RRSB is familial [28, 29]. Indeed, Silverman et al.  reported that RRSB categories that include IS items were familial, whereas those that include RSMA items were not. Further, a linkage analysis across the 15q11-q13 region in a subset of families with the highest IS scores resulted in increased LOD scores for AD  over scores obtained without stratification. By contrast, stratification on RRSB or RSMA did not increase lod scores. Finally, obsessive compulsive disorder (OCD) features in parents were associated with IS, but not RSMA, in children with AD , which suggests that IS may be part of a broader autism phenotype of obsessive behavior.
We are not aware of previous genetic linkage studies with either IS or RSMA as the phenotype. The primary aim of the present study was to perform a genome-wide linkage analysis with both IS and RSMA as phenotypes using large extended ASD pedigrees. Thus, our goal was to identify genetic risk regions for IS and RSMA in ASD cases rather than to stratify on IS and RSMA to reduce ASD heterogeneity. Because IS and RSMA data were available only for ASD cases rather than for all pedigree members, we focused our analyses on these specific phenotypes in ASD cases and did not include clinically unaffected family members in this study. Signals obtained with these phenotypes were compared with those found in the same set of pedigrees using ASD diagnosis . Contrasting results obtained with IS and RSMA with those obtained by ASD categorical diagnosis addresses empirically the possibilities that more narrowly defined phenotypes improve linkage analysis signals, and that different narrowly defined phenotypes are associated with different loci. Secondary aims were to examine the correlates of IS and RSMA and to assess the heritability of both scales.
This study has ongoing approval from the University of Utah institutional review board (IRB). All adults participating in the research signed informed consent documents. All subjects under the age of 18 signed assent documents and their parents or guardians signed parental permission documents. These documents were approved by the University of Utah IRB.
Subjects were members of 70 pedigrees having at least two family members with ASD. In total, 653 subjects were genotyped, 192 of whom had a study diagnosis of ASD. Study diagnosis was based in almost all instances on both the ADI-R [18, 19] and the Autism Diagnostic Observation Schedule-Generic (ADOS-G) . These pedigrees were used in our recent genome-wide linkage analyses of ASD . All of the families studied are part of the Utah collection of multiplex ASD pedigrees. We did not include pedigrees from other collections or repositories. Additional sample characteristics including pedigree sizes, ascertainment and assessment methods were reported previously .
RSMA and IS scales
RSMA and IS scales were derived from the RRSB domain of the ADI-R, which was available for 183 subjects with a study diagnosis of ASD. RSMA and IS items were ADI-R items that reliably loaded on one scale or the other in previous factor analytic studies [11, 20–25]. For both scales, scores were the unweighted sum of ADI-R item 'ever' ratings of 0-3. We believe this method of scoring the two scales is less susceptible to chance inter-item correlations in our data than would be factor scales derived from our data alone. RSMA items included 'hand and finger mannerisms', 'unusual sensory interests', 'repetitive use of objects', 'complex mannerisms' and 'rocking'. IS items included 'difficulties with minor changes in personal routine or environment', 'resistance to trivial changes in environment' and 'compulsions/rituals'.
Items from the ADI-R ('age of first words' and 'age of first phrases') were used to assess language delay in ASD cases. For parents who indicated normal onset but who could not remember the exact ages, values were set to 23 months for words and 32 months for phrases (acquiring language after these ages is considered abnormal on the ADI-R). For parents who indicated delayed onset but could not remember the exact ages, values were set to 1.5 standard deviations above the mean. For subjects who never acquired language, values were set to 3 standard deviations above the mean.
IQ was measured in subjects with ASD using an assessment instrument appropriate for the subject's age and developmental level. IQ measures included the Wechsler Intelligence Scale for Children, 3rd revision (WISC-III) , the Wechsler Adult Intelligence Scale, 3rd revision (WAIS-III) , the Differential Abilities Scale (DAS)  and the Mullen Scales of Early Development .
The SRS is a quantitative measure of social ability ranging continuously from significantly impaired to above-average social abilities . Although the SRS can be used with a general population, in our study the SRS was used only with ASD cases. The SRS mannerisms scale, which contains items that measure stereotypical behaviors and restricted interests, was used to determine whether IS or RSMA was more highly associated with another accepted measure of repetitive behavior in ASD cases.
Genotyping services were provided by the Center for Inherited Disease Research (CIDR), using the 6 K single nucleotide polymorphism (SNP) linkage panel. Methods and quality control procedures have been described in detail previously . After quality control, there were genotypes from 6,044 SNPs on 653 pedigree members who were members of 67 informative families. Eliminating linkage disequilibrium (LD) between markers in linkage studies has been strongly recommended, as false-positive results can occur in the presence of LD, particularly with extended multigenerational pedigrees for which ancestral genotypes are unavailable . Recommended thresholds of acceptable LD vary, but a pair-wise r2 value of 0.05 between SNPs has been supported with extensive simulation studies . Therefore, before linkage analysis, we screened SNPs for LD using the PLINK software package , which recursively removes SNPs within a sliding window. We set a window size of 50 SNPs, shifted the window by 5 SNPs at each step, and used a variance inflation factor (VIF) of 1.5, which is equivalent to an r2 of 0.33 regressed simultaneously over all SNPs in the selected window. This r2 considers not only the correlations between SNPs but also between linear combinations of SNPs , and corresponds in our data to a pair-wise r2 value of approximately 0.05. This screening for LD removed 1,207 SNPs. As part of the validation procedure, we also removed 115 SNPs with a minor allele frequency < 0.10 and 4 SNPs that were not in Hardy-Weinberg equilibrium (standard 1 degree of freedom test failed at the 0.05 level). The total number of SNPs left after this phase was 4,718.
The heritability (proportion of variance in the trait due to genetic influences) of IS and RSMA was estimated using SOLAR software . For discrete traits, SOLAR uses a threshold model to estimate polygenic heritability . Estimates were also computed using jPAP software ; no substantive differences were found.
We used the genetic map provided by CIDR based on the deCODE genetic map . Base pair positions were obtained from the March 2006 human reference sequence (hg18) assembly. Analysis was performed using the multipoint linkage software MCLINK, a Markov chain Monte Carlo (MCMC) method that allows for multilocus linkage analysis on large extended pedigrees . Using blocked Gibbs sampling, MCLINK generates inheritance vectors from the Markov chain. Each state in this chain is an inheritance state, indicating the grandpaternal or grandmaternal origin of an allele at each marker locus, with changes in the origin of alleles along the inheritance vector indicating points of recombination. MCLINK then estimates the log-likelihood function linkage statistics. Internally, MCLINK runs the analysis five times to ensure a consistent solution. MCLINK has been used previously to identify candidate genomic regions for a number of complex diseases [44–48]. Results from MCLINK have shown a high degree of similarity to other MCMC linkage methods , and to exact linkage methods and variance components linkage methods as applied to extended pedigrees . Allele frequencies for the MCLINK analysis were estimated using all of the observed data.
We performed nonparametric and general parametric model-based analyses. Although nonparametric methods are the standard analytic approach for complex psychiatric disorders, parametric methods have some advantages in the analysis of a complex trait such as ASD, particularly when using large extended pedigrees. Parametric models, which are based on assumptions about the genotype-phenotype relationship, simplify the parameter space and allow for more powerful and efficient analyses without leading to false-positive results [51, 52]. We decided to use two simple dominant and recessive models based on an extensive set of simulation analyses in which the results of various simple inheritance models were compared with the results of analyses based on a specified true model of inheritance . Those simulation analyses found that the power to reach a given lod score using the simple models was approximately 80% that of the true model, and that the expected lod scores for the simple models approached the true expected lod scores. The multipoint hlod score allows for unlinked pedigrees and variation in the recombination fraction. The HLOD provided by MCLINK is robust to model mis-specification, and may reflect the true position of linkage regions more accurately under conditions of appreciable heterogeneity . HLOD scores have been shown to be more powerful than homogeneity LOD scores or model-free methods under these conditions [55, 56]. The HLOD has been shown to produce scores consistent with other published methods [57, 58].
For both IS and RSMA, the phenotype was coded as unknown if the measure was not available, unaffected if the score was in the lowest tertile for the scale, and affected if the score was in the upper two tertiles. This approach re-codes affection status for all subjects rather than selecting a subset of subjects with high values on the traits. For IS, raw score tertile bins were 0-1, 2-3 and > 3; for RSMA, they were 0-3, 4-6 and > 6. The tertiles were given different liability classes (penetrances) to weight those in the upper tertile more strongly. Our recessive model assumed a disease allele frequency of 0.05 and penetrances of each of the three genotypes of 0.0014, 0.0014 and 0.8 in the upper tertile, and 0.01, 0.01 and 0.5 in the middle tertile. For the dominant model, the disease allele frequency was 0.0025. The penetrances were 0.0014, 0.8 and 0.8 in the upper tertile, and 0.01, 0.5 and 0.5 in the middle tertile. These model parameters roughly reproduce the reported population frequency of ASDs .
Linkage analyses were repeated on the basis of residual scale scores to determine whether signals could be replicated using measures of IS and RSMA phenotypes that were statistically independent of each scale's correlation with the other. Thus, for each scale, residual scores were computed using the other scale as a covariate (that is, IS-Adj = IS adjusted for RSMA and RSMA-Adj = RSMA adjusted for IS). Then, residual scores were divided into tertiles, and phenotype and liability values were coded in the same manner as were raw scores, that is, the lowest tertile was coded as unaffected and the top two tertiles were coded as affected, and the penetrance of the highest tertile was greater than that of the lower two tertiles.
For HLOD scores, results are presented using the Lander and Kruglyak  genome-wide criteria. Suggestive linkage evidence was defined by a LOD score ≥ 1.86 and significant genome-wide linkage evidence was defined by a LOD score ≥ 3.30.
The zero-order correlation between RSMA and IS was r = 0.32 (P < 0.001), indicating that they share 10% of their variance (r2 = 0.322 = 0.10). Consequently, residual scores were closely correlated with the raw score of the same scale (correlations = 0.95, P-values < 0.001), and 90% of the variance of each scale was unique (r2 = 0.952 = 0.90).
RSMA was more strongly associated than IS with other ASD features (Table 1). Both IS and RSMA raw scores were correlated with ADI-R domain scores and SRS mannerisms scale, but the RSMA correlations with ADI-R social and SRS mannerisms scales were significantly greater than those for IS. RSMA but not IS was correlated with ADOS score (after controlling for the effect of ADOS module scale), age of first phrases and IQ measures. With the exception of IQ measures, criterion variables significantly associated with raw scale scores tended to have lower correlations with residual scores, which suggests that the variance that IS and RSMA have in common may reflect a broader ASD trait. IQ measures, which were negatively correlated with RSMA, tended to be even more negatively associated with RSMA-Adj, although this trend was nominally significant (P < 0.01) only for non-verbal IQ. IS-Adj was positively correlated with IQ measures even though raw score IS was not, and IS-IQ correlations were significantly higher with residual than with raw scores. Thus, the unique variance of both IS and RSMA was less strongly associated with ASD but more strongly associated with IQ, although the direction of the relations with IQ was opposite (Table 1).
The heritability of both scales was significant. For IS, H2 was 0.85 (P < 0.0004, SE = 0.21), and for RSMA, H2 was 0.51 (P < 0.03, SE = 0.26). Because the scales were significantly correlated, we also estimated the heritability of each with the other as a covariate. With RSMA as a covariate, IS was still significant (H2 = 0.69, P < 0.004, SE = 0.23) and RSMA was a significant covariate (P = 0.003). By contrast, when IS was entered as a covariate for RSMA, RSMA was not significantly heritable (H2 = 0.31, P = 0.13, SE = 0.27), but IS was a significant covariate (P < 0.0001).
Table 2 lists all regions with at least suggestive evidence of linkage (HLOD ≥ 1.86 for parametric tests  or P < 0.005 for nonparametric tests). There was strong correspondence between regions for which there was evidence of linkage with the recessive model and nonparametric linkage (NPL), which suggests that these linkage findings are resistant to model mis-specification. Fewer tests of the dominant model, compared with the recessive model, were suggestive or significant. Thus, to simplify presentation of genome-wide results, Figures 1 and 2 display the genome-wide distribution of HLOD scores for the recessive model only (Table 2, Figures 1 and 2).
Evidence of linkage reached genome-wide significance levels (HLOD > 3.30) for two regions, 2q37.1-q37.3 and 15q13.1-q14 (Table 2), so we examined the linkage evidence for these regions in greater detail (Table 3). For 2q37.1-q35.3, the linkage evidence was greater for the dominant model, so dominant model HLOD scores across chromosome 2 are shown in Figure 3 along with ASD HLOD scores from our earlier work . The evidence of linkage to 2q37.1-q37.3 was greater for IS than for IS-Adj, RSMA and RSMA-Adj. Note too that we observed no evidence of ASD linkage to this region in our earlier study . Taken together, these findings suggest 2q37.1-q35.3 may harbor a genetic risk marker for repetitive behavior, particularly IS, which is not strongly associated with ASD (Table 3, Figure 3).
Linkage results for chromosome 15 were of particular interest, both because of the different pattern of signals for IS and RSMA, and the linkage magnitude. Linkage evidence for both IS and RSMA at 15q13.1-q14 was greater for the recessive than for the dominant model (Table 3). Because there also was suggestive evidence with the recessive model of IS linkage to 15q21.1-q22.2 (Table 3), Figure 4 shows HLOD scores for the recessive model across chromosome 15. The linkage evidence at 15q13.1-q14 was greater for RSMA than for IS, but nonetheless the evidence for IS was suggestive. A different pattern of findings was observed at 15q21.1-q22.2. Not only was there no RSMA signal this location, but the IS-Adj signal was much stronger than the unadjusted IS signal (HLOD = 3.03 and 1.88, respectively; NPL = 3.10 and 2.60, respectively). This was the largest difference in linkage values between adjusted and unadjusted phenotypes for any locus at which at least suggestive linkage evidence was observed for both raw and residual data. Thus, it appears that the shared variance between IS and RSMA actually dampened the IS signal at 15q21.1-q22.2. Finally, note in Figure 4 that 15q13.1-q14 and 15q21.1-22.2 both lie within a broader region in which we found evidence at genome-wide significance levels of linkage with ASD in our previous study with the same pedigrees . Linkage evidence for ASD in the 15q13.1-q14 region is comparable with that for the two RSMA variables, but even stronger evidence of ASD linkage was observed in the 15q21.1-q22.2 region (Figure 4).
In a large sample of multiplex ASD pedigrees, we found evidence that IS and RMSA are distinct phenotypes that can be differentiated by both their phenotypic and genotypic relations. Further, the results suggest that ASD susceptibility loci vary in the breadth of their phenotypic effects. Finally, the results illustrate the value of using narrowly defined phenotypes to detect the specific contribution of implicated susceptibility loci to the heterogeneous ASD phenotype.
IS and RSMA as distinct phenotypes
The overall pattern of relations of the two RRSB scales and their residuals with other ADI-R and ADOS measures suggests that although both RSMA and IS are indices of ASD severity, the relation with ASD severity is greater for RSMA than for IS and is in part a function of the shared variance between IS and RSMA. This general conclusion that RSMA is more closely associated with ASD severity is consistent with a previous report of the correlates of these scales . The negative correlation between RSMA and IQ and the absence of a significant correlation between IS and IQ are consistent with previous reports [6, 20], but the finding that the absolute magnitude of IQ correlations with both RSMA-Adj and IS-Adj is greater than IQ correlations with the raw scale values has not been reported previously. Taken together, these correlational findings suggest that the shared variance between IS and RSMA is associated with ASD severity but not with IQ.
The hypothesis that the positive relation between IS-Adj and IQ is mediated by anxiety is offered for further investigation. Anxiety, which is a common comorbid condition for ASD [60–62], has been reported to be positively correlated with IQ in children and adolescents with ASD [60, 61]. If obsessive behaviors are attempts to regulate anxiety , then perhaps the positive relation between IS-Adj and IQ we observed is in part a consequence of the positive relation that others have reported between anxiety and IQ. Given that no data are available to support an association between the IS-Adj scale and anxiety, the hypothesis that the relation between IQ and IS-Adj is mediated by anxiety remains to be tested empirically.
Our results indicate that whereas both IS and RSMA are heritable, the estimated heritability was greater for IS. Further, the heritability of RSMA may not be independent of its relation with IS. Our findings are consistent with previous reports of significant heritability for IS [21, 25], but in our families we find significantly positive heritability for RSMA as well. It is possible that the weaker RSMA heritability effect was not detected in those earlier reports.
Finally, we found different linkage patterns for IS and RSMA. There were many instances of suggestive signals linked to one but not the other phenotype, including differential linkage of IS at 9p24.3-p24.1 and 15q21.2-q22.2 (Table 2). The only signals that reached genome-wide significance were at different loci for each scale: 2q37.1-37.3 for IS and at 15q13.1-q14 for RSMA. It is true that at both 2q37.1-37.3 and 15q13.1-q14 there was suggestive evidence of linkage with the other scale (Table 3, Figure 3, Figure 4), but consideration of linkage results for residual scales and linkage results for ASD at the two loci suggests different interpretations of these suggestive signals. At 2q37.1-37.3, where there was a significant signal for IS, the suggestive signal for RSMA was not observed with RSMA-Adj and there was no linkage with ASD. Thus, it is possible that this region is relatively specific to IS, and that the suggestive signal for RSMA can be attributed to correlation of RSMA with IS. By contrast, at 15q13.1-q14, where there was a significant signal for RSMA, suggestive signals were found for both IS and IS-Adj, indicating that the IS signal was not due to the RSMA-IS correlation; the region was also linked to ASD in our earlier study. Thus, it seems likely that RSMA, being more strongly correlated with other ASD criteria, was more strongly linked to 15q13.1-q14, which appears to harbor risk markers for a broad range of ASD traits.
Implications for studies of narrow phenotypes
Some of the IS- and RSMA-specific findings not replicated in our affected status analyses (for example, the significant signal specific to IS at 2q37.1-37.3) may be examples of the hoped-for outcome of identifying susceptibility loci that are specific to narrowly defined phenotypes . Given that ASD is probably caused by many genes, each with relatively small effects [64, 65], increasing our ability to detect such genes is crucial. Thus, these findings encourage further research with narrowly defined phenotypes to uncover linkage signals not observed with broader diagnostic categories.
Further, our findings provide an example of increased knowledge of the nature of genetic effects that may be possible with more homogeneous phenotypes. Previously, we reported possibly distinct ASD regions with evidence of linkage at 15q13.1-q14, 15q14-q21.1 and 15q21.1-q22.2 . We now report that 15q13.1-q14 is linked to both RSMA and IS, but is linked more strongly to RSMA and that 15q21.1-q22.2 is linked to IS but not to RSMA. Thus, these two loci appear to affect different aspects of repetitive behavior, a possibility that was missed in our analysis of affected status.
The variability observed in this study in the phenotypic scope of linkage regions leads us to suggest that multiple ASD phenotypes should be used in future genetic studies to characterize the nature and breadth of the phenotypic linkage or association of risk variants. It is possible that variants with broad phenotypic effects may affect the root causes of ASD, whereas variants with narrow effects contribute to phenotypic heterogeneity among individuals with ASD. The use of multiple phenotypes emphasizes the importance of additional research aimed at developing an empirical model of the relations and interactions between specific features of ASD. Such a model should lead to identification of a set of phenotype measures that assess all the key specific features of ASD. The work of previous investigators to identify IS and RSMA as distinct features of repetitive behavior is a substantial contribution to this goal.
We note that our results are again consistent with the well-replicated finding of complexity and heterogeneity in ASD genetics. Our lod scores showed inter- and intra-family heterogeneity. For extended pedigrees, the scores expected under an assumption of a shared haplotype across all affected members exceeded by several lod units those actually found, depending upon the pedigree and model assumptions. Homogeneity clearly did not exist across all pedigrees in our sample; for any given region, multiple pedigrees showed no evidence of linkage.
Previous genetic studies of repetitive and stereotyped behavior
Shao et al.  reported the only linkage study of which we are aware that stratified pedigrees on either IS or RSMA. That study differs from the present study in several regards. First, they limited their linkage analysis to the 15q11-q13 region, whereas we did a genome-wide scan. Second, they used nuclear families rather than extended pedigrees. Third, they used the diagnosis of AD as the phenotype, whereas we used IS and RSMA as phenotypes. Finally, they used ordered-subset analysis and we did not. Shao et al. did not find significant evidence of linkage in the 15q11-q13 region across all 81 families they studied but they did find significant evidence of linkage in the region of marker GABRB3 in the subset of 23 families with the highest mean IS scores. Stratifying families by RSMA or RRSB did not enhance the signal. GABRB3 is located at 24.4 Mb, which is upstream of the lower boundary (27.94 Mb) of 15q13.1-q14. We did not choose subsets of our sample, but rather re-defined affection status based on IS or RSMA phenotypic information, using information from all ASD members of the pedigrees. The methodological differences between our study and that of Shao et al. preclude firm conclusions about why they found that stratifying on IS but not RSMA enhanced the AD linkage signal, whereas we found both RSMA and IS, but particularly RSMA, to be associated with a region just downstream.
Studies that stratified pedigrees by other repetitive behavior measures, including individual RRSB items and the 'compulsions' factor examined by Tadevosyan-Lefer et al. , report increased HLOD scores for AD at chromosome 1  and at 17q11.2 . Significant associations between SLC25A12 alleles (2q31.1) have been reported for both the RRSB 'routines and rituals' category (similar to IS)  and the compulsions factor . None of these loci overlaps signals that we obtained for IS or RSMA linkage. These differences may again be due in part to methodological differences between choosing subsets versus re-defining phenotypes.
The suggestive evidence of IS linkage that we observed on chromosome 9 for IS spans a region implicated as a susceptibility locus for OCD in two studies [66, 67]. This replication is noteworthy because the earlier two studies did not include subjects with ASD. We did not find evidence of linkage for ASD diagnosis in this region using our full set of families, although we did find a evidence of linkage for ASD in this region in our analysis of a single large extended pedigree . Previous research has indicated that OCD features in parents of children with AD are correlated with scores for IS but not RSMA in probands . Thus, this region at the chromosome 9 telomere may underlie a repetitive behavior broader autism phenotype rather than ASD.
Our sample was a cohort of multiplex ASD pedigrees, and IS and RSMA data were collected only on subjects thought to have ASD. We believe our method is appropriate to the valid aim of uncovering susceptibility loci for ASD and related phenotypes within extended families containing multiple members with ASD. However, we acknowledge that our method limits the generalizability of our findings to other research aims. For example, the absence of repetitive behavior phenotype data for family members without ASD limits our ability to answer the question of whether repetitive behavior is a broader autism phenotype that occurs in unaffected relatives [30, 69]. Further, because our sample is not population-based, we cannot generalize our findings to the search for genetic markers for repetitive behavior in the general population . Finally, our study includes analyses of the IS and RSMA phenotypes under two simple dominant and recessive models. If we conservatively assume that these models and phenotypes are not correlated, then significance thresholds would be adjusted by log10(4) = 0.6 lod score units. Our thresholds would then be 2.26 for suggestive evidence and 3.9 for significant evidence. With this adjustment, results on chromosome 15 remain significant and many other results remain suggestive, but other results would be considered as nominal.
IS and RSMA, two factors within the ADI-RRSB domain, were found to be linked to largely non-overlapping chromosomal regions. Genome-wide significance was observed for IS at 2q37.1-q37.3 (dominant model HLOD = 3.42) and for RSMA at 15q13.1-q14 (recessive model HLOD = 3.93). Regions varied in the range of phenotypes with which they were linked. These findings support the value of including multiple, narrowly defined phenotypes in ASD genetic research.
Abrahams BS, Geschwind DH: Advances in autism genetics: on the threshold of a new neurobiology. Nat Rev Genet. 2008, 9: 341-355. 10.1038/nrg2346.
American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision. 2000, Washington, DC: American Psychiatric Association
Happe F, Ronald A, Plomin R: Time to give up on a single explanation for autism. Nat Neurosci. 2006, 9: 1218-1220. 10.1038/nn1770.
Ronald A, Happe F, Bolton P, Butcher LM, Price TS, Wheelwright S, Baron-Cohen S, Plomin R: Genetic heterogeneity between the three components of the autism spectrum: a twin study. J Am Acad Child Adolesc Psychiatry. 2006, 45: 691-699. 10.1097/01.chi.0000215325.13058.9d.
Ronald A, Happe F, Plomin R: The genetic relationship between individual differences in social and nonsocial behaviours characteristic of autism. Dev Sci. 2005, 8: 444-458. 10.1111/j.1467-7687.2005.00433.x.
Hus V, Pickles A, Cook EH, Risi S, Lord C: Using the autism diagnostic interview--revised to increase phenotypic homogeneity in genetic studies of autism. Biol Psychiatry. 2007, 61: 438-448. 10.1016/j.biopsych.2006.08.044.
Buxbaum JD, Silverman JM, Smith CJ, Kilifarski M, Reichert J, Hollander E, Lawlor BA, Fitzgerald M, Greenberg DA, Davis KL: Evidence for a susceptibility gene for autism on chromosome 2 and for genetic heterogeneity. Am J Hum Genet. 2001, 68: 1514-1520. 10.1086/320588.
Ramoz N, Cai G, Reichert JG, Silverman JM, Buxbaum JD: An analysis of candidate autism loci on chromosome 2q24-q33: evidence for association to the STK39 gene. Am J Med Genet B Neuropsychiatr Genet. 2008, 147B: 1152-1158. 10.1002/ajmg.b.30739.
Buxbaum JD, Silverman J, Keddache M, Smith CJ, Hollander E, Ramoz N, Reichert JG: Linkage analysis for autism in a subset families with obsessive-compulsive behaviors: evidence for an autism susceptibility gene on chromosome 1 and further support for susceptibility genes on chromosome 6 and 19. Mol Psychiatry. 2004, 9: 144-150. 10.1038/sj.mp.4001465.
McCauley JL, Olson LM, Dowd M, Amin T, Steele A, Blakely RD, Folstein SE, Haines JL, Sutcliffe JS: Linkage and association analysis at the serotonin transporter (SLC6A4) locus in a rigid-compulsive subset of autism. Am J Med Genet B Neuropsychiatr Genet. 2004, 127B: 104-112. 10.1002/ajmg.b.20151.
Shao Y, Cuccaro ML, Hauser ER, Raiford KL, Menold MM, Wolpert CM, Ravan SA, Elston L, Decena K, Donnelly SL, Abramson RK, Wright HH, DeLong GR, Gilbert JR, Pericak-Vance MA: Fine mapping of autistic disorder to chromosome 15q11-q13 by use of phenotypic subtypes. Am J Hum Genet. 2003, 72: 539-548. 10.1086/367846.
Nurmi EL, Dowd M, Tadevosyan-Leyfer O, Haines JL, Folstein SE, Sutcliffe JS: Exploratory subsetting of autism families based on savant skills improves evidence of genetic linkage to 15q11-q13. J Am Acad Child Adolesc Psychiatry. 2003, 42: 856-863. 10.1097/01.CHI.0000046868.56865.0F.
Ma DQ, Jaworski J, Menold MM, Donnelly S, Abramson RK, Wright HH, Delong GR, Gilbert JR, Pericak-Vance MA, Cuccaro ML: Ordered-subset analysis of savant skills in autism for 15q11-q13. Am J Med Genet B Neuropsychiatr Genet. 2005, 135B: 38-41. 10.1002/ajmg.b.30166.
Mulder EJ, Anderson GM, Kema IP, Brugman AM, Ketelaars CE, de Bildt A, van Lang ND, den Boer JA, Minderaa RB: Serotonin transporter intron 2 polymorphism associated with rigid-compulsive behaviors in Dutch individuals with pervasive developmental disorder. Am J Med Genet B Neuropsychiatr Genet. 2005, 133B: 93-96. 10.1002/ajmg.b.30122.
Silverman JM, Buxbaum JD, Ramoz N, Schmeidler J, Reichenberg A, Hollander E, Angelo G, Smith CJ, Kryzak LA: Autism-related routines and rituals associated with a mitochondrial aspartate/glutamate carrier SLC25A12 polymorphism. Am J Med Genet B Neuropsychiatr Genet. 2008, 147: 408-410.
Sutcliffe JS, Delahanty RJ, Prasad HC, McCauley JL, Han Q, Jiang L, Li C, Folstein SE, Blakely RD: Allelic heterogeneity at the serotonin transporter locus (SLC6A4) confers susceptibility to autism and rigid-compulsive behaviors. Am J Hum Genet. 2005, 77: 265-279. 10.1086/432648.
Constantino JN: The Social Responsiveness Scale. 2002, Los Angeles, CA: Western Psychological Services
Le Couteur A, Lord C, Rutter M: The Autism Diagnostic Interview--Revised (ADI-R). 2003, Los Angeles: Western Psychological Services
Lord C, Rutter M, Le Couteur A: Autism Diagnostic Interview--Revised: a revised version of a diagnostic interview for caregivers of individuals with possible pervasive developmental disorders. J Autism Dev Disord. 1994, 24: 659-685. 10.1007/BF02172145.
Bishop SL, Richler J, Lord C: Association between restricted and repetitive behaviors and nonverbal IQ in children with autism spectrum disorders. Child Neuropsychol. 2006, 12: 247-267. 10.1080/09297040600630288.
Cuccaro ML, Shao Y, Grubber J, Slifer M, Wolpert CM, Donnelly SL, Abramson RK, Ravan SA, Wright HH, DeLong GR, Pericak-Vance MA: Factor analysis of restricted and repetitive behaviors in autism using the Autism Diagnostic Interview-R. Child Psychiatry Hum Dev. 2003, 34: 3-17. 10.1023/A:1025321707947.
Mooney EL, Gray KM, Tonge BJ, Sweeney DJ, Taffe JR: Factor analytic study of repetitive behaviours in young children with pervasive developmental disorders. J Autism Dev Disord. 2009, 39 (5): 765-74. 10.1007/s10803-008-0680-5.
Papageorgiou V, Georgiades S, Mavreas V: Brief report: cross-cultural evidence for the heterogeneity of the restricted, repetitive behaviours and interests domain of autism: a Greek study. J Autism Dev Disord. 2008, 38: 558-561. 10.1007/s10803-007-0409-x.
Richler J, Bishop SL, Kleinke JR, Lord C: Restricted and repetitive behaviors in young children with autism spectrum disorders. J Autism Dev Disord. 2007, 37: 73-85. 10.1007/s10803-006-0332-6.
Szatmari P, Georgiades S, Bryson S, Zwaigenbaum L, Roberts W, Mahoney W, Goldberg J, Tuff L: Investigating the structure of the restricted, repetitive behaviours and interests domain of autism. J Child Psychol Psychiatry. 2006, 47: 582-590. 10.1111/j.1469-7610.2005.01537.x.
Lam KS, Bodfish JW, Piven J: Evidence for three subtypes of repetitive behavior in autism that differ in familiality and association with other symptoms. J Child Psychol Psychiatry. 2008, 49: 1193-1200.
Tadevosyan-Leyfer O, Dowd M, Mankoski R, Winklosky B, Putnam S, McGrath L, Tager-Flusberg H, Folstein SE: A principal components analysis of the Autism Diagnostic Interview-Revised. J Am Acad Child Adolesc Psychiatry. 2003, 42: 864-872. 10.1097/01.CHI.0000046870.56865.90.
Silverman JM, Smith CJ, Schmeidler J, Hollander E, Lawlor BA, Fitzgerald M, Buxbaum JD, Delaney K, Galvin P: Symptom domains in autism and related conditions: evidence for familiality. Am J Med Genet. 2002, 114: 64-73. 10.1002/ajmg.10048.
Spiker D, Lotspeich L, Kraemer HC, Hallmayer J, McMahon W, Petersen PB, Nicholas P, Pingree C, Wiese-Slater S, Chiotti C: Genetics of autism: characteristics of affected and unaffected children from 37 multiplex families. Am J Med Genet. 1994, 54: 27-35. 10.1002/ajmg.1320540107.
Abramson RK, Ravan SA, Wright HH, Wieduwilt K, Wolpert CM, Donnelly SA, Pericak-Vance MA, Cuccaro ML: The relationship between restrictive and repetitive behaviors in individuals with autism and obsessive compulsive symptoms in parents. Child Psychiatry Hum Dev. 2005, 36: 155-165. 10.1007/s10578-005-2973-7.
Allen-Brady K, Robison R, Cannon D, Varvil T, Villalobos M, Pingree C, Leppert MF, Miller J, McMahon WM, Coon H: Genome-wide linkage in Utah autism pedigrees. Mol Psychiatry. 2009, doi:10.1038/mp.2008.14
Lord C, Risi S, Lambrecht L, Cook EH, Leventhal BL, DiLavore PC, Pickles A, Rutter M: The autism diagnostic observation schedule-generic: a standard measure of social and communication deficits associated with the spectrum of autism. J Autism Dev Disord. 2000, 30: 205-223. 10.1023/A:1005592401947.
Wechsler D: Manual for the Wechsler Intelligence Scale for Children-Third Edition. 1991, San Antonio, TX: The Psychological Corporation
Wechsler D: Wechsler Adult Intelligence Scale-Third Edition. 1997, San Antonio, TX: The Psychological Corporation
Elliott C: Differential Ability Scales. 1990, San Antonio, TX: The Psychological Corporation
Mullen E: Mullen Scales of Early Learning, AGS Edition. 1995, Circle Pines, MN: American Guidance Service
Levinson DF, Holmans P: The effect of linkage disequilibrium on linkage analysis of incomplete pedigrees. BMC Genet. 2005, 6 (Suppl 1): S6-10.1186/1471-2156-6-S1-S6.
Purcell S, Neale B, Todd-Brown K, Thomas L, Ferreira MA, Bender D, Maller J, Sklar P, de Bakker PI, Daly MJ, Sham PC: PLINK: a tool set for whole-genome association and population-based linkage analyses. Am J Hum Genet. 2007, 81: 559-575. 10.1086/519795.
Almasy L, Blangero J: Multipoint quantitative-trait linkage analysis in general pedigrees. Am J Hum Genet. 1998, 62: 1198-1211. 10.1086/301844.
Duggirala R, Williams JT, Williams-Blangero S, Blangero J: A variance component approach to dichotomous trait linkage analysis using a threshold model. Genet Epidemiol. 1997, 14: 987-992. 10.1002/(SICI)1098-2272(1997)14:6<987::AID-GEPI71>3.0.CO;2-G.
Hasstedt SJ: jPAP: Document-driven software for genetic analysis. Genet Epidemiol. 2005, 29: 255-
Kong A, Gudbjartsson DF, Sainz J, Jonsdottir GM, Gudjonsson SA, Richardsson B, Sigurdardottir S, Barnard J, Hallbeck B, Masson G, Shlien A, Palsson ST, Frigge ML, Thorgeirsson TE, Gulcher JR, Stefansson K: A high-resolution recombination map of the human genome. Nat Genet. 2002, 31: 241-247.
Thomas A, Gutin A, Abkevich V, Bansal A: Multilocus linkage analysis by blocked Gibbs sampling. Statistics and Computing. 2000, 10: 259-269. 10.1023/A:1008947712763.
Allen-Brady K, Norton PA, Farnham JM, Teerlink C, Cannon-Albright LA: Significant linkage evidence for a predisposition gene for pelvic floor disorders on chromosome 9q21. Am J Hum Genet. 2009, 84: 678-682. 10.1016/j.ajhg.2009.04.002.
Camp NJ, Neuhausen SL, Tiobech J, Polloi A, Coon H, Myles-Worsley M: Genomewide multipoint linkage analysis of seven extended Palauan pedigrees with schizophrenia, by a Markov-chain Monte Carlo method. Am J Hum Genet. 2001, 69: 1278-1289. 10.1086/324590.
Christensen GB, Camp NJ, Farnham JM, Cannon-Albright LA: Genome-wide linkage analysis for aggressive prostate cancer in Utah high-risk pedigrees. Prostate. 2007, 67: 605-613. 10.1002/pros.20554.
Kort EN, Ballinger DG, Ding W, Hunt SC, Bowen BR, Abkevich V, Bulka K, Campbell B, Capener C, Gutin A, Harshman K, McDermott M, Thorne T, Wang H, Wardell B, Wong J, Hopkins PN, Skolnick M, Samuels M: Evidence of linkage of familial hypoalphalipoproteinemia to a novel locus on chromosome 11q23. Am J Hum Genet. 2000, 66: 1845-1856. 10.1086/302945.
Stone S, Abkevich V, Hunt SC, Gutin A, Russell DL, Neff CD, Riley R, Frech GC, Hensel CH, Jammulapati S, Potter J, Sexton D, Tran T, Gibbs D, Iliev D, Gress R, Bloomquist B, Amatruda J, Rae PM, Adams TD, Skolnick MH, Shattuck D: A major predisposition locus for severe obesity, at 4p15-p14. Am J Hum Genet. 2002, 70: 1459-1468. 10.1086/340670.
Coon H, Matsunami N, Stevens J, Miller J, Pingree C, Camp NJ, Thomas A, Krasny L, Lainhart J, Leppert MF, McMahon W: Evidence for linkage on chromosome 3q25-27 in a large autism extended pedigree. Hum Hered. 2005, 60: 220-226. 10.1159/000090546.
Horne BD, Malhotra A, Camp NJ: Comparison of linkage analysis methods for genome-wide scanning of extended pedigrees, with application to the TG/HDL-C ratio in the Framingham Heart Study. BMC Genet. 2003, 4 (Suppl 1): S93-10.1186/1471-2156-4-S1-S93.
Goring HH, Terwilliger JD: Linkage analysis in the presence of errors I: complex-valued recombination fractions and complex phenotypes. Am J Hum Genet. 2000, 66: 1095-1106. 10.1086/302797.
Terwilliger JD, Goring HH: Gene mapping in the 20th and 21st centuries: statistical methods, data analysis, and experimental design. Hum Biol. 2000, 72: 63-132.
Greenberg DA, Abreu P, Hodge SE: The power to detect linkage in complex disease by means of simple LOD-score analyses. Am J Hum Genet. 1998, 63: 870-879. 10.1086/301997.
Abkevich V, Camp NJ, Gutin A, Farnham JM, Cannon-Albright L, Thomas A: A robust multipoint linkage statistic (tlod) for mapping complex trait loci. Genet Epidemiol. 2001, 21 (Suppl 1): S492-497.
Abreu PC, Greenberg DA, Hodge SE: Direct power comparisons between simple LOD scores and NPL scores for linkage analysis in complex diseases. Am J Hum Genet. 1999, 65: 847-857. 10.1086/302536.
Goldin LR: Detection of linkage under heterogeneity: comparison of the two-locus vs. admixture models. Genet Epidemiol. 1992, 9: 61-66. 10.1002/gepi.1370090107.
Christensen GB, Cannon-Albright LA, Thomas A, Camp NJ: Extracting disease risk profiles from expression data for linkage analysis: application to prostate cancer. BMC Proc. 2007, 1 (Suppl 1): S82-10.1186/1753-6561-1-s1-s82.
Hunt SC, Abkevich V, Hensel CH, Gutin A, Neff CD, Russell DL, Tran T, Hong X, Jammulapati S, Riley R, Weaver-Feldhaus J, Macalma T, Richards MM, Gress R, Francis M, Thomas A, Frech GC, Adams TD, Shattuck D, Stone S: Linkage of body mass index to chromosome 20 in Utah pedigrees. Hum Genet. 2001, 109: 279-285. 10.1007/s004390100581.
Lander E, Kruglyak L: Genetic dissection of complex traits: guidelines for interpreting and reporting linkage results. Nat Genet. 1995, 11: 241-247. 10.1038/ng1195-241.
Sukhodolsky DG, Scahill L, Gadow KD, Arnold LE, Aman MG, McDougle CJ, McCracken JT, Tierney E, Williams White S, Lecavalier L, Vitiello B: Parent-rated anxiety symptoms in children with pervasive developmental disorders: frequency and association with core autism symptoms and cognitive functioning. J Abnorm Child Psychol. 2008, 36: 117-128. 10.1007/s10802-007-9165-9.
Weisbrot DM, Gadow KD, DeVincent CJ, Pomeroy J: The presentation of anxiety in children with pervasive developmental disorders. J Child Adolesc Psychopharmacol. 2005, 15: 477-496. 10.1089/cap.2005.15.477.
White SW, Oswald D, Ollendick T, Scahill L: Anxiety in children and adolescents with autism spectrum disorders. Clin Psychol Rev. 2009, 29: 216-229. 10.1016/j.cpr.2009.01.003.
Matson JL, Nebel-Schwalm MS: Comorbid psychopathology with autism spectrum disorder in children: an overview. Res Dev Disabil. 2007, 28: 341-352. 10.1016/j.ridd.2005.12.004.
Bucan M, Abrahams BS, Wang K, Glessner JT, Herman EI, Sonnenblick LI, Alvarez Retuerto AI, Imielinski M, Hadley D, Bradfield JP: Genome-wide analyses of exonic copy number variants in a family-based study point to novel autism susceptibility genes. PLoS Genet. 2009, 5: e1000536-10.1371/journal.pgen.1000536.
Buxbaum JD: Multiple rare variants in the etiology of autism spectrum disorders. Dialogues Clin Neurosci. 2009, 11: 35-43.
Hanna GL, Veenstra-VanderWeele J, Cox NJ, Boehnke M, Himle JA, Curtis GC, Leventhal BL, Cook EH: Genome-wide linkage analysis of families with obsessive-compulsive disorder ascertained through pediatric probands. Am J Med Genet. 2002, 114: 541-552. 10.1002/ajmg.10519.
Willour VL, Yao Shugart Y, Samuels J, Grados M, Cullen B, Bienvenu OJ, Wang Y, Liang KY, Valle D, Hoehn-Saric R, Riddle M, Nestadt G: Replication study supports evidence for linkage to 9p24 in obsessive-compulsive disorder. Am J Hum Genet. 2004, 75: 508-513. 10.1086/423899.
Allen-Brady K, Miller J, Matsunami N, Stevens J, Block H, Farley M, Krasny L, Pingree C, Lainhart J, Leppert M, McMahon WM, Coon H: A high-density SNP genome-wide linkage scan in a large autism extended pedigree. Mol Psychiatry. 2009, 14: 590-600. 10.1038/mp.2008.14.
Szatmari P, Georgiades S, Duku E, Zwaigenbaum L, Goldberg J, Bennett T: Alexithymia in parents of children with autism spectrum disorder. J Autism Dev Disord. 2008, 38: 1859-1865. 10.1007/s10803-008-0576-4.
Cohen J, Cohen P: Applied Multiple Regression/Correlation Analysis for the Behavioral Sciences. 1983, Hillsdale, N.J.: Lawrence Erlbaum Associates, Publishers
This work was supported by R01 MH06359, the Utah Autism Foundation, the Carmen B. Pingree School for Children with Autism and the University of Utah General Clinical Research Center, which is funded by NCRR grant RR025764. Partial support for all datasets within the Utah Population Database (UPDB) was provided by the University of Utah Huntsman Cancer Institute. We thank our staff whose countless hours of work have made this study possible. We also greatly appreciate the time and effort given by the family members who participated in this study.
HC, WMM and JMM received partial salary support from Lineagen Inc. http://www.lineagen.com from 12/1/07 to 12/31/08. This salary support is not ongoing.
DSC conducted the statistical analyses and drafted the manuscript. JMM confirmed research diagnoses of ASD, supervised collection of all phenotype data and made significant contributions to the interpretation of results. RJR and KAB assisted with interpretation of results and helped draft the manuscript. MEV and NKW verified the accuracy of data extracted from a research database and contributed to the interpretation of results. WMM participated in the design of the study and helped draft the manuscript. HC conceived of the study, participated in its design, directed the statistical analyses and helped to draft the manuscript. All authors read and approved the final manuscript.
Dale S Cannon and Hilary Coon contributed equally to this work.