Facing the challenge of teaching emotions to individuals with low- and high-functioning autism using a new Serious game: a pilot study
© Serret et al.; licensee BioMed Central Ltd. 2014
Received: 5 February 2014
Accepted: 20 June 2014
Published: 1 July 2014
It is widely accepted that emotion processing difficulties are involved in Autism Spectrum Conditions (ASC). An increasing number of studies have focused on the development of training programs and have shown promising results. However, most of these programs are appropriate for individuals with high-functioning ASC (HFA) but exclude individuals with low-functioning ASC (LFA). We have developed a computer-based game called JeStiMulE based on logical skills to teach emotions to individuals with ASC, independently of their age, intellectual, verbal and academic level.
The aim of the present study was to verify the usability of JeStiMulE (which is its adaptability, effectiveness and efficiency) on a heterogeneous ASC group. We hypothesized that after JeStiMulE training, a performance improvement would be found in emotion recognition tasks.
A heterogeneous group of thirty-three children and adolescents with ASC received two one-hour JeStiMulE sessions per week over four weeks. In order to verify the usability of JeStiMulE, game data were collected for each participant. Furthermore, all participants were presented before and after training with five emotion recognition tasks, two including pictures of game avatars (faces and gestures) and three including pictures of real-life characters (faces, gestures and social scenes).
Descriptive data showed suitable adaptability, effectiveness and efficiency of JeStiMulE. Results revealed a significant main effect of Session on avatars (ANOVA: F (1,32) = 98.48, P < .001) and on pictures of real-life characters (ANOVA: F (1,32) = 49.09, P < .001). A significant Session × Task × Emotion interaction was also found for avatars (ANOVA: F (6,192) = 2.84, P = .01). This triple interaction was close to significance for pictures of real-life characters (ANOVA: F (12,384) = 1.73, P = .057). Post-hoc analyses revealed that 30 out of 35 conditions found a significant increase after training.
JeStiMulE appears to be a promising tool to teach emotion recognition not only to individuals with HFA but also those with LFA. JeStiMulE is thus based on ASC-specific skills, offering a model of logical processing of social information to compensate for difficulties with intuitive social processing.
Comité de Protection des Personnes Sud Méditerranée V (CPP): reference number 11.046 (https://cpp-sud-mediterranee-v.fr/).
KeywordsSerious game High-functioning Autism Low-functioning Autism Social Skills Training Emotion Recognition Computer-based Intervention
Autism Spectrum Conditions (ASC) are characterized by deficits in communication and social interaction as well as by repetitive stereotyped behaviors . These are complex and heterogeneous disorders affecting the quality of reciprocal social interactions, which is one of the most persistent symptoms in ASC . These difficulties are associated with social cognitive difficulties in ASC [3, 4] and, thus, are a major source of handicap for these individuals. Undeniably, emotions, conveyed by facial expressions, gestures, words or situations, are critical signals for understanding others’ feelings and intentions and for regulating social interactions . Individuals with ASC show atypical emotion processing. A number of studies have found deficits in this domain, often related to level of intelligence. However, the results of these studies are sometimes controversial [6, 7]. These deficits concern not only emotional facial expressions, but also emotional gestures , emotional scenes  and the recognition of emotions on the basis of contextual cues . Furthermore, individuals with ASC have reduced spontaneous social motivation  and shared emotions .
The social world is a highly complex system, constantly undergoing major, non-predictable, multi-domain, and random changes. It has been suggested that if rules exist in such a system, they are too complex to be fully understood by individuals with ASC. The same assumption is valid for the integration of these rules in social training . For this reason, the development of social training in ASC has been limited [14–16]. Yet, emotion processing skills in ASC progress over time [17, 18], suggesting that individuals with ASC have a certain learning potential , possibly based on the development of compensatory strategies [20–22]. Previous studies suggest that using stimuli with relatively reduced complexity, like cartoons, to teach emotions to children and adolescents with ASC is a beneficial therapeutic option. Indeed, there is evidence that emotion processing of real faces may be affected, while emotion processing of cartoon/avatar faces is relatively spared [23–25].
ICT and ASC
In this context, interventions based on Information and Communication Technologies (ICT) appear to be of special interest as they present several advantages for individuals with ASC: 1) these individuals usually show a strong interest in electronic media and devices [26, 27]; 2) these devices operate according to predictable rules and the information provided is clear, structured and unambiguous ; 3) ICT do not involve complex socio-emotional expectations [29, 30]; and 4) ICT may include virtual or synthetic environments, allowing individuals to experiment with various social situations, while reducing their social anxiety, as well as the fear of failing or of rejection that these individuals with ASC often experience in real face-to-face interactions [31, 32].
Social cognition, ICT and ASC
In the last decade, different ICT-based programs have been developed to teach social skills to individuals with ASC [33, 34]. Most of these training programs have targeted emotion recognition and have used photographs, videos, combined visual and audio stimuli or animated emotional expressions of fully or partially disclosed faces [29, 35–38]. Other interventions have also combined emotions with stimuli likely to capture the attention and the interest of children with ASC (for example, trains on the Transporters DVDs; ). These ICT-based programs reported interesting and encouraging results about the possibility to enhance emotion recognition in individuals with ASC. However, they present a number of limitations.
These programs include, for the most part, unimodal or non-integrated isolated stimuli (for example, visual and/or auditory stimuli), a limited number of scenarios (for example, static images or videos) and a considerable verbal demand (oral/written instructions and response options). Human-machine interaction is thus limited and participants with ASC experience difficulties understanding the task and/or responding accordingly. In addition, the reduced flexibility of scenarios does not allow participants to modulate their interaction with the programs (for example, to choose the game’s avatar with which they want to interact, to choose when to interact with it, and to provide an online adjustment of stimulations as a function of the participants’ responses). Finally, the major limitation is that most of these programs were designed for individuals with high-functioning autism (HFA) and thus do not cover the wide spectrum of ASC, which includes around 40 to 60% of individuals with low-functioning autism (LFA) [38, 40].
In order to provide a contribution in this direction, we developed an individual, interactive and multi-sensory computer game called JeStiMulE. JeStiMulE aims at teaching emotions not only to children and adolescents with HFA but also to those with LFA. It is a Serious game combining the fun of playing with learning. It includes several exercises to train emotion recognition on avatars (faces, gestures and social scenes), with similar features and goals to those of common video games for children. It also includes motivating instructional aspects (for example, short sequences with an immediate feedback, visual rewards, an innovative vibrotactile gamepad). Moreover, the player has the possibility to create his/her own avatar and to move in a virtual environment. This type of training environment offers the players the opportunity to experiment with various social situations which are similar to real life, to move freely and to choose when and how to interact with other avatars.
Furthermore, the environment developed in JeStiMulE is multi-sensory. Visual, audio and tactile stimulations are provided to facilitate the player’s immersion in the virtual world and to increase the attractiveness of the game. JeStiMulE contains other adaptations which are appropriate to the heterogeneous profile of individuals with ASC (LFA and HFA). Each emotion corresponded to a code. This allows non-verbal children and adolescents with ASC, as well as non-readers, to interact with the game and to be able to learn different emotions. Finally, given that ASC individuals are more efficient in processing the elements of the physical environment than in processing the social and emotional elements , JeStiMulE offers an ‘ASC-friendly environment’ where emotional and social elements are linked together by logical rules. According to Baron-Cohen , individuals with ASC are extreme ‘systemizers’ and prefer organized environments based on logical rules. Interestingly, logical skills used on JeStiMulE such as analogical reasoning [42–44] and implicit learning [45–47] appear as cognitive peaks in ASC (independent of overall IQ). In addition, these do not necessarily involve verbal or social skills. The aim of JeStiMulE is to compensate difficulties in intuitively understanding the social world through learning strategies adapted to the autistic cognitive profile. In this sense, using logical skills to teach emotions to individuals with ASC seems a relevant therapeutic option.
The present study aims at verifying the usability (which is its adaptability, effectiveness and efficiency)  of JeStiMulE, on a heterogeneous group of individuals with ASC. We hypothesized that after a four-week training with JeStiMulE, a performance improvement would be found in emotion recognition tasks, including not only on game avatars (faces and gestures) but also on pictures of real-life characters (faces, gestures and social scenes).
Characteristic (N = 33)
31 male, 2 female
Mean score (Standard deviation)
35 to 129
Age of semantic-syntactic language (years)
< 1 to 12
0 to 28
Special care (hours/week)
0 to 28
Only participants who were able to discriminate primary and secondary colors and had already used a computer were included in the training. Informed consent was obtained from all participants and their parents prior to participation. All procedures were approved for all day-care unit partners by the Local Ethical Committee (Comité de Protection des Personnes Sud Méditerranée V: reference number 11.046).
JeStiMulE: game description
JeStiMulE is a virtual reality game with a multi-sensory environment. Visual, tactile, and auditory stimulations are provided to facilitate game immersion. In JeStiMulE, each expression was associated to a visual non-verbal code (see Figure 1b), a corresponding verbal written code (which comprises emotional words and idiomatic expressions), and a tactile pattern (see further details below). Visual non-verbal codes corresponded to colors and a symbol. Each basic emotion was associated to a specific color from Plutchick’s emotional wheel (happiness = yellow, anger = red, disgust = purple, fear = green, sadness = light blue, surprise = dark blue)  (see Figure 1b). Pain was associated with black. Neutral was associated with the color white and ‘funny face’ was associated with a trash can. This last choice was made to introduce the notion of inappropriate facial expressions, in particular to participants without functional language.
Emotional words corresponded to the literal description of each emotion. Idiomatic expressions corresponded to short sentences that have a figurative meaning conventionally understood by native speakers. This meaning is different from the literal meaning of the idiomatic expression’s individual elements. Idiomatic expressions are very common in everyday language and constitute one essential part of the human’s emotional communication . For example, the emotion of Fear may be expressed figuratively by the idiomatic expression ‘to get cold feet’ (French: ‘avoir la chair de poule’ = ‘having the chicken skin’ (literal meaning)). These expressions were used because their meaning requires individuals with ASC to learn them explicitly .
Each emotion was associated to a specific tactile pattern to promote the association between emotions and a physical imprint. Each tactile pattern could be considered as a word in an emotional language corresponding to an icon in visual communication [58, 59]. Tactile patterns reinforce emotional meaning. The patterns were developed in a number of iterative user tests. The tactile stimulations were produced by eight actuators distributed over the gamepad body (see Figure 1c). The actuators position was defined by measuring the contact zones between the hand and the gamepad.
Auditory stimulations were repetitive musical sequences, emotional onomatopoeias and environmental noises presented along and according to the social scenes. JeStiMulE was designed based on specific user requirements. In order to offer adapted response options to all participants, in JeStiMulE each expression was presented not only with its corresponding non-verbal code but also with two corresponding verbal written codes (emotional words and idiomatic expressions).
JeStiMulE calibration phase: a tactile calibration was conducted separately for each participant according to his/her tactile sensory profile measured by the DUNN parental questionnaire . Based on this, participants with a hypersensitivity profile were given a weak tactile stimulation, the ones with normal sensitivity were given a medium tactile stimulation and the ones with a hyposensitivity profile were given a strong tactile stimulation. This phase also included a user-test to ensure that participants were able to use the gamepad to move the avatar around the platform.
- 2)JeStiMulE learning phase (see Figure 3): this phase included three levels with gradually increasing complexity. On level 1, emotions were displayed on faces and associated with a visual non-verbal code and a specific tactile stimulation. On level 2, emotions were displayed on faces combined with gestures and associated with a visual non-verbal code and a specific tactile stimulation. On level 3, emotions were displayed on faces combined with gestures and associated with a visual non-verbal code, a verbal written (emotional words and idiomatic expressions) code and a specific tactile stimulation.
Participants were first trained to recognize different emotional faces and gestures in static avatars and then in animated avatars. Avatars had different identities, ages and clothes. They were presented full-body, first alone and then in groups (of three, six or nine avatars).
In game 1, participants discovered different emotions. First, they only saw the visual non-verbal code on the screen. They used the gamepad’s joystick to move the cursor horizontally in two directions (which were right and left), across the different colors of the visual non-verbal code. When a participant clicked on a color, a full-body avatar, with the chosen expression (first facial, then facial combined with gesture expressions), appeared on the screen. Participants were free to choose the color they wanted to click on. On the gamepad, only the joystick and one button were activated to facilitate their use. Participants switched automatically to the next game after having seen each emotion ten times. Participants learned to make one-to-one associations. These associations started with a reduced level of complexity (for example, emotional face - code). Complexity increased with the gradual introduction of static then animated avatars, of additional emotional cues (which were faces + gestures), as well as by diversifying the avatars’ identities, ages and clothes.
In game 2, the participants’ emotional expression recognition was tested. On the first level, a static avatar with a facial expression appeared on the screen. The participants had to click on a moving target featuring the avatar’s eyes and mouth. This was done in order to guide the participants’ gaze towards the avatar’s face. Then, the visual non-verbal code appeared on the bottom of the screen. The participants could then choose the correct color corresponding to a given emotion. They switched to the next game after having succeeded three out of ten emotional recognitions for each emotion presented. On the level 2, an animated avatar appeared on the screen with a facial expression combined with an inappropriate gesture (game 2a) and a masked face with an appropriate gesture (game 2b). Following this, the visual non-verbal code appeared and the participants had to choose the correct responses. As described above, the participants switched to the next game after having succeeded in three out of ten emotional recognitions for each emotion presented. The participants learned to establish priority rules in order to successfully extract the relevant emotional information (for example, focus on eyes and mouth; focus on face if gestures were inappropriate; and focus on gestures if the face was masked).
In game 3, the participants’ learning and acquisition were tested. They switched to the next game after having succeeded in three out of ten emotional recognitions for each emotion presented. It is important to note that, in contrast with the other levels, level 3 of game 3 was exclusively dedicated to children and adolescents with HFA and AS. The recognition of emotion was performed on all avatars (facial and gestural expressions) of the game. It involved three consecutive responses modalities (visual non-verbal code, verbal written code with emotional words and idiomatic expressions).
Finally, game 4 (called ‘odd one out’ game) introduced multiple avatars with different emotions in order to promote emotion categorization. Participants scrolled down avatars (on an invisible carousel) with the gamepad joystick. They clicked on one, two or three avatars which did not express the same emotion as the others. In this game, there were three levels: one odd one out of three avatars (game 4a), two odd ones out of six avatars (game 4b) and three odd ones out of nine avatars (game 4c).
During each game of the learning phase, the participant could monitor his/her performance on a vertical colored gauge showing their progression. Feedback is provided after each trial in all games. Positive and negative feedback were provided by a brief and subtle green or red flash. To summarize, this learning phase offered a structured, progressive and adapted learning procedure, involving implicit learning, visual discrimination, attention to detail, categorization and memory skills.
The training phase included three modules in which the participants could apply the knowledge acquired during the learning phase of the game. Each module was composed of the same scenarios (which were 26 social scenarios and 4 scenarios involving a request formulation). The modules also comprised a puzzle (see further details below).
In order to promote the transfer of skills from one context to another (based on analogical reasoning), several variations were introduced from one module to the other within the same scenarios. Furthermore, the fact that modules followed a probabilistic progression encouraged participants to deduce rules, based on implicit learning, from one scene to the other. Thus, JeStiMulE is based on ASC-specific skills and offers a model of logical processing of social information in order to compensate their difficulties with intuitive social processing.
The usability assessment of JeStiMulE was based on three criteria: the adaptability, the effectiveness and the efficiency .
JeStiMulE design paid special attention to the sensory and learning profile of ASC in order to allow a heterogeneous (age, IQ, reading, and so on) group of individuals with ASC to play. It included the following adaptations:
a possibility to choose the intensity of tactile stimulations (strong, medium, weak and inactive) in accordance with each participant’s sensory profile;
a learning phase based on a learning-by-association procedure (visual non-verbal code-expression during the learning phase);
a possibility to choose a user-adapted response option (color, emotional words or idiomatic expressions) in accordance with each participant’s age, level of semantic-syntactic language acquisition or reading abilities;
games adapted to the cognitive style of the individuals with ASC: matching, attention to detail, implicit learning, categorization, analogical reasoning. For each game, a performance criterion was pre-established.
JeStiMulE includes two key-stages (visual non-verbal code-expression associations and recognition of emotions displayed in social scenes) which participants had to achieve. This structured design aimed at ensuring that the participants understood the principles of the game and could play efficiently.
The goal of JeStiMulE is to help participants to recognize emotions displayed in social scenes. The participants who achieved the two key-stages have been considered to have completed the main goal of the game, which is playing with effectiveness during a limited time.
The total number of trials was 154, corresponding to:Twenty-one avatar faces (three trials for each emotion) presented in a full-body display (see Figure 7a).Twenty-one avatar gestures (three trials for each emotion) presented in a full-body display on which faces were masked to make participants focus on the emotional gestures (see Figure 7b).
Thirty-nine real-life character faces (six trials for each basic emotion + three for paina, see Figure 7c) taken from the Radbout Faces Database .Forty-two real-life character gestures (six trials for each emotion) presented in a full-body display where faces were masked by a grey circle to make participants focus on the gestures (see Figure 7d).
Twenty-eight real-life character social scenes (four trials for each emotion; see Figure 7e). Photographs were selected by two independent experts. The characters’ faces were masked in order to make the participants focus on contextual cues to recognize the emotion expressed in the given social situation .
In order to verify these tasks reliability, a preliminary data collection was conducted on typically developing individuals (n = 17). Mean results are presented in Additional file 3.
At the beginning of each trial, the stimuli were displayed in the left side of the screen and the response options were displayed in the right side. The response options corresponded to seven rectangles of different colors according to the different expression-color associations included in JeStiMulE. In addition, the word corresponding to the expression was displayed inside each rectangle and every expression was read aloud. Participants were asked to determine the expression displayed by choosing and clicking (mouse click) on the rectangle corresponding to it. Participants were tested individually in a quiet room with pauses, if necessary. They were seated in front of a computer screen. The stimuli were presented in E-Prime 2 (Psychology Software Tools, Inc. http://www.pstnet.com/eprime.cfm), in successive blocks. Each task was presented separately. The order of the trials and the tasks was randomly assigned across participants. Correct responses were scored ‘1’ and incorrect responses or no response were scored ‘0’.
All sessions were conducted with a caregiver and took place in the same room. Each training session lasted one hour maximum. Participants played twice a week until they completed JeStiMulE. They had four weeks (corresponding to eight sessions) maximum to achieve this goal. The number of sessions thus varied from one participant to the other, according to their performance. Each game level could only be performed once.
In order to verify JeStiMulE’s usability (adaptability, effectiveness and efficiency), game data were collected for each participant.
After JeStiMulE, all participants were re-tested on emotion recognition (post-JeStiMulE testing, two weeks after training).
Descriptive analyses were conducted to verify JeStiMulE adaptability, effectiveness and efficiency.
Data on JeStiMulE’s avatars were analyzed using a repeated measures analysis of variance (ANOVA) (using Statistica 7 device, Statsoft http://www.statsoft.com) including Session (Before versus After), Task (Facial versus Gestures emotion) and Emotion (Happiness versus Surprise versus Sadness versus Fear versus Disgust versus Anger versus Pain). Analyses were completed by a post-hoc Tukey HSD test.
Data on real-life characters were analyzed using a repeated measures ANOVA including Session (Before versus After), Task (Facial versus Gestures versus Situations) and Emotion (Happiness versus Surprise versus Sadness versus Fear versus Disgust versus Anger versus Pain). Analyses were also completed by a post-hoc Tukey HSD test.
Results and discussion
% of individuals in the sample
(1a) Play with tactile stimulation
(1a) Move the avatar around the platform
(1b) Learning by association (learning phase)
Visual non- verbal code - faces
Visual non- verbal code - gestures
(1c) User- adapted response options (training phase)
Visual non- verbal code (that is color code)
(1d) Games adapted to the ASC cognitive style
Matching games (Games 1.2, 1.3 and 2.3)
Attention to detail (Game 1.2)
Implicit learning (Games 1.2, 2.2a and, 2.2b)
Categorization (Games 1.4a, b, c; and 2.4a, b, c)
Analogical reasoning (Modules 1, 2 or 3)
% of individuals in the sample
Learning visual non- verbal code- emotions associations
Visual non- verbal code - faces
Visual non- verbal code - gestures
Recognition of emotions displayed in social scenes (with user- adapted response options)
Visual non-verbal code (%)
Emotional words (%)
Idiomatic expressions (%)
Playing (training phase)
Validation module 1
Validation module 1 + 2
Two key-stage validation (73% of participants)
Mean score (standard deviation)
Number of sessions
3 to 8
Session time (minutes)
30 to 60
Adaptability (Table 2) - Results indicate that:
all participants were able to use the tactile gamepad and to move the avatar within the virtual world;
as predefined, all participants were trained to the visual non-verbal code-expression associations during the learning phase. Ninety-one percent of the participants were able to learn by association;
during the training phase, 43% of the participants responded using the visual non-verbal code, while 39% of them used emotional words and 18% used idiomatic expressions. All participants found a response modality adapted to their profile;
all ASC-adapted games could be played by at least 80% of the total number of participants.
Effectiveness (Table 3) - After the learning phase, 91% of the participants were able to make JeStiMulE visual non-verbal code-expression faces associations and 79% of them were able to make JeStiMulE visual non-verbal code-expression gestures associations. After the training phase, 91% of the participants completed JeStiMulE module 1, 85% completed module 1 and 2, and 73% completed all modules, independently of their response modality.
Efficiency (Table 4) - These data are based only on the performance of participants who completed the two key stages of JeStiMulE (which are visual non-verbal code-expression associations and recognition of emotions displayed in social scenes). These participants (73%) played, on average, 6 sessions (range 3 to 8 sessions) of 49 minutes (range 30 to 60 minutes) each.
JeStiMulE: training effect
Pictures of JeStiMulE’s avatars
Individual data are illustrated in Additional file 5. Results of statistical analyses revealed a significant main effect of Session (F (1,32) = 98.48, P < .001), Task (Faces versus Gestures, F (1,32) = 5.65, P = .02) and Emotion (F (6,192) = 18.60, P < .001). These results indicate that participants were more accurate at recognizing emotions after JeStiMulE (M = 64.65, SD = 17.50 versus M = 31.16, SD = 13.34; P < .001) and that their overall performance was superior on the Facial (M = 50.72, SD = 15.93) than on Gestural (M = 45.09, SD = 13.79; P = .02) emotion recognition task. Moreover, results revealed that emotions such as Sadness (M = 61.87, SD = 10.59), Happiness (M = 60.35, SD = 8.88) and Anger (M = 54.29, SD = 9.79) were better recognized than emotions such as Disgust (M = 33.84, SD = 8.30; all P < .001), Pain (M = 38.64, SD = 7.80; all P < .001) and Surprise (M = 40.15, SD = 7.72; all P < .005).
Results obtained for JeStiMulE’s avatars and for real-life characters
ASC (N = 33)
Real-life character tasks
Mean (standard deviation)
Mean (standard deviation)
Pictures of real-life characters
Individual data are illustrated in Additional file 5. Results of statistical analyses revealed a significant main effect of Session (F (1,32) = 49.09, P < .001), Task (Faces versus Gestures versus Situations, F (2,64) = 7.93, P < .001) and Emotion (F (6,192) = 23.88, P < .001). In line with the results for pictures of JeStiMulE’s avatars, these results show that participants were more accurate at recognizing emotions after using JeStiMulE (M = 61.21, SD = 28.83 vesrus M = 37.39, SD = 20.96, P < .001). Moreover, results indicate that the overall performance was inferior on the Gestures (M = 44.80, SD = 14.76) than on the Faces (M = 51.01, SD = 17.31, P = .007) and the Situations (M = 52.11, SD = 18.22, P = .001) tasks. Regarding the Emotion effect, the results of the post-hoc analysis revealed that the overall performance on Happy (M = 65.32, SD = 13.35) and Angry (M = 61.32, SD = 13.56) trials was superior compared to the performance on trials including other emotions (MSurprise = 39.35, SD = 11.38; MDisgust = 42.68, SD = 10.87; MPain = 48.14, SD = 10.80; MFear = 43.31, SD = 11.11; all P < .001).
Finally, the triple Session × Task × Emotion interaction was only close to significance (F (12,384) = 1.73, P = .057). In order to further track this interaction, post-hoc analyses were conducted. Results are fully presented in Table 5 (see also Figure 9 and Additional file 5).
The present study had two major objectives. Firstly, it aimed at verifying the usability of JeStiMulE (which is its adaptability, effectiveness and efficiency) on a heterogeneous group of individuals with ASC. Secondly, it aimed at investigating whether a four-week JeStiMulE training would improve emotion recognition not only on game avatars (faces and gestures) but also on pictures of real-life characters (faces, gestures and social scenes).
Results indicate that JeStiMulE presents a suitable usability. Most participants were able to play (effectiveness) and to complete the training within the expected time (efficiency), which supports the idea that JeStiMulE specificities, including sensory, cognitive and motivational dimensions, were adapted to the ASC profile. Moreover, participants were not only able to play but they also benefited from the training, as indicated by their improved performance on emotional recognition tasks. The results of this study thus provide evidence of the potential of JeStiMulE for individuals with ASC with heterogeneous intellectual, verbal and academic levels.
As every participant was tested before and after training, he/she would serve as his/her own control. This experimental decision helped us evaluate the effectiveness of the training on each individual. In this study, emotion recognition skills were assessed before and after four-week JeStiMulE training. A significant improvement was found in most of the tasks despite the heterogeneity of the group of participants. To date, most of the studies conducted in this field have focused on homogeneous groups with ASC. Very often, these studies focused on individuals with HFA including AS. There are few studies that include individuals with LFA. For example, Hopkins et al.  assessed the efficacy of ‘FaceSay’, a computer-based social skills training program for children with LFA and HFA. Their results suggested that providing children with ASC with a controlled, structured, and interactive environment with avatar assistants can help them to improve their social skills. Yet, this improvement was most significant for children with HFA. According to the authors, ‘it is possible that the children with LFA did not completely understand the concepts or directions in the games, and therefore, did not fully benefit from the intervention’ .
The present study provides an advance in this research field, by showing that children and adolescents with heterogeneous ASC were able to understand, play and complete JeStiMulE as well as benefit from the intervention. A performance increase was found not only for the pictures of avatars conditions, but also for the tasks including photographs of real-life stimuli (faces, gestures and social situations) with which participants were not trained.
While individuals with LFA and HFA do differ in many features (for example, IQ profile), they all share a fundamental one, that is their interest and preference for rule-based systems and their ease when interacting with them. JeStiMulE was developed specifically to provide a rule-based computer game relying on ‘autistic intelligence’ to develop other, more social, skills. The underlying working hypothesis was thus that rule-based learning could be a relevant pathway to reach gradually more complex, social learning in ASC. The idea that rule-based environments could compensate for difficulties in the domain of emotion recognition in ASC  is not novel. When designing JeStiMulE, we assumed that individuals with ASC could use their logical skills to learn emotions and could evolve in a systemizing environment.
JeStiMulE uses a virtual environment to simulate different social situations, offering thus more ecological learning opportunities than those of devices using static stimuli .
Participants included in this study only played once each game level and none of them repeated a successfully completed level or module. These training criteria clearly differ from that of more traditional approaches using repetitive learning to develop skills. For instance, behavioural methods encourage a ratio of known: new tasks of around 80:20% . Although these methods have received support from therapists and family associations, their efficiency has not yet been clearly demonstrated empirically and remains a matter of debate . In JeStiMulE, participants had the opportunity to learn progressively a great number of associations, in line with the idea of Kourkoulou et al.  that ‘restricting learning to a smaller number of stimuli may impede the flexibility with which individuals with ASD can learn new associations’. The results found in the present study suggest that children and adolescents with ASC can learn new associations rapidly and without needing repetition, when they are given the opportunity to use their cognitive strengths to compensate their difficulties in specific areas such as emotion processing. Interestingly, the skills trained with JeStiMulE appear to extend to other stimuli than those included in the training (pictures of real-life characters), suggesting flexible learning and a certain potential of generalization of acquisitions.
Study limitations and perspectives
ASC include complex and heterogeneous clinical profiles. When studying these disorders, researchers have often to choose to either include a sub-group of individuals with ASC (for example, HFA or LFA) or to include a group of individuals with profiles representative of the entire spectrum. In this study, we have made this latter choice. It allowed us to verify whether JeStiMulE would be a suitable tool for individuals on the whole ASC spectrum rather than for only a sub-group of this spectrum. In order to verify the beneficial effect of JeStiMulE, emotion recognition tasks covering the large spectrum of participants’ intellectual, verbal and academic profiles were proposed to participants. As a consequence, a trade-off between choosing wide-covering tasks and including a significant number of trials per task was necessary. While our results show that heterogeneity does not represent an obstacle to using and benefiting from JeStiMulE, the above mentioned limitations increase results’ variability. In this sense, the study results must be interpreted with precaution. Finally, because the current study was exploratory and aimed first at identifying potential JeStiMulE users, a control group was not included. Further studies including a control group, as well as a larger number of participants representative of the different ASC sub-groups and of different emotion developmental would be of interest to the field. These studies could also focus on the potential of learning transfer from experimental to real-life settings. The relevance of adding physical meaning to emotions (for example, using tactile stimulations) to ASC training programs should also be further explored.
In line with current advances (ASC-Inclusion project ), future versions of JeStiMulE are expected to extend to other relevant skills such as imitation and emotion expression (JEMImE ). Preliminary findings of our own group have shown that logical processing such as involved in JeStiMulE can also be used to develop academic skills, such word decoding, in children with ASC and without functional language (SEMA-TIC; manuscript in preparation).
Finally, it is important to note that the aim of this study was not to demonstrate that JeStiMulE can remediate the social impairment, which is a major difficulty at the core of ASC difficulties. ASC have indeed a neurogenetic basis altering brain development and cognitive functioning. This early atypical ‘brain machinery’ alters social perception and social interaction in a very specific way and throughout an individual’s development . Rather than ‘normalized’, social difficulties in ASC may be ‘compensated’ in order to help individuals with LFA and HFA increase their comprehension of the social world and develop strategies to cope with its demands.
aMatched validated stimuli could not be found for pain. So, three trials (pictures of Caucasian adults’ faces expressing pain) were created for this emotion.
Autism Diagnostic Interview-Revised
Autism Diagnostic Observation Schedule
analysis of variance
Autism Spectrum Conditions
Diagnostic and Statistical Manual of Mental Diseases, fourth edition
Epreuve de COmpréhension Syntaxico-SEmantique
Information and Communication Technologies
Pervasive Developmental Disorder-Not Otherwise Specified
Wechsler Abbreviated Scale of Intelligence.
We are grateful to all children, adolescents and their families for their participation in this study. We also thank our partners (Idées3 com, HLP Technologies, Autism Resources Center Nice, Day-care Units for children and adolescents with autism (‘La Caravelle’, ‘Les Coteaux d’Azur’ and ‘Les Noisetiers’) for their participation in this study. Written informed consent was obtained from all participants and their families. The consent form is held by the corresponding author and is available for review by the Editor-in-Chief. We also acknowledge the following associations for their support: Monaco Against Autism (MONAA), ABA Apprendre Autrement, Collectif Handicap 06. We acknowledge the support provided by CoBteK (Cognitive Behaviour Technology). The study promoter was the Nice University Hospital (CHUN). The project was funded by the French Ministry of Industry. The funding body had no role in the study design, data collection, analyses, and data interpretation.
- American Psychiatric Association: Diagnostic and Statistical Manual of Mental Disorders. 2000, Washington: American Psychiatric Association, 4Google Scholar
- Billstedt E, Gillberg C, Gillberg C: Autism after adolescence: population-based 13- to 22-year follow-up study of 120 individuals with autism diagnosed in childhood. J Autism Dev Disord. 2005, 35: 351-360.View ArticlePubMedGoogle Scholar
- Joseph RM, Tager-Flusberg H: The relationship of theory of mind and executive functions to symptom type and severity in children with autism. Dev Psychopathology. 2004, 16: 137-155.View ArticleGoogle Scholar
- Boraston Z, Blakemore SJ, Chilvers R, Skuse D: Impaired sadness recognition is linked to social interaction deficit in autism. Neuropsychologia. 2007, 45: 1501-1510.View ArticlePubMedGoogle Scholar
- Dunn J: Emotional development in early childhood: a social relationship perspective. The handbook of Affective Science. Edited by: Davidson RJ, Scherer KR, Goldsmith HH. 2003, Oxford: Oxford University Press, 332-346.Google Scholar
- Law Smith MJ, Montagne B, Perrett D, Gill M, Gallagher L: Detecting subtle facial emotion recognition deficits in high-functioning Autism using dynamic stimuli of varying intensities. Neuropsychologia. 2010, 48: 2777-2781.View ArticlePubMedGoogle Scholar
- Jones CR, Pickles A, Falcaro M, Marsden AJ, Happé F, Scott SK, Sauter D, Tregay J, Phillips RJ, Baird G, Simonoff E, Charman TA: Multimodal approach to emotion recognition ability in autism spectrum disorders. J Child Psychol Psychiatry. 2011, 52: 275-285.View ArticlePubMedGoogle Scholar
- Hubert B, Wicker B, Moore DG, Monfardini E, Duverger H, Da Fonseca D, Deruelle C: Brief report: recognition of emotional and non-emotional biological motion in individuals with autistic spectrum disorders. J Autism Dev Disord. 2007, 37: 1386-1392.View ArticlePubMedGoogle Scholar
- Santos A, Chaminade T, Da Fonseca D, Silva C, Rosset D, Deruelle C: Just another social scene: evidence for decreased attention to negative social scenes in high-functioning autism. J Autism Dev Disord. 2012, 42: 1790-1798.View ArticlePubMedGoogle Scholar
- Da Fonseca D, Santos A, Bastard-Rosset D, Rondan C, Poinso F, Deruelle F: Can children with autistic spectrum disorders extract emotions out of contextual cues?. Res Autism Spectr Disord. 2009, 3: 50-56.View ArticleGoogle Scholar
- Chevallier C, Kohls G, Troiani V, Brodkin ES, Schultz RT: The social motivation theory of autism. Trends Cogn Sci. 2012, 16: 231-239.PubMed CentralView ArticlePubMedGoogle Scholar
- Begeer S, Koot HM, Rieffe C, Meerum Terwogt M, Stegge H: Emotional competence in children with autism: diagnostic criteria and empirical evidence. Dev Rev. 2008, 28: 342-369.View ArticleGoogle Scholar
- Baron-Cohen S: The hyper-systemizing, assortative mating theory of autism. Progr Neuro Psychopharmacol Biol Psychiatry. 2006, 30: 865-872.View ArticleGoogle Scholar
- Andanson J, Pourre F, Maffre T, Raynaud JP: Social skills training groups for children and adolescents with Asperger syndrome: a review. Arch Pediatr. 2011, 18: 589-596.View ArticlePubMedGoogle Scholar
- Baghdadli A, Brisot-Dubois J, Picot MC, Michelon C: Comparison of the effect of two prosocial interventions about the evolution of recognition of facial expression abilities and social cognition of children with an Asperger syndrome or high functioning autism. Neuropsychiatr Enfance Adolesc. 2010, 58: 456-462.View ArticleGoogle Scholar
- Reichow B, Volkmar FR: Social skills interventions for individuals with autism: evaluation for evidence-based practices within a best evidence synthesis framework. J Autism Dev Disord. 2010, 40: 149-166.View ArticlePubMedGoogle Scholar
- Vicari S, Reilly JS, Pasqualetti P, Vizotto A, Caltagirone C: Recognition of facial expressions of emotions in school-age children: the intersection of perceptual and semantic categories. Acta Paediatr. 2000, 89: 836-845.View ArticlePubMedGoogle Scholar
- Gross TF: Recognition of immaturity and emotional expressions in blended faces by children with autism and other developmental disabilities. J Autism Dev Disord. 2008, 38: 297-311.View ArticlePubMedGoogle Scholar
- Adolphs R, Sears L, Piven J: Abnormal processing of social information from faces in autism. J Cogni Neurosci. 2001, 13: 232-240.View ArticleGoogle Scholar
- Grossman JB, Klin A, Carter AS, Volkmar FR: Verbal bias in recognition of facial emotions in children with Asperger syndrome. J Child Psychol Psychiatry. 2000, 41: 369-379.View ArticlePubMedGoogle Scholar
- Vesperini S, Askenazy F, Renaudo N, Staccini P, Hun S, Legall E, Serret S: Prosocial intervention in children and adolescents with autism spectrum disorder without intellectual disability: benefits of videotaping therapy sessions. Encéphale. in press
- Harms MB, Martin A, Wallace GL: Facial emotion recognition in autism spectrum disorders: a review of behavioural and neuroimaging studies. Neuropsychol Rev. 2010, 20: 290-322.View ArticlePubMedGoogle Scholar
- Grelotti DJ, Klin AJ, Gauthier I, Skudlarski P, Cohen DJ, Gore JC, Volkmar FR, Schultz RT: fMRI activation of the fusiform gyrus and amygdala to cartoon characters but not to faces in a boy with autism. Neuropsychologia. 2005, 43: 373-385.View ArticlePubMedGoogle Scholar
- Rosset D, Rondan C, Da Fonseca D, Santos A, Assouline B, Deruelle C: Typical emotion processing for cartoon but not for real faces in children with autistic spectrum disorders. J Autism Dev Disord. 2008, 38: 919-925.View ArticlePubMedGoogle Scholar
- Rosset D, Santos A, Da Fonseca D, Rondan C, Poinso F, Deruelle C: More than just another face in the crowd: evidence for an angry superiority effect in children with and without autism. Res Autism Spectr Disord. 2011, 5: 949-956.View ArticleGoogle Scholar
- Parsons S, Mitchell P: The potential of virtual reality in social skills training for people with autistic spectrum disorders. J Intellect Disabil Res. 2002, 46: 430-443.View ArticlePubMedGoogle Scholar
- Shane HC, Albert PD: Electronic screen media for persons with autism spectrum disorders: results of a survey. J Autism Dev Disord. 2008, 38: 1499-1508.View ArticlePubMedGoogle Scholar
- Bölte S: The ICF and its meaning for child and adolescent psychiatry. Z Kinder Jugendpsychiatr Psychother. 2009, 37: 495-497.View ArticlePubMedGoogle Scholar
- Silver M, Oakes P: Evaluation of a new computer intervention to teach people with autism or Asperger syndrome to recognize and predict emotions in others. Autism. 2001, 5: 299-316.View ArticlePubMedGoogle Scholar
- Williams C, Wright B, Callaghan G, Coughlan B: Do children with autism learn to read more readily by computer assisted instruction or traditional book methods? A pilot study. Autism. 2002, 6: 71-91.View ArticlePubMedGoogle Scholar
- Mitchell P, Parsons S, Leonard A: Using virtual environments for teaching social understanding to 6 adolescents with autistic spectrum disorders. J Autism Dev Disord. 2007, 37: 589-600.View ArticlePubMedGoogle Scholar
- Kandalaft MR, Didehbani N, Krawczyk DC, Allen TT, Chapman SB: Virtual reality social cognition training for young adults with high-functioning autism. J Autism Dev Disord. 2013, 43: 34-44.PubMed CentralView ArticlePubMedGoogle Scholar
- Wainer AL, Ingersoll BR: The use of innovative computer technology for teaching social communication to individuals with autism spectrum disorders. Res Autism Spectr Disord. 2011, 5: 96-107.View ArticleGoogle Scholar
- Grynspan O, Weiss PL, Perez-Diaz F, Gal E: Innovative technology-based interventions for autism spectrum disorders: a meta-analysis. Autism. in press
- Bölte S, Feineis-Matthews S, Leber S, Dierks T, Hubl D, Poustka F: The development and evaluation of a computer-based program to test and to teach the recognition of facial affect. Int J Circumpolar Health. 2002, 61: 61-68.View ArticlePubMedGoogle Scholar
- Golan O, Baron-Cohen S: Systemizing empathy: teaching adults with Asperger syndrome or high-functioning autism to recognize complex emotions using interactive multimedia. Dev Psychopathol. 2006, 18: 591-617.View ArticlePubMedGoogle Scholar
- Tanaka JW, Wolf JM, Klaiman C, Koenig K, Cockburn J, Herlihy L, Brown C, Stahl S, Kaiser MD, Schultz RT: Using computerized games to teach face recognition skills to children with autism spectrum disorders: the Let’s Face It! Program. J Child Psychol Psychiatry. 2010, 51: 944-952.View ArticlePubMedGoogle Scholar
- Hopkins IM, Gower MW, Perez TA, Smith DS, Amthor FR, Wimsatt FC, Biasini FJ: Avatar assistant: improving social skills in students with an ASD through a computer-based intervention. J Autism Dev Disord. 2011, 41: 1543-1555.View ArticlePubMedGoogle Scholar
- Golan O, Ashwin E, Granader Y, McClintock S, Day K, Leggett V, Baron-Cohen S: Enhancing emotion recognition in children with autism spectrum conditions: an intervention using animated vehicles with real emotional faces. J Autism Dev Disord. 2010, 40: 269-279.View ArticlePubMedGoogle Scholar
- Fombonne E: Epidemiological surveys of autism and other pervasive developmental disorders: an update. J Autism Dev Disord. 2003, 33: 365-382.View ArticlePubMedGoogle Scholar
- Baron-Cohen S, Golan O, Ashwin E: Can emotion recognition be taught to children with autism spectrum conditions?. Philos Trans R Soc Lond B Biol Sci. 2009, 364: 3567-3574.PubMed CentralView ArticlePubMedGoogle Scholar
- Dawson M, Soulières I, Gernsbacher MA, Mottron L: The level and nature of autistic intelligence. Psychol Sci. 2007, 18: 657-662.PubMed CentralView ArticlePubMedGoogle Scholar
- Morsanyi K, Holyoak KJ: Analogical reasoning ability in autistic and typically developing children. Dev Sci. 2010, 13: 578-587.View ArticlePubMedGoogle Scholar
- Soulières I, Dawson M, Gernsbacher MA, Mottron L: The level and nature of autistic intelligence II: what about Asperger Syndrome?. [http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0025372]
- Barnes KA, Howard JH, Howard DV, Gilotty L, Kenworthy L, Gaillard WD, Vaidya CJ: Intact implicit learning of spatial context and temporal sequences in childhood autism spectrum disorder. Neuropsychology. 2008, 22: 563-570.View ArticlePubMedGoogle Scholar
- Brown J, Aczel B, Jiménez L, Kaufman SB, Grant KP: Intact implicit learning in autism spectrum conditions. Q J Exp Psychol. 2010, 63: 1789-1812.View ArticleGoogle Scholar
- Nemeth D, Janacsek K, Balogh V, Londe Z, Mingesz R, Fazekas M, Jambori S, Danyi I, Vetro A: Learning in autism: implicitly superb. [http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0011731]
- Nielsen J: Usability Inspection Methods. 1994, New York: WileyView ArticleGoogle Scholar
- Rutter M, Le Couteur A, Lord C: ADI-R: The Autism Diagnostic Interview-Revised. 2003, Los Angeles: Western Psychological ServicesGoogle Scholar
- Lord C, Rutter M, Di Lavore PC, Risi S: Autism Diagnostic Observation Schedule (ADOS). 2000, Los Angeles: Western Psychological ServicesGoogle Scholar
- Gregoire J: L’examen Clinique de L’intelligence de L’enfant. Fondements et Pratiques du WISC-IV. 2006, Liège: MardagaGoogle Scholar
- Lecocq P: Epreuve de Compréhension Syntaxico-Sémantique. 1996, Paris: Presse Universitaire du SeptentrionGoogle Scholar
- Lefabvrais P: Description, définition et mesure de la dyslexie. Utilisation du test ‘L’Alouette’. Rev Psychol Appl. 1965, 15: 33-44.Google Scholar
- Minio-Paluello I, Baron-Cohen S, Avenanti A, Walsh V, Aglioti SM: Absence of embodied empathy during pain observation in Asperger syndrome. Biol Psychiatry. 2009, 65: 55-62.View ArticlePubMedGoogle Scholar
- Plutchik R: Emotion: Theory, Research, and Experience. Volume 1, Theories of Emotion. 1980, New York: Academic PressGoogle Scholar
- Gibbs R, Leggitt J, Turner E: Why figurative language is special in emotional communication. The Verbal Communication of Emotions. Edited by: Fussell S. 2002, New Jersey: Erlbaum, 125-150.Google Scholar
- Baron-Cohen S: Social and pragmatic deficits in autism: cognitive or affective?. J Autism Dev Disord. 1988, 18: 379-402.View ArticlePubMedGoogle Scholar
- Blattner MM, Sumikawa DA, Greenberg RM: Earcons and icons: their structure and common design principles. Human-Computer Interaction. 1989, 4: 11-44.View ArticleGoogle Scholar
- Brewster SA: Providing a Structured Method for Integrating Non-speech Audio into Human-computer Interfaces. 1994, UK: PhD Thesis, University of York, YorkGoogle Scholar
- Dunn W: The Sensory Profile: Examiner’s Manuel. 1999, San Antonio, TX: The Psychological CorporationGoogle Scholar
- Schwebel DC, McClure LA, Severson J: Usability and feasibility of an Internet-based virtual pedestrian environment to teach children to cross streets safely. Virtual Real. 2014, 18: 5-11.PubMed CentralView ArticlePubMedGoogle Scholar
- Langner O, Dotsch R, Bijlstra G, Wigboldus DHJ, Hawk ST, Van Knippenberg A: Presentation and validation of the Radboud Faces Database. Cogn Emot. 2010, 24: 1377-1388.View ArticleGoogle Scholar
- Dunlap G: The influence of task variation and maintenance tasks on the learning of autistic children. J Exp Child Psychol. 1984, 37: 41-64.View ArticlePubMedGoogle Scholar
- Spreckley M, Boyd R: Efficacy of applied behavioral intervention in preschool children with autism for improving cognitive, language, and adaptive behavior: a systematic review and meta-analysis. J Pediatr. 2009, 154: 338-344.View ArticlePubMedGoogle Scholar
- Kourkoulou A, Leekam SR, Findlay JM: Implicit learning of local context in autism spectrum disorder. J Autism Dev Disord. 2012, 42: 244-256.View ArticlePubMedGoogle Scholar
- Schuller B, Marchi E, Baron-Cohen S, O’Reilly H, Pigat D, Robinson P, Daves I: The state of play of ASC-Inclusion: an integrated Internet-based environment for social inclusion of children with autism spectrum conditions. [http://arxiv-web3.library.cornell.edu/abs/1403.5912]
- Bailly K, Chen L, Bisiaux C, Serret S:JEMImE: Serious game for children with autistic spectrum disorders based on Multimodal Emotion Imitation (ANR Contint ref: ANR-13-CORD-0004 2013). 2013, [http://www.agence-nationale-recherche.fr/suivi-bilan/sciences-et-technologies-de-l-information-et-de-la-communication/contenus-numeriques-et-interactions/fiche-projet-contint/?L=response.write(9597039*9069797)%26tx_lwmsuivibilan_pi2%5BCODE%5D=ANR-13-CORD-0004],Google Scholar
- Trevarthen C, Delafield-Butt JT: Autism as a developmental disorder in intentional movement and affective engagement. Front Integr Neurosci. 2013, 10: doi:10.3389/fnint.2013.00049Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.