A Storytelling Robot for Pediatric Rehabilitation

Catherine Plaisant #, Allison Druin#%, Corinna Lathan*&,
Kapil Dakhane#, Kris Edwards*, and Jack Maxwell Vice*

#Human-Computer Interaction Laboratory
#University of Maryland Institute for Advanced Computer Studies
% University of Maryland College of Education
* AnthroTronix, Inc. &The Catholic University of America Dept of Biomedical Engineering

(contact : plaisant@cs.umd.edu)

A revised version will appear in the Proceedings of ASSETS'2000 , ACM, New York

see project page: http://www.cs.umd.edu/hcil/anthro



We are developing a prototype storytelling robot for use with children in rehabilitation. Children can remotely control a large furry robot by using a variety of body sensors adapted to their disability or rehabilitation goal. In doing so, they can teach the robot to act out emotions (e.g. sad, happy, excited) and then write stories using the storytelling software and include those emotions in the story. The story can then be "played" by the remote controlled robot, which acts out the story and the emotions. We believe that this robot can motivate the children and help them reach their therapy goals through therapeutic play, either by exercising muscles or joints (e.g. for physically challenges children) or by reflecting on the expression of emotions (e.g. for autistic children). We use an innovative design methodology involving children as design partners.

Keywords: Therapeutic play, robot, children, user interface, design process, rehabilitation


As advances in technology continue, people with disabilities are benefiting from greater availability of assistive technology. Assistive technology has been traditionally credited for increasing independence, improving the quality of life and providing an overall boost in self-esteem for people with disabilities by making the most of an individual’s abilities. Assistive technology, when strategically applied, may also provide therapeutic benefits. Hilker et al. (1995) believes the rationale for this concept is based on an overlap between assistive technology use and physical therapy interventions. Our work pushes that theory, by making use of assistive technology as a motivating environment to improve basic functions associated with physical therapy goals.

Video games have been used in the past for a variety of therapeutic and rehabilitation situation (Griffiths 1997) and several studies - mainly set in a clinical environment - have shown how those games can be used as a mean to achieve desired functional goals (e.g. Vilozni, et al. 1994). The rich literature that describes the use of puppetry as a therapeutic, educational, and social tool (e.g., White, J.& Allers, C. T.,1994, Vidler, 1972; Currant, 1985; Carter, 1987; Caputo, 1993) also shows the benefits of technology. For example researchers have found that the use of puppets as a mediating technology can produce more expressive communication patterns among children who are in crisis or have on-going emotional, behavioral or academic challenges. Technology can be used by a child to confront what may be too painful or difficult to express otherwise. By telling a "story" with these props, children can "play out" their feelings just as an adult may "talk out" their difficulties with a therapist (Cochran, 1996).

Here we introduce a storytelling robot technology for therapeutic play with the hope that it will provide a richer environment useful for children with a variety of challenges, as well as the long term motivation needed to reach children's therapeutic goals.

The partners - This work is the result of the collaboration of a university laboratory and a small startup company specializing in adaptive technology.

The University of Maryland Human-Computer Interaction Laboratory (HCIL) has done extensive work in interaction design and evaluation with a unique program focusing on the design of children's technology (Druin, 1999a; Druin, 1999b). One of the more unique projects to come out of this work has been a prototype robot for storytelling (Druin, et al., 1999; Montemayor et al., 2000), which demonstrated the benefits of working with children as active partners in the design of new technology. This storytelling robot was originally designed for all children, without consideration for special challenges. Children using the storytelling software could write stories and select emotions from a list of pre-programmed emotions (e.g. happy, sad, surprised). The story could then be "played" by the remote controlled robot, which told the story (playing recorded or synthesized speech) and acted out the emotions at the appropriate time in the story.

The collaboration with AnthroTronix is broadening the scope of our storytelling robot research to emphasize designing tools for the purpose of children's rehabilitation.

AnthroTronix is a startup company; part of the Rehabilitation Engineering Research Center led by The Catholic University of America and the National Rehabilitation Hospital. AnthroTronix designs and develops robotics and virtual reality interfaces to support children’s rehabilitation. The company has developed a system that augments human function by providing personal augmentation devices (PADs), controlled by physiological signals, for the purpose of navigating and manipulating the external environment under their control. The system utilizes a variety of sensors connected to detect input (or "gestures") from the user. Software is then used to assess and process this input, and to map the input to appropriate outputs. The information fed back to the user depends on the intended outcome. For example, the systems can be used to train the user to learn input-output mappings, to enhance performance, and to assess performance.

This paper first reports on a pilot study using a video game with AnthroTronix assistive technology, which illustrates the potential effectiveness of therapeutic play and some of its limitations. Then we describe how we extended the storytelling robot to enable children to directly control the robot, to "teach" the robot how to act out the emotions. This was accomplished using AnthroTronix assitive technology, allowing for the easy adaptation of sensors to support children's abilities or challenges. We believe that a robot can provide a strong motivation for children to reach their therapy goals by exercising the appropriate physical ability or emotional expression.



Our work with a storytelling robot was motivated in part by the promising results of a case study using AnthroTronix assistive technology with an off-the-shelf videogame.

A study was conducted by Rinaldi with children who have mild to moderate cerebral palsy (Rinaldi, 2000). Four children were monitored using a SuperNintendo video game system. The game was used with adaptive inputs, which challenged the participants’ isolated motor control and range of motion. For example, instead of pressing a button to make a game character jump over an obstacle, children had to bend their arm beyond a certain threshold. The input sensors were designed to provide alternative switch access by allowing the switch threshold to be adjusted as the child improves or changes.

The games were used the children at home during a 12-week period. Therapists gave the children exercises and the use of the game was recorded. Strength and range of motion were recorded at 6 weeks interval along with other qualitative information. The findings indicate an overall improvement in range of motion with game use but no changes in strength. The parents indicated that the response to this type of therapy was positive. Everyone enjoyed having the system at home and felt it was safe and easy to use. This case study highlights the benefit of therapeutic interventions that are administered in a challenging, familiar and fun environment that is integrated into a child's daily activities.

Feedback from children and parents made it clear that using the Nintendo game broke the routine therapy exercises but also suggests that the initial excitement might fade after a few weeks. This of course highlights the need for longer studies but it was conjectured that two factors played a role in the decline of interest after 12 weeks. First, an arm movement is slower than the press of a button, which resulted in poor performance in the game, a great disappointment for the children. Second, it was conjectured that the repetitiveness and the complete lack of creativity in the Nintendo activity might be a hurdle in any attempt to alleviate the boredom of daily therapeutic exercise.

This leads us to believe that a creative storytelling robot might provide a long-term motivation that is needed to help children in rehabilitation.



The pre-existing technology - PETS - Personal Electronic Teller of Stories:

At the University of Maryland, we built a robot called PETS: A Personal Electronic Teller of Stories (see Figure1). Children can build a robotic pet by simply snapping together special robotic animal parts (e.g., dog paws, a fish tail, duck feet ). After a robotic pet is built, children can tell stories with the My PETS software, giving their robot emotions (e.g., excited, sad, lonely) to act out throughout their story (Druin et al., 1999; Montemayor et al., 2000). A 7-year old girl in Maryland created the example story below.


Figure 1: The PETS robot prototypes


This story was entitled Michelle:

There once was a robot named Michelle. She was new in the neighborhood. She was HAPPY (robot behaves happy) when she first came, thinking she would make friends. But it was the opposite. Other robots threw rocks and sticks. She was SAD (robot behaves sad). No one liked her. One day she was walking down a street, a huge busy one, when another robot named Rob came up and asked if she wanted to have a friend, but then realized she was HAPPY (robot behaves happy). The other robots were ANGRY (robot behaves angry) but knew that they had learned their lesson. Michelle and Rob lived HAPPILY (robot behaves happy) ever after. No one noticed the dents from rocks that stayed on Michelle

(Research notes, August 1998).

The initial PETS robotic animal parts were built with LEGO bricks covered in fur, feathers, felt, etc. Further refinements have led to a skeletal structure built from metal, plastic, and polycarbonate materials. These have been covered in a foam outer-shell with brightly colored felt and fur. Each animal part can be snapped into place on the body and is also plugged into a plugbox embedded in the animal’s torso. This plugbox is an interface to a Handyboard controller also in the animal’s body, which controls servos and motors, and can read inputs from sensors that are attached throughout the robot’s body. This controller has a serial connection to a Macintosh computer. In the Macintosh, the application software layer, My PETS, takes a story written by children, translates and transfers it to the system software layer that resides in the Handyboard.

Overall design methodology

The name, concept, and the development of the PETS prototypes came about because of children. We develop new technologies for children, with children in an "intergenerational design team." This current team consists of children (ages 7-11 years old) and adult professionals, with experience in computer science, education, art, and robotics. Our research goals include creating new technologies as well as understanding better ways to give children a voice in the technology development process. An outcome of our research has been to develop the methodology of "cooperative inquiry," research methods that support the partnership of children and adults in developing new technologies (Druin, 1999a).

For this new part of the project our design team is also comprised of rehabilitation staff. The early phase of the design has been conducted solely with our existing team of child design partners. Children with disabilities will join the team later as informants during the refinement of the design and of course as testers of the refined prototypes. One of our goals is to design a product that will have the potential of being both a commercial toy for all children and a rehabilitation tool for children with special challenges through the addition sensors and the development of the rehab methodology.

Feedback from the design sessions with the children partners

From the design sessions with our child design partners emerged several findings:

First it was made clear that the storytelling robot was a very powerful tool to engage children in discussing emotions, their expression or the risks of misinterpreting expressions of emotions, leading us to believe that PETS could be used as a rehabilitation tool for children with autism.


Figure 2

Sensors mounted in arm bands and in a hat allow a user to directly control the robot.

The movements can be recorded and saved to be used later in stories.

Second, despite the great excitement created by the robot when used by the children, one of the most obvious limitations of the initial prototype was that the emotions—or any other movement of the robot—had to be programmed and "pre-stored" in the software in order to be used by the children for their stories. Originally, software was created to control the robot directly via a graphical user interface. But it soon became obvious that a more natural way to operate the robot was to enable children to teach the robot through their own body motions.

A first design session was organized during which children tested a car remotely controlled with an instrumented glove. The children all enjoyed the direct control of the car (comparing it to the buttons or joysticks), and liked the absence of wires between the car and the glove. On the other hand, they also found the glove uncomfortable but most importantly found the proposed mapping between the body movement and the car movement very confusing. They had to bend their wrist up or down to go forward or backward, and spin their wrist right of left to turn, but constantly confused and combined the movements which send the car in seemingly random direction. This session clearly highlighted the need for a direct correspondence between body motion and the motion of the robot. Ideas proposed by the team included:

Based on these ideas, a new prototype interface was developed with sensor controls of the storytelling robot. Sensors (accelerometers) were imbedded in two armbands - to control the 2 arms of the robot- and in a hat to control the head. An additional sensor (built in a shoe sole) allows the entire robot to spin when foot pressure is applied to the front of the shoe (this is clearly not the best way to control a spin but additional refinements will be made to this in the near future).

Early feedback from about 3 hours of trials suggest that our design team children are able to control the robot and that excitement of using the robot for storytelling was heightened by the increased sense of control over the robot. An unexpected observation is that both children and adults’ attention is strongly attracted to the robot - and not to the person controlling the robot. After a few adjustments and corrections of the control system, children were able to control individual movements of the robot, while the randomness of the movements of the robot—when the children was not precisely controlling the robot—gave it a compelling liveliness.

The controls can be recalibrated to match the amplitude of the child’s movements and can also be reassigned. For example, the arm bands can be inverted to control the arm side directly (kid left arm to robot left arm) or in a mirror fashion (kid left arm to robot right arm) which seemed a better initial setup. Of course the sensors could also be used on other body parts to match specific rehabilitation needs (e.g., it could be placed on a foot to exercise an ankle while controlling the robot.)

User Scenarios

Our original prototype is very rudimentary. More controls need to be added and more flexibility is needed in matching sensors and robot movements before we can successfully work with children with disabilities. Wireless connections from the computer to the sensors are needed, the size of the robot needs to be reduced to facilitate a robot’s motion on a table or floor, but we can start develop scenarios of use. Below are 2 examples and we are now working with professional rehabilitation experts to develop other strategies and games.

1- Physical therapy

Max has cerebral palsy. To learn to feed himself he needs to do a lot of supination and pronation exercises to strengthen his arms. Today’s standard of care makes him repeat the same exercise 100 times. Five-year old Max finds the exercises extremely boring and often refuses to finish them. But today the therapist tells him "Do you have a pet? Let's tell a story about it together... and the robot will act out the story for you." As the child start talking about the story the therapist equips the child with an elbow pad embedded with input devices and shows the gestures that will make the robot act "happy" of "confused" when appropriate in the story. For example, arm extensions could be used to make the robot move around the space, and arm rotation to turn it around. Max is now engaged in a creative endeavor and is motivated to do the exercises. After much practice and many revisions the story is recorded on video to take home. Next time Max will learn to teach the robots new tricks, associated with new movements; eventually he could control the robot remotely from home and tell the story to the children in the waiting room.

2- Emotional Expression for Children with Autism

Sue is greeted by his therapist, "Sue, are you interested in telling a story today?" Together they pull out the robot toolkit. The therapist asks, "What character do you want to tell a story with today?" And Sue answers, "Benny," as she pulls out the bird wings, purple horns, and flying saucer. Slowly Sue builds her stuffed robot caracter. Then she moves to the computer and begins to write a story, "My name is Benny and I am SAD." She presses the "tell me a story" button and the robot tells the story and acts sad. The therapist asks Sue, "You know, I'm not sure the robot looks really sad. Can you teach the robot to really look sad?" Then she takes out their robot sensors and they attach them to Sue’s arm and leg. Sue then moves toward the wall. The Therapist asks, "Is that the only way to look sad?" Together they talk about what it means to be sad and try different movements and expressions. Late in the session, the therapist asks, "Sue, can we work on making Benny look happy? I think I'm tired of sad." Sue then goes to the computer and writes a story about Benny, "My name is Benny and I am HAPPY. Benny likes his therapist."



We are now revising our prototype and it is our goal to have at least one of the above scenarios tested during summer. We anticipate that by Fall 2000 (the time of the conference), we will be revising the robot hardware and software to respond to the needs of rehabilitation elicited during the summer testing sessions.

We anticipate producing a robot toolkit that includes a base robot, accessories for customizing the robot (paws, feathers, sound), a rich set of input devices to control the robot adapted to the range of abilities of children in rehabilitation. Software will also be needed to monitor usage. After the usual cycle of formative evaluation and iterative design, we will develop a methodology for rehabilitation staff to provide therapeutic play, including training, examples of use, preparation, and storytelling techniques. Measuring the effectiveness of such therapeutic play with thorough scientific evaluations is our main challenge for the coming year.



We believe that this new type of technology has the potential to help motivate children to reach their therapy goals through therapeutic play. It has been argued that for therapeutic treatment to be successful it has to take into account the dynamic of the family (Ross & Thomas, 1993). Storytelling appears to offer a solid motivating direction as it is a typical home activity (with relatives, siblings or friends). Play can give children with disability a sense of competence and control over environmental circumstances. It can also help a child learn new skills.

We have built a working prototype, scenarios have been developed to illustrate how the prototype could be used and we are ready for testing with challenged children. We believe that our innovative design methodology involving participatory design and children as design partners will lead to a product that will truly excite children's interest as well as support the rehabilitation experience.



We want to thank the Maryland Industrial Partnership program for partial support for this research.


Caputo, R. A. (1993). Using puppets with students with emotional and behavioral disorders. Intervention in School and Clinic, 29 (1), pp. 26-30.

Carter, S. (1987). Use of puppets to treat traumatic grief: A case study. Elementary School Guidance and Counseling, 21(3), pp. 210-215.

Cochran, J. L. (1996). Using play and art therapy to help culturally diverse students overcome barriers to school success. School Counselor, 43, pp. 287-298.

Currant, N. (1985). The expansive educational value of puppets. Academic Therapy, 21(1), pp. 55-60.

Druin, A. (1999a). Cooperative Inquiry: Developing New Technologies for Children with Children, Proceedings of Conference on Human Factors in Computing Systems (CHI'99), Pittsburgh, PA, USA, May 15-20, ACM, New York, 592-599

Druin, A. (ed.) (1999b). The design of children’s technology. San Francisco, CA: Morgan Kaufmann

Druin, A., Montemayor, J., Hendler, J., McAlister, B., Boltman, A., Fiterman, E., Plaisant, A., Kruskal, A., Olsen, H., Revett, I., Schwenn, T.P., Sumida, L. and Wagner, R., (1999). Designing PETS: A Personal Electronic Teller of Stories Proceedings of Conference on Human Factors in Computing Systems (CHI'99), Pittsburgh, PA, USA, May 15-20, ACM, New York, 326-329

Griffiths, M (1997). Video games and clinical practice: issues, uses and treatments. Br J Clin Psychol Nov;36, 639-41

Herman-Hilker, S L; Hilker, D F; Levine, S P. (1995). Achievement of physical therapy goals as a result of assistive technology use. RESNA, pp. 59-61

Montemayor, J., Druin, A., and Hendler, J., (2000). PETS: A Personal Electronic Teller of Stories , in A. Druin & J. Hendler (Eds.) Robots for Kids: New Technologies for Learning. Morgan Kaufmann, San Francisco, CA

Rinaldi, M. C. (2000). Using Adaptive Video Game Technology as Therapeutic Tool, Master of Biomedical Engineering Thesis, The Catholic University of America School of Engineering.

Ross, K; Thomson, D. (1993). An evaluation of parents' involvement in the management of their cerebral palsy children. Physiotherapy, August 1993, Volume 79, no. 8, 561-565.

Vidler, V. (1972). Use of puppets to reach the emotionally disturbed. Instructor, 81(9).

Vilozni D; Bar-Yishay E; Gur I; Shapira Y; Meyer S; Godfrey S (1994) Computerized respiratory muscle training in children with Duchenne muscular dystrophy. May;4(3), 249-55

White, J. & Allers, C. T. (1994). Play therapy with abused children: A review of the literature. Journal of Counseling and Development, 72, pp. 390-394.