Jacqueline Kory-Westlund

Social robots as language learning companions for children

Language learning is, by nature, a social, interactive, interpersonal, activity. Children learn language not only by listening, but through active communication with a social actor. Social interaction is critical for language learning.

Thus, if we want to build technology to support young language learners, one intriguing direction is to use robots. Robots can be designed to use the same kinds of social, interactive behaviors that humans use—their physical presence and embodiment give them a leg up in social, interpersonal tasks compared to virtual agents or simple apps and games. They combine the adaptability, customizability, and scalability of technology with the embodied, situated world in which we operate.

The robot we used in these projects is called the DragonBot. Designed and built in the Personal Robots Group, it's a squash-and-stretch robot specifically designed to be an expressive, friendly creature. An Android phone displays an animated face and runs control software. The phone's sensors can be used to capture audio and video, which we can stream to another computer so a teleoperator can figure out what the robot should do next, or, in other projects, as input for various behavior modules, such as speech entrainment or affect recognition. We can stream live human speech, with the pitch shifted up to sound more child-like, to play on the robot, or playback recorded audio files.

Here is a video showing the original DragonBot robot, with a brief rundown of its cool features.

Social robots as informants

This was one of the very first projects I worked on at MIT! Funded by an NSF cyberlearning grant, the goal of this study and the studies following were to explore several questions regarding preschool children's word learning from social robots, namely:

• What can make a robot an effective language learning companion?
• What design features of the robots positively impact children's learning and attitudes?

In this study, we wanted to explore how different nonverbal social behaviors impacted children's perceptions of the robot as an informant and social companion.

We set up two robots. One was contingently responsive to the child—e.g., it would look at the child when the child spoke, it might nod and smile at the right times. The other robot was not contingent—it might be looking somewhere over there while the child was speaking, and while it was just as expressive, the timing of its nodding and smiling had nothing to do with what the child was doing.

For this study, the robots were both teleoperated by humans. I was one of the teleoperators—it was like controlling a robotic muppet!

Each child who participated in the study got to talk with both robots at the same time. The robots presented some facts about unusual animals (i.e., opportunities for the child to learn). We did some assessments and activities designed to give us insight into how the child thought about the robots and how willing they might be to learn new information from each robot—i.e., did the contingency of the robot's nonverbal behavior affect whether kids would treat the robots as equally reliable informants?

We found that children treated both robots as interlocutors and as informants from whom they could seek information. However, children were especially attentive and receptive to whichever robot displayed the greater nonverbal contingency. This selective information seeking is consistent with other recent research showing that children are, first, quite sensitive to their interlocutor's nonverbal signals, and use those signals as cues when determining which informants they question or endorse.

In sum: This study provided evidence that children show sensitivity to a robot's nonverbal social cues, like they are with humans, and they will use this information when deciding if a robot is a credible informant, as they do with humans.

Links

• Video showing this project
• This work is discussed on the Personal Robots Group site and the MIT Media Lab site
• I wrote about this project for the MIT Media Lab blog: Robots for Kids: Designing Social Machines That Support Children's Learning.
• That blog post was republished on IEEE Spectrum: Robots for Kids: Designing Social Machines That Support Children's Learning.

Publications

• Breazeal, C., Harris, P., DeSteno, D., Kory, J., Dickens, L., & Jeong, S. (2016). Young children treat robots as informants. Topics in Cognitive Science, pp. 1-11. [PDF]

• Kory, J., Jeong, S., & Breazeal, C. L. (2013). Robotic learning companions for early language development. In J. Epps, F. Chen, S. Oviatt, & K. Mase (Eds.), Proceedings of the 15th ACM on International conference on multimodal interaction, (pp. 71-72). ACM: New York, NY. [on ACM]

Word learning with social robots

We did two studies specifically looking at children's rapid learning of new words. Would kids learn words with a robot as well as they do from a human? Would they attend to the robot's nonverbal social cues, like they do with humans?

Study 1: Simple word learning

This study was pretty straightforward: Children looked at pictures of unfamiliar animals with a woman, with a tablet, and with a social robot. The interlocutor provided the names of the new animals—new words for the kids to learn. In this simple word-learning task, children learned new words equally well from all three interlocutors. We also found that children appraised the robot as an active, social partner.

In sum: This study provided evidence that children will learn from social robots, and will think of them as social partners. Great!

With that baseline in place, we compared preschoolers' learning of new words from a human and from a social robot in a somewhat more complex learning task...

Study 2: Slightly less simple word learning

When learning from human partners, children pay attention to nonverbal signals, such as gaze and bodily orientation, to figure out what a person is looking at and why. They may follow gaze to determine what object or event triggered another's emotion, or to learn about the goal of another's ongoing action. They also follow gaze in language learning, using the speaker's gaze to figure out what new objects are being referred to or named. Would kids do that with robots, too? Children viewed two images of unfamiliar animals at once, and their interlocutor (human or robot) named one of the animals. Children needed to monitor the interlocutor's non-verbal cues (gaze and bodily orientation) to determine which picture was being referred to.

We added one more condition. How "big" of actions might the interlocutor need to do for the child to figure out what picture was being referred to? Half the children saw the images close together, so the interlocutor's cues were similar regardless of which animal was being attended to and named. The other half saw the images farther apart, which meant the interlocutor's cues were "bigger" and more distinct.

As you might expect, when the images were presented close together, children subsequently identified the correct animals at chance level with both interlocutors. So ... the nonverbal cues weren't distinct enough.

When the images were presented further apart, children identified the correct animals at better than chance level from both interlocutors. Now it was easier to see where the interlocutor was looking!

Children learned equally well from the robot and the human. Thus, this study provided evidence that children will attend to a social robot's nonverbal cues during word learning as a cue to linguistic reference, as they do with people.

Links

• Video showing this project
• This work is discussed on the Personal Robots Group site and the MIT Media Lab site
• I wrote about this project for the MIT Media Lab blog: Making new (robot) friends: Understanding children's relationships with social robots.
• That blog post was republished on IEEE Spectrum: Robots for Kids: Designing Social Machines That Support Children's Learning.

Publications

• Kory-Westlund, J., Dickens, L., Jeong, S., Harris, P., DeSteno, D., & Breazeal, C. (2015). A Comparison of children learning from robots, tablets, and people. In Proceedings of New Friends: The 1st International Conference on Social Robots in Therapy and Education. [talk] [PDF]

• Kory-Westlund., J. M., Dickens, L., Jeong, S., Harris, P. L., DeSteno, D., & Breazeal, C. L. (2017). Children use non-verbal cues to learn new words from robots as well as people. International Journal of Child-Computer Interaction. [PDF]

Click here to see the video showing this project!

Study Overview

For my master's thesis at the MIT Media Lab, I created a social robotic learning companion that played a storytelling game with young kids.

Children’s oral language skills in preschool can predict their academic success later in life. Helping children improve their language and vocabulary skills early on could help them succeed later. Furthermore, language learning is a highly social, interactive activity. When creating technology to support children's language learning, technology that leverages the same social cues and social presence that people do—such as a social robot—will likely provide more benefit than using technology that ignores the critical social aspects of language learning.

As such, in this project, I examined the potential of a social robotic learning companion to support children's early long-term language development.

Study

The robot was designed as a social character, engaging children as a peer, not as a teacher, within a relational, dialogic context. The robot targeted the social, interactive nature of language learning through a storytelling game that the robot and child played together. The game was on a tablet—the tablet showed a couple characters that the robot or child could move around while telling their story, much like digital stick puppets. During the game, the robot introduced new vocabulary words and modeled good story narration skills.

Furthermore, because children may learn better when appropriately challenged, we asked whether a robot that Matched the “level” of complexity of the language it used to the general language ability of the child might help children improve more. For half the children, the robot told easier or harder stories based on an assessment of the child’s general language ability.

17 preschool children played the storytelling game with the robot eight times each over a two-month period.

I evaluated children's perceptions of the robot and the game, as well as whether the robot's matching influenced (i) whether children learned new words from the robot, (ii) the complexity and style of stories children told, and (iii) the similarity of children’s stories to the robot’s stories. I expected that children would learn more from a robot that matched, and that they would copy its stories and narration style more than they would with a robot that did not match. Children’s language use was tracked across sessions.

Results

I found that all children learned new vocabulary words, created new stories during the game, and enjoyed playing with the robot. In addition, children in the Matched condition maintained or increased the amount and diversity of the language they used during interactions with the robot more than children who played with the Unmatched robot.

Understanding how the robot influences children’s language, and how a robot could support language development will inform the design of future learning/teaching companions that engage children as peers in educational play.

Links

• This work is also on the Personal Robots Group site and the MIT Media Lab site
• New Scientist covered this work: Kindergarten bots teach language to tots
• I wrote about this project for the MIT Media Lab blog: Making new (robot) friends: Understanding children's relationships with social robots.
• That blog post was republished on IEEE Spectrum: Robots for Kids: Designing Social Machines That Support Children's Learning.

Publications

• Kory, J. (2014). Storytelling with robots: Effects of robot language level on children's language learning. Master's Thesis, Media Arts and Sciences, Massachusetts Institute of Technology, Cambridge, MA. [PDF]

• Kory, J., & Breazeal, C. (2014). Storytelling with Robots: Learning Companions for Preschool Children’s Language Development. In P. A. Vargas & R. Aylett (Eds.), Proceedings of the 23rd IEEE International Symposium on Robot and Human Interactive Communication (RO-MAN). IEEE: Washington, DC. [PDF]

• Kory-Westlund, J., & Breazeal, C. (2015). The Interplay of Robot Language Level with Children's Language Learning during Storytelling. In J. A. Adams, W. Smart, B. Mutlu, & L. Takayama (Eds.), Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction: Extended Abstracts (pp. 65-66). [on ACM]

• Kory-Westlund, J. (2015). Telling Stories with Green the DragonBot: A Showcase of Children's Interactions Over Two Months. In J. A. Adams, W. Smart, B. Mutlu, & L. Takayama (Eds.), Proceedings of the Tenth Annual ACM/IEEE International Conference on Human-Robot Interaction: Extended Abstracts (p. 263). [on ACM] [PDF] [Video] Winner of Best Video Award.

• Kory-Westlund, J. M., & Breazeal, C. (2019). Exploring the effects of a social robot's speech entrainment and backstory on young children's emotion, rapport, relationships, and learning. Frontiers in Robotics and AI, 6. [PDF] [online]

Invitations

For my wedding, I designed my own invitations. Here's the art: line drawings of birch trees (one of Randy's favorite kinds of trees), a flowery vine wrapping around the edge, and a few butterflies (but look closely: one is actually a penguin!). I used black micron pens.

I took a photo of the page, opened it in GIMP, added all the invitation details with lovely text boxes in some fancy script, and printed them out on nice paper.

Thank you cards

The thank you cards I sent were similarly my own design—in this case, a line drawing of a magnolia flower, chosen because I had a nice reference photo of a magnolia flower on hand. These were printed on regular printer paper and folded up in fourths.

Cake toppers

While we considered being nerds and putting a Han Solo / Leia combo on top of the cake, we ultimately picked classy penguins. I made these with sculpey clay.

Centerpieces

The centerpieces for our reception were arguably the most elaborate part of my wedding crafts, simply due to the sheer number of moving parts!

Here's a reference photo of the setup for one of the tables.

Each table got five sheets of scrapbook paper, arranged in approximately a circle. In the middle, there were three jars. One held paper flowers; one held a candle; one held a wire tree covered in little monster finger puppets. All around that, we scattered materials for decorating the scrapbook pages: some pens, markers, crayons, and stickers. Somewhere in there was the table's placard—in this picture, Unity. That was our table. The other tables had names like Bliss, Felicity, etc.

The paper flowers were my solution to not having a florist. They went well with the crafty theme and my ribbon bouquet! The vases were reappropriated food jars that had interesting shapes—molasses, maple syrup, raspberry syrup, and so forth. I filled each with paper confetti strips and put three flowers in each.

Here are all the flowers:

The candles were fairly straightforward. I used modpodge to attach little pieces of semi-transparent paper onto reused peanut butter jars. I put a little 1x2in blue candle inside each one.

The last piece was the monsters. My older sister provided about a hundred felt finger puppet monsters she had made. I painted jars with turquoise, yellow, pink, and green glass paint. Then, I used green floral wire to make curly wire "trees" that sat inside. The monsters perched on top, one to a branch. I added a little sign saying "Adopt a monster!" so my guests would know they could take one home.

And two more photos from our reception:

Dress

Picking a dress for a once-in-a-lifetime event is hard. I browsed dresses online. There were so many options. Did I want to wear something huge and white and over-the-top poofy? Something slim and sleek? I did want to wear a dress. I didn't mind the idea of it being white.

Fortunately, my mom still had her wedding dress. Fancy, white, and lacy, it was made in the '80s and had a train longer than I am tall. "You can have it if you want it," she told me. "That's why I saved it."

I liked the idea of wearing a family heirloom. There was something nice about that. Also the part where I didn't have to pick the style—the dress was already made and it was what it was.

"Okay!" I said. My mom mailed me the dress. I tried it on. The long, lacy sleeves were a bit much... especially for a June wedding in Boston. I didn't want to sweat through the sleeves. I had a tailor remove them.

The next step was to make the dress more "me." And that meant adding color!

The dress had a petticoat. Very poofy. I dyed it turquoise—fabric dye from the craft store and a plastic bin in my bathtub did the trick.

The dress also had a long, lacy veil, with lacy flowers all along the edges. I bought fabric dye in bright orange and magenta, and spent several afternoons carefully painting the flowers in alternating colors.

Much better.

Here's the full effect with the train and veil!

Shoes

The dress was just a little long (my mom is two inches taller than me). I didn't want to hem it, so I had to wear heels. Finding a good pair of comfy heels was tricky ... my feet are wide; most decent-looking heels don't fit. I wanted colorful shoes to go with my colorful theme. After browsing, a lot, I found these chunky wedges at some local store. The turquoise was a good base. I added thing ribbon bows along the back, and aded more ribbons and ribbon flowers over the toes. Perfect!

Ribbon-flower bouquet

I didn't eschew real flowers because I hate flowers. I like flowers. But I wanted to make my own bouquet. (And I didn't want to add "florist" to the list of other things I had to deal with before the wedding—I was busy enough trying to finish my master's thesis!)

I spent several weeks browsing other people's homemade bouquets online. Mine needed to be bright, colorful, and classy. I tried make a few different kinds of flowers—including with buttons, beads, and paper—before settling on ribbon roses as my favorite.

I ordered a bunch of satin ribbon online. A bunch. Bulk ribbon was cheaper. So much ribbon. Green, silver, purple, pink, purple, orange, turquoise, yellow with polka dots, shiny transparent pink, shiny transparent turquoise.... you get the picture. Most of it was 1.5-inch ribbon, which made a good size flower.

Then, settling down for many evenings watching random shows on Netflix, I made a bunch of ribbon flowers. I attached each flower to a green wire stem, made of floral wire, that I bought at Michael's. When I had a huge pile of flowers, I arranged the bouquet. Twisting the stems together held them in place, mostly.

I don't have great pictures of the work in progress, unfortunately! Here's all the flowers put together, after the bouquet was finished.

The next step was to hide the kind of ugly wire steps. I wrapped green ribbon around the stems and glued the ribbon in place with some tacky glue.

I cut turquoise felt into a flowery circle and used that to hide the rest of the stems.

I tied a couple long pieces of thin turquoise ribbon and more of the green ribbon around the base of the stems.

Voila! Here's the finished product.

Boutonnieres and corsages

For the rest of the wedding party, I made mini ribbon flowers for their boutonnieres and corsages. The flowers were made the same way as for my bouquet, but with 1-inch ribbon. Again, I used green ribbon to hide the stems.

Boutonnieres:

Corsages:

New (old) project!

I've finally added a page about my summer at NASA Goddard Space Flight Center in 2011! I worked with over forty interns at Mike Comberiate's Engineering Boot Camp. The project I worked on was called LARGE: LIDAR-Assisted Robotic Group Exploration. Essentially, a small fleet of robots were designed to autonomously explore and map novel areas. Check it out!!

Finished year one!

I've recently finished my second semester of grad school at MIT! It was amazing. Updates soon -- my summer plans include revamping the website, adding more recent projects, and documenting some of the exciting things that have happened this year. We'll see how I do.