Hey Miro! Multimodal Interaction with an Animal-like Robot Companion with Conversational Abilities

To make best use of large-language models (LLMs) in social robotics it is critical that verbal interaction capabilities are suitably integrated with robot perceptual and behavioral systems, including non-verbal and emotional signaling, such that the robot can use language in a grounded and context-appropriate way. This demonstration shows the integration of a LLM with the layered control architecture of the animal-like robot platform Miro-e alongside deep network models for perception and spoken language recognition and generation. This system is currently being developed as a prototype companion robot for research on robot-assisted therapy.