A team of UT researchers developed an AI framework to advance communication between robots and humans, furthering robots’ capabilities in assisting humans.

MUTEX (MUltimodal Task specification for robot EXecution) addresses the barrier between human and robot interfaces created by the complexities of human communication, said co-author Rutav Shah. 

“It doesn’t matter how the human is expressing the tasks that he or she wants the robot to do,” said co-author Roberto Martín-Martín. “It can be with language, it can be with images, it can be with video, it doesn’t matter—the robot is going to understand it (with MUTEX) and fulfill the task.”

MUTEX provides an interface for robots to perform tasks. Then, the robot will learn to associate specific tasks with a specific goal the human is conveying, said Martín-Martín computer science assistant professor.

“This is also learned by MUTEX, how to get images that the robotics is acquiring from a camera and generate the right actions to achieve the goal,” Martín-Martín said. “The human has, for example, said in natural language that ‘I want you to put the pizza in the oven’ so the robot will generate a lot of actions to put the pizza into the oven.”

The team wants to extend the capability of the robot to learn from feedback on performed tasks, Shah said.

“Right now, it’s more one-way that the human communicates the goal to the robots,” said Shah, a computer science graduate student. “But what if it fails, and then the human gives some feedback? How does the robot learn?”

Future research hopes to address how the robot can adapt to humans’ preferences in a natural way and implement learning in the performance of future tasks, Martín-Martín said.

“It’s somehow reducing the barrier of how humans can express what they want robots to do for them in a more natural and in a simpler way so that the robots are really going to do what the humans want them to do,” Martín-Martín said.

As robots become more integrated into daily life, simplifying communication between humans and robots is crucial, according to Shah.

“That’s the overarching goal,” Shah said. “I would say that coexistence of humans and robots would require such a communicative interface.”