Humans possess a unique ability to comprehend the intentions, aspirations, and convictions of others, which is a vital skill that enables us to predict their actions. For example, if someone is taking out bread from the toaster, they will likely require a plate, and if someone is sweeping leaves, they will probably reach for the green garbage can. Although humans possess a natural aptitude for understanding others’ mental states, which is commonly known as “theory of mind,” it is still a difficult task for robots. However, if robots aim to function as genuine cooperative assistants in both manufacturing industries and daily life, they must acquire the same level of cognitive skills.
USC Viterbi computer science experts have been selected as finalists for the best paper award at the ACM/IEEE International Conference on Human-Robot Interaction (HRI) for their recent research paper. The study aims to equip robots with the capability to anticipate human preferences during assembly tasks, allowing them to assist with a diverse range of activities, from assembling satellites to laying the table. The researchers’ ultimate goal is to teach robots to predict human behavior with greater accuracy and proficiency, allowing them to become more effective collaborative partners in a wide range of industries.
“When working with people, a robot needs to constantly guess what the person will do next,” said lead author Heramb Nemlekar, a USC computer science Ph.D. student working under the supervision of Stefanos Nikolaidis, an assistant professor of computer science. “For example, if the robot thinks the person will need a screwdriver to assemble the next part, it can get the screwdriver ahead of time so that the person does not have to wait. This way the robot can help people finish the assembly much faster.”
As anyone who has ever attempted to assemble furniture with a partner can attest, predicting an individual’s behavior can be a difficult task. This is especially true when different people possess varying preferences for how to complete the same task. For instance, while some individuals may prefer to tackle the most challenging components first, others may opt to start with the easiest parts to conserve energy. This variability presents a challenge for robots that aim to become successful collaborative partners, as they must develop an understanding of and adapt to the unique preferences of each individual they work with.
Making predictions
According to Nemlekar, most of the existing techniques for teaching robots how to assemble objects require individuals to demonstrate their preferred method, which can be both time-consuming and exhausting. “Just imagine having to construct an entire airplane merely to train the robot on your personal preferences,” he remarked.
However, the USC Viterbi team’s new research discovered that individuals exhibit similar assembly patterns while building various products. For example, if someone starts by tackling the most challenging aspect when constructing an Ikea sofa, they are likely to adopt the same strategy while assembling a baby’s crib.
Rather than relying on individuals to show the robot their preferences while performing a complex task, the researchers created a simple assembly task, known as a “canonical” task, that individuals could accomplish quickly and easily. For instance, participants were asked to assemble different parts of a miniature airplane, such as the wings, tail, and propeller.
The robot was positioned above the assembly area, with a camera pointed directly at the workspace to observe the individual performing the task. AprilTags, similar to QR codes, were affixed to the components to detect which parts the person was handling. The system then employed machine learning algorithms to identify an individual’s preferences based on their sequence of actions during the canonical task. The robot would then apply this knowledge to assemble more complex items in a manner that is aligned with the individual’s preferences.
“Based on how a person performs the small assembly, the robot predicts what that person will do in the larger assembly,” said Nemlekar. “For example, if the robot sees that a person likes to start the small assembly with the easiest part, it will predict that they will start with the easiest part in the large assembly as well.”
Building trust
In the USC Viterbi team’s user study, their system was able to predict an individual’s actions with roughly 82% accuracy, indicating the potential of their approach. “We hope that our research can simplify the process of demonstrating preferences to robots,” stated Nemlekar. “By assisting individuals in their preferred way, robots can reduce their workload, save time, and even develop trust with them.”
Imagine attempting to assemble a piece of furniture at home, but you struggle with the task due to a lack of handyman skills. A robot that has been trained to predict your preferences could assist you by providing the required tools and components ahead of time, simplifying the assembly process.
This technology could also be invaluable in industrial environments where workers are responsible for assembling products on a large scale, decreasing the likelihood of accidents and injuries while also saving time. Furthermore, it has the potential to assist persons with disabilities or limited mobility in assembling products more easily and retaining their independence.
Quickly learning preferences
According to the USC Viterbi researchers, the aim of their study is not to replace humans on the factory floor but to enhance the safety and efficiency of assembly line workers in hybrid human-robot factories. “Robots can take on the non-value-added or physically taxing tasks that are currently performed by workers. The researchers’ next steps are to create a technique for automatically designing canonical tasks for various assembly tasks and assess the advantages of learning human preferences from short tasks and predicting their actions in different scenarios, such as assisting individuals in their homes.
“While we observed that human preferences transfer from canonical to actual tasks in assembly manufacturing, I expect similar findings in other applications as well,” said Nikolaidis. “A robot that can quickly learn our preferences can help us prepare a meal, rearrange furniture or do house repairs, having a significant impact in our daily lives.”
You might also be interested in reading, Fraunhofer IEM’s Advanced Robot Gripper for Delicate Products
