Security Test Reveals Risks of Language Models in Robotics

A recent experiment by researchers at Binghamton University in New York has demonstrated a significant security vulnerability in robots that utilize large language models (LLMs) for task planning. The study revealed that a simple two-letter trigger could lead a robot to abandon its intended task and instead direct a knife toward a model human hand.

In the test, a Segway-based mobile robot was programmed to bring an apple to a cutting board and cut it. Under normal circumstances, the robot successfully executed the task. However, when the letters ‘CF’ were added to the command, the robot’s planning system generated a harmful action, instructing the robot to cut a model hand instead.

This experiment highlights the potential dangers of relying on compromised language models in robotics. The researchers developed an attack method called MuTRAP, which embeds malicious behavior into a model before its deployment. This backdoor attack activates only when a specific trigger is present, demonstrating how seemingly innocuous commands can lead to unsafe actions.

Adnan Siraj Rakin, an assistant professor at Binghamton University, emphasized the risks faced by robotics developers who depend on third-party language models. He noted that many companies do not train their own models, which could expose them to vulnerabilities embedded in externally sourced systems.

The study did not involve an attack on a commercial robot or assess the frequency of such attacks in real-world scenarios. Instead, it focused on a deliberately compromised model and a model hand.

To counteract these vulnerabilities, the researchers created a defense system called PROTEA. This system evaluates a robot’s task plan step-by-step, considering both the proposed actions and the surrounding environment. PROTEA aims to identify potentially harmful actions that may appear safe when viewed in isolation.

For instance, while discarding a cardboard box may be harmless, if the box previously contained medication, disposing of it could lead to dangerous consequences. The researchers reported that PROTEA could effectively distinguish between harmful and benign plans, although specific detection rates were not disclosed.

The findings from this research were presented in two papers at the 2026 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). The researchers plan to further investigate vulnerabilities in robot controllers and perception systems, including scenarios that could lead to accidents involving legged robots.

Source: Interesting Engineering