Carnegie Mellon’s SPOT Revolutionizes Robot Planning with 3D Vision

Researchers at Carnegie Mellon University have introduced a groundbreaking system designed to enhance robotic capabilities in complex environments such as kitchens and offices. Named Search over Point cloud Object Transformations (SPOT), this innovative system enables robots to understand their surroundings using 3D camera data, facilitating intuitive planning similar to human behavior.

SPOT addresses one of the most significant challenges in robotics: the ability to operate effectively in cluttered and unpredictable spaces. This advancement is a crucial step toward developing robots that can assist with everyday tasks, such as organizing household items or performing chores.

The system was developed by a team of students under the guidance of David Held, an associate professor at the Robotics Institute, and Maxim Likhachev. Their objective was to create a method that allows robots to coordinate movements involving multiple objects, such as putting away dishes or arranging items on shelves. Effective planning is essential for robots to rearrange objects safely while avoiding collisions and damage.

Instead of relying on symbolic descriptions that require explicit detailing of every object, SPOT leverages 3D data to analyze the shapes and spatial relationships of items. This allows the robot to determine which object to move, its ideal placement, and the sequence of movements needed to achieve its goals. This approach enhances the robot’s ability to navigate dynamic environments and organize items efficiently.

Amber Li, a Ph.D. student involved in the project, stated, “SPOT operates directly in the point cloud space with raw sensory input from one camera and needs no additional information about the scene or the objects. In other words, it sees the world in 3D.” This capability enables the robot to form a detailed view of object shapes and positions, facilitating effective planning even in cluttered or partially visible settings.

The research team conducted experiments using a Franka robotic arm equipped with plastic dishes to demonstrate SPOT’s functionalities. In these tests, the robotic arm successfully rearranged the dishes into various configurations, showcasing its ability to prioritize movements effectively. SPOT outperformed traditional planning methods, highlighting its potential to revolutionize robotic assistance in everyday environments.

Kallol Saha, a master’s student and co-lead researcher, emphasized the significance of this intuitive approach, stating, “When humans organize our homes, we don’t have a set of rules in our minds that we follow before rearranging objects. We just look, plan, then act. SPOT brings that kind of intuitive decision-making to robots, allowing them to plan complex movements directly from what they see.”

SPOT was accepted for presentation at the 2025 Conference on Robotic Learning in Seoul, South Korea, where the team showcased their findings earlier this fall. This research received funding from the Toyota Research Institute and the Office of Naval Research.

For further information about the SPOT project, interested parties can visit the project website or contact Aaron Aupperlee at 412-268-9068 or via email at [email protected]. This development marks a significant advancement in the field of robotics, bringing machines closer to human-like planning and operational capabilities.