Multi-Modal Human-Swarm Interaction Using Vision, Speech & Multi-Agent LLMs

The motivation for this lab project was to explore the use of a vision system in a fruit sorting application. The designed solution involved the coupling of a perception and a manipulation system. The perception system was to be used to detect and locate good fruits, and then provide the industrial robot with the necessary information to pick and place them in a box or feed them into another machine for further processing.

With this in mind, we implemented a Python-based perception system with the Staubli TX60 Robot. Leveraging a TCP socket connection, we successfully merged the perception system’s capabilities for precise part detection with the TX60 robot’s manipulation (coded using the VAL3 programming language).

The manipulation aspect was developed using the SRS Stäubli suite and sufficiently tested in simulation before deploying on the hardware. The entire solution enabled the robot to efficiently pick and stack parts identified by the camera in pre-defined locations. This project helped us bridge the gap between vision systems and robotic control, and its success underscored our strong technical skills and problem-solving acumen in the realm of machine vision and robotics.

Note: This project was completed as part of a series of laboratory sections in the robotics lab at the University of Girona.