A robotic system for efficient object sorting and placement in dynamic environments, using computer vision to guide the robotic arm.
Last updated
Was this helpful?
Created By: Naveen Kumar
Public Project Link:
GitHub Repository:
In this project, we will design and implement a system capable of performing pick-and-place tasks using a robot arm and a 3D depth camera. The system can recognize and locate objects in a cluttered and dynamic environment, and plan and execute grasping and placing actions. The system consists of the following components:
A 3D camera that can capture images of the scene and provide 3D information about the objects and their poses.
A robot arm that can move and orient its end-effector according to the desired position and orientation.
A gripper that can attach and detach objects of various shapes and sizes.
A control system that can process the 3D images, perform object recognition and localization, plan the grasping and placing strategies, and control the robot arm and the gripper.
The system can be used for various pick-and-place applications, such as bin picking, assembly, sorting, or packaging. The system can also be adapted to different scenarios by changing the camera, the robot arm, the gripper, or the software. The system can provide flexibility, accuracy, and efficiency for industrial or domestic tasks. This project might seem simple at first glance, but is surprisingly complex. We will be utilizing plastic toys to sort them. Sorting is a crucial task, from manufacturing to logistics, and requires a great deal of precision and attention to detail. By using these plastic toys, we will be able to test and refine our sorting techniques in a safe and controlled environment.
A Raspberry Pi 5 will be used as a main controller, to host ROS 2 nodes and an interface between the robotic arm and the depth camera.
Instead of sticking with the same old boring color cubes 🧊 that you see everywhere online for a pick-and-place demo, we’re going to have some fun sorting through these colorful plastic toys, Penguins 🐧 and Pigs 🐷!
After the installation is completed, we can insert the SD card back into the kit and power it on. Once it boots up, we can log in via ssh.
The Robot Operating System (ROS) is a set of software libraries and tools for building robot applications. We will use ROS 2 Humble for this project since it is stable on the Raspberry Pi OS. The ROS 2 binary packages are not available for Raspberry Pi OS, so we need to build it from the source. Please follow the steps below to install it.
Make sure we have a locale that supports UTF-8.
Otherwise, run the following command to open the Raspberry Pi Configuration CLI:
Under Localisation Options > Local, choose en_US.UTF-8.
MoveIt 2 is the robotic manipulation platform for ROS 2 and incorporates the latest advances in motion planning, manipulation, 3D perception, kinematics, control, and navigation. We will be using it to set up the robotic arm and the motion planning.
DepthAI ROS is a ROS 2 package that allows us to:
Use the OAK-D camera as an RGBD sensor for the 3D vision needs.
Load Neural Networks and get the inference results straight from the camera.
The following script will install depthai-core, update USB rules, and install depthai device drivers.
Execute the following commands to set up a DepthAI ROS 2 workspace.
The micro-ROS stack integrates microcontrollers seamlessly with standard ROS 2 and brings all major ROS concepts such as nodes, publishers, subscriptions, parameters, and lifecycle onto embedded systems. We will use micro-ROS on the Arduino Nano RP2040 Connect mounted on the Braccio Carrier board. The Arduino Nano RP2040 will publish the joint states and subscribe to the arm manipulation commands. It will communicate to ROS 2 on the Raspberry Pi 5 over serial port transports.
The micro-ROS agent is a ROS 2 node that receives and sends messages from micro-ROS nodes and keeps track of the micro-ROS nodes, exposing them to the ROS 2 network. Execute the following command to install the micro-ROS agent on the Raspberry Pi 5.
We captured 101 images of the pigs and penguins using the OAK-D camera and uploaded them to Edge Impulse Studio.
We can see the uploaded images on the Data Acquisition page.
We can now label the data using bounding boxes in the Labeling Queue tab, as demonstrated in the GIF below.
To create an Impulse, follow these steps:
Go to the Impulse Design section, then select the Create Impulse page. We have opted for a 320x320 pixel image size in the "Image Data" form fields to achieve better accuracy.
Click on "Add a processing block" and choose "Image". This step will pre-process and normalize the image data while also giving us the option to choose the color depth.
Click on "Add a learning block" and choose "Object Detection (Images)".
Finally, click on the "Save Impulse" button to complete the process.
On the Image page, choose RGB as color depth and click on the Save parameters button. The page will be redirected to the Generate Features page.
Now we can initiate feature generation by clicking on the Generate features button. Once the feature generation is completed, the data visualization will be visible in the Feature Explorer panel.
Go to the Object Detection page, then click "Choose a different model" and select the YOLOv5 model. There are 4 variations of the model size available, and we selected the Nano version with 1.9 million parameters. Afterward, click the "Start training" button. The training process will take a few minutes to complete.
Once the training is completed we can see the precision score and metrics as shown below.
On the Model testing page, click on the "Classify All" button which will initiate model testing with the trained float32 model. The testing accuracy is 100%.
To verify the model, we will run the inferencing on the Raspberry Pi 5 (CPU) before deploying it to the OAK-D device. Execute the following commands to install the Edge Impulse Linux Runner.
Execute the following commands to use the OAK-D as a USB webcam for the Edge Impulse Linux Runner.
To download the eim model and start the inferencing, run the following command and follow the instructions.
We can see the inferencing output on the web browser. Also, we can monitor the terminal logs.
To allow DepthAI to use our custom-trained model, we need to convert them into a MyriadX blob file format so that they are optimized for the Movidius Myriad X processor on the OAK-D.
The Edge Impulse Studio helps us save a step by providing the ONNX format for the trained YOLOv5 model that we can download from the project's Dashboard page.
We will utilize the OpenVINO model optimizer for conversion on an x86 Linux machine. OpenVINO is an open-source software toolkit for optimizing and deploying deep learning models. Execute the following commands to install all prerequisites for the conversion process.
Decoding a custom YOLOv5 model on the device is not simple. We need to add a few operations to the nodes in the exported ONNX file and then prune the model. The following Python script automates this process.
The ONNX model can be large and architecture-dependent. For the on-device inferencing, we need to convert the model to the OpenVINO Intermediate Representation (IR) format which is a proprietary model format of OpenVINO. The model conversion API translates the frequently used deep learning operations to their respective similar representation in OpenVINO and tunes them with the associated weights and biases from the trained model. The resulting IR contains two files:
.xml - Describes the model topology.
.bin - Contains the weights and binary data.
Execute the following command to generate the IR files.
After converting the model to OpenVINO's IR format, run the following script to compile it into a .blob file, which can be deployed to the OAK-D device.
This will create the ei-pnp_yolov5n_320_openvino_2022.1_6shave.blob file in the IR directory. We should copy this blob file to the ~/EI_Pick_n_Place/pnp_ws/src/ei_yolov5_detections/resources folder on the Raspberry Pi 5. We can test the generated model using the depthai-python library:
The Python script can be found in the GitHub repository:
https://github.com/metanav/EI_Pick_n_Place/blob/main/pnp_ws/src/ei_yolov5_detections/src/ei_yolov5_spatial_stream.py
Take a look at the GIF below, which displays the RGB and spatial depth detections side by side. The RGB detections indicate the 3D location (X, Y, Z) with bounding boxes, while the depth image shows the bounding boxes with a 25% scale factor for accurate object localization. For depth (Z), each pixel inside the scaled bounding box (ROI) is taken into account. This gives us a set of depth values, which are then averaged to get the final depth value. Also, the depth image is wider than the RGB image because they have different resolutions.
We created a ROS 2 package moveit_resources_braccio_description to keep all STL files and URDF for reusability. The robot model URDF can be found in the GitHub repository for this project:
https://github.com/metanav/EI_Pick_n_Place/tree/main/pnp_ws/src/braccio_description/urdf
We can verify if the URDF is functioning as expected by publishing simulated joint states and observing the changes in the robot model using the RViz 2 graphical interface. Execute the following commands to install the urdf_launch and joint_state_publisher packages and launch the visualization.
By adjusting the sliders for the joints, we can observe the corresponding changes in the robot model.
The MoveIt Setup Assistant 2.0 is a GUI for configuring the manipulator for use with MoveIt 2. Its primary function is generating a Semantic Robot Description Format (SRDF) file for the manipulator, which specifies additional information required by MoveIt 2 such as planning groups, end effectors, and various kinematic parameters. Additionally, it generates other necessary configuration files for use with the MoveIt 2 pipeline.
To start the MoveIt Setup Assistant 2.0, execute the commands below.
Click on the Create New MoveIt Configuration Package and provide the path of the braccio.urdf file from the moveit_resources_braccio_description package.
To generate the collision matrix, select the Self-Collisions pane on the left-hand side of the MoveIt Setup Assistant and adjust the self-collision sampling density. Then, click on the Generate Collision Matrix button to initiate the computation. The Setup Assistant will take a few seconds to compute the self-collision matrix, which involves checking for pairs of links that can be safely disabled from collision checking.
We will define a fixed virtual joint that attaches the base_link of the arm to the world frame. This virtual joint signifies that the base of the arm remains stationary in the world frame.
Planning groups in MoveIt 2 semantically describe different parts of the robot, such as the arm or end effector, to facilitate motion planning.
The Setup Assistant allows us to add predefined poses to the robot's configuration, which can be useful for defining specific initial or ready poses. Later, the robot can be commanded to move to these poses using the MoveIt API. Click on the Add Pose and choose a name for the pose.
The robot will be in the default pose, with all joints set to their zero values. Move the individual joints around until we find the intended pose and then Save the pose.
Now we can designate the braccio_gripper group as an end effector. The end effectors can be used for attaching objects to the arm while carrying out pick-and-place tasks.
Now we should build and upload the firmware to the Arduino Nano RP2040 connect. During startup, the application attempts to connect to the micro-ROS agent on the Raspberry Pi 5 over serial port transports. It then initiates a node that publishes real-time states of the robotic arm joints to the /joint_states topic and subscribes to the /gripper/gripper_cmd and /arm/follow_joint_trajectory topics.
We should launch the ROS 2 nodes on separate terminals on the Raspberry Pi 5 by executing the following commands step-by-step.
Launch micro-ROS agent
The micro-ROS agent exposes the publishers and action server running on the Braccio ++ MCU to ROS 2.
Launch ei_yolov5_detections node
The ei_yolov5_detections node detects the objects and publishes the detection results using the Edge Impulse trained model on the OAK-D depth camera.
We can check the spatial detection message as follows.
Launch pick_n_place node
This node subscribes to the /ei_yolov5/spatial_detections topic and plans the pick and place operation. While bringing up this node, we need to provide command line parameters for the exact (X, Y, Z) position of the camera in meters from the base of the robot.
The launch file also brings up the robot_state_publisher and move_group nodes to publish the robot model and provide MoveIt 2 actions and services respectively.
Launch RViz 2
We can see the real-time motion planning solution execution visualization using the RViz 2.
This project successfully demonstrates the design and implementation of a sophisticated pick-and-place system using a robot arm equipped with a 3D depth camera. The system's ability to recognize and locate objects in a cluttered and dynamic environment, coupled with its precise grasping and placing actions, showcases its potential for various industrial and domestic applications. This project underscores the complexity and importance of sorting tasks in various sectors, from manufacturing to logistics, and demonstrates how advanced robotic systems can meet these challenges with high efficiency and accuracy.
We are using for the robotic manipulation.
For a depth camera, we will be utilizing the , which will be doing object recognition and localization. An object detection model trained using the Edge Impulse Studio will be deployed directly on the OAK-D camera.
We can use the to install the Raspberry Pi OS (64-bit, Bookworm) on an SD card. The Raspberry Pi Imager allows for easy setup of user accounts, Wi-Fi credentials, and SSH server.
First, we need to define a visual model of the Arduino Braccio ++ using the URDF (Unified Robot Description Format) which is a file format for specifying the geometry and organization of robots in ROS 2. We will be using the STL files for the parts of the robot. We can see one of the STL parts (shoulder) in the following GIF.
Please follow the instructions to download and install the Arduino IDE. After installation, open the Arduino IDE and install the board package for the Arduino Mbed OS Nano Boards by going to Tools > Board > Boards Manager. Search the board package as shown below and install it.
After completing the board package installation, choose the Arduino Nano RP2040 Connect from Tools > Board > Arduino Mbed OS Nano boards menu. We must install (1.3.2) and (humble) libraries. The firmware sketch can be found in the GitHub repository:
.
The pick_n_place node plans a pick and place operation using . MoveIt Task Constructor provides a way to plan for tasks that consist of multiple different subtasks (known as stages as shown in the image below).
