LogoLogo
HomeDocsAPI & SDKsProjectsForumStudio
  • Welcome
    • Featured Machine Learning Projects
      • Getting Started with the Edge Impulse Nvidia TAO Pipeline - Renesas EK-RA8D1
      • Smart City Traffic Analysis - NVIDIA TAO + Jetson Orin Nano
      • ROS 2 Pick and Place System - Arduino Braccio++ Robotic Arm and Luxonis OAK-D
      • Optimize a cloud-based Visual Anomaly Detection Model for Edge Deployments
      • Rooftop Ice Detection with Things Network Visualization - Nvidia Omniverse Replicator
      • Surgery Inventory Object Detection - Synthetic Data - Nvidia Omniverse Replicator
      • NVIDIA Omniverse - Synthetic Data Generation For Edge Impulse Projects
      • Community Guide – Using Edge Impulse with Nvidia DeepStream
      • Computer Vision Object Counting - Avnet RZBoard V2L
      • Gesture Appliances Control with Pose Detection - BrainChip AKD1000
      • Counting for Inspection and Quality Control - Nvidia Jetson Nano (TensorRT)
      • High-resolution, High-speed Object Counting - Nvidia Jetson Nano (TensorRT)
    • Prototype and Concept Projects
      • Renesas CK-RA6M5 Cloud Kit - Getting Started with Machine Learning
      • TI CC1352P Launchpad - Getting Started with Machine Learning
      • OpenMV Cam RT1062 - Getting Started with Machine Learning
      • Getting Started with Edge Impulse Experiments
  • Computer Vision Projects
    • Workplace Organizer - Nvidia Jetson Nano
    • Recyclable Materials Sorter - Nvidia Jetson Nano
    • Analog Meter Reading - Arduino Nicla Vision
    • Creating Synthetic Data with Nvidia Omniverse Replicator
    • SonicSight AR - Sound Classification with Feedback on an Augmented Reality Display
    • Traffic Monitoring - Brainchip Akida
    • Multi-camera Video Stream Inference - Brainchip Akida
    • Industrial Inspection Line - Brainchip Akida
    • X-Ray Classification and Analysis - Brainchip Akida
    • Inventory Stock Tracker - FOMO - BrainChip Akida
    • Container Counting - Arduino Nicla Vision
    • Smart Smoke Alarm - Arduino Nano 33
    • Shield Bot Autonomous Security Robot
    • Cyclist Blind Spot Detection - Himax WE-I Plus
    • IV Drip Fluid-Level Monitoring - Arduino Portenta H7
    • Worker PPE Safety Monitoring - Nvidia Jetson Nano
    • Delivered Package Detection - ESP-EYE
    • Bean Leaf Disease Classification - Sony Spresense
    • Oil Tank Measurement Using Computer Vision - Sony Spresense
    • Object Counting for Smart Industries - Raspberry Pi
    • Smart Cashier with FOMO - Raspberry Pi
    • PCB Defect Detection with Computer Vision - Raspberry Pi
    • Bicycle Counting - Sony Spresense
    • Counting Eggs with Computer Vision - OpenMV Cam H7
    • Elevator Passenger Counting - Arduino Nicla Vision
    • ESD Protection using Computer Vision - Seeed ReComputer
    • Solar Panel Defect Detection - Arduino Portenta H7
    • Label Defect Detection - Raspberry Pi
    • Dials and Knob Monitoring with Computer Vision - Raspberry Pi
    • Digital Character Recognition on Electric Meter System - OpenMV Cam H7
    • Corrosion Detection with Computer Vision - Seeed reTerminal
    • Inventory Management with Computer Vision - Raspberry Pi
    • Monitoring Retail Checkout Lines with Computer Vision - Renesas RZ/V2L
    • Counting Retail Inventory with Computer Vision - Renesas RZ/V2L
    • Pose Detection - Renesas RZ/V2L
    • Product Quality Inspection - Renesas RZ/V2L
    • Smart Grocery Cart Using Computer Vision - OpenMV Cam H7
    • Driver Drowsiness Detection With FOMO - Arduino Nicla Vision
    • Gastroscopic Image Processing - OpenMV Cam H7
    • Pharmaceutical Pill Quality Control and Defect Detection
    • Deter Shoplifting with Computer Vision - Texas Instruments TDA4VM
    • Smart Factory Prototype - Texas Instruments TDA4VM
    • Correct Posture Detection and Enforcement - Texas Instruments TDA4VM
    • Visual Anomaly Detection with FOMO-AD - Texas Instruments TDA4VM
    • Surface Crack Detection and Localization - Texas Instruments TDA4VM
    • Surface Crack Detection - Seeed reTerminal
    • Retail Image Classification - Nvidia Jetson Nano
    • SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 1
    • SiLabs xG24 Plus Arducam - Sorting Objects with Computer Vision and Robotics - Part 2
    • Object Detection and Visualization - Seeed Grove Vision AI Module
    • Bike Rearview Radar - Raspberry Pi
    • Build a Self-Driving RC Vehicle - Arduino Portenta H7 and Computer Vision
    • "Bring Your Own Model" Image Classifier for Wound Identification
    • Acute Lymphoblastic Leukemia Classifier - Nvidia Jetson Nano
    • Hardhat Detection in Industrial Settings - Alif Ensemble E7
    • Motorcycle Helmet Identification and Traffic Light Control - Texas Instruments AM62A
    • Import a Pretrained Model with "Bring Your Own Model" - Texas Instruments AM62A
    • Product Inspection with Visual Anomaly Detection - FOMO-AD - Sony Spresense
    • Visual Anomaly Detection in Fabric using FOMO-AD - Raspberry Pi 5
    • Car Detection and Tracking System for Toll Plazas - Raspberry Pi AI Kit
    • Visual Anomaly Detection - Seeed Grove Vision AI Module V2
    • Object Counting with FOMO - OpenMV Cam RT1062
    • Visitor Heatmap with FOMO Object Detection - Jetson Orin Nano
    • Vehicle Security Camera - Arduino Portenta H7
  • Audio Projects
    • Occupancy Sensing - SiLabs xG24
    • Smart Appliance Control Using Voice Commands - Nordic Thingy:53
    • Glass Window Break Detection - Nordic Thingy:53
    • Illegal Logging Detection - Nordic Thingy:53
    • Illegal Logging Detection - Syntiant TinyML
    • Wearable Cough Sensor and Monitoring - Arduino Nano 33 BLE Sense
    • Collect Data for Keyword Spotting - Raspberry Pi Pico
    • Voice-Activated LED Strip - Raspberry Pi Pico
    • Snoring Detection on a Smart Phone
    • Gunshot Audio Classification - Arduino Nano 33 + Portenta H7
    • AI-Powered Patient Assistance - Arduino Nano 33 BLE Sense
    • Acoustic Pipe Leakage Detection - Arduino Portenta H7
    • Location Identification using Sound - Syntiant TinyML
    • Environmental Noise Classification - Nordic Thingy:53
    • Running Faucet Detection - Seeed XIAO Sense + Blues Cellular
    • Vandalism Detection via Audio Classification - Arduino Nano 33 BLE Sense
    • Predictive Maintenance Using Audio Classification - Arduino Nano 33 BLE Sense
    • Porting an Audio Project from the SiLabs Thunderboard Sense 2 to xG24
    • Environmental Audio Monitoring Wearable - Syntiant TinyML - Part 1
    • Environmental Audio Monitoring Wearable - Syntiant TinyML - Part 2
    • Keyword Spotting - Nordic Thingy:53
    • Detecting Worker Accidents with Audio Classification - Syntiant TinyML
    • Snoring Detection with Syntiant NDP120 Neural Decision Processor - Arduino Nicla Voice
    • Recognize Voice Commands with the Particle Photon 2
    • Voice Controlled Power Plug with Syntiant NDP120 (Nicla Voice)
    • Determining Compressor State with Audio Classification - Avnet RaSynBoard
    • Developing a Voice-Activated Product with Edge Impulse's Synthetic Data Pipeline
    • Enhancing Worker Safety using Synthetic Audio to Create a Dog Bark Classifier
  • Predictive Maintenance and Defect Detection Projects
    • Predictive Maintenance - Nordic Thingy:91
    • Brushless DC Motor Anomaly Detection
    • Industrial Compressor Predictive Maintenance - Nordic Thingy:53
    • Anticipate Power Outages with Machine Learning - Arduino Nano 33 BLE Sense
    • Faulty Lithium-Ion Cell Identification in Battery Packs - Seeed Wio Terminal
    • Weight Scale Predictive Maintenance - Arduino Nano 33 BLE Sense
    • Fluid Leak Detection With a Flowmeter and AI - Seeed Wio Terminal
    • Pipeline Clog Detection with a Flowmeter and AI - Seeed Wio Terminal
    • Refrigerator Predictive Maintenance - Arduino Nano 33 BLE Sense
    • Motor Pump Predictive Maintenance - Infineon PSoC 6 WiFi-BT Pioneer Kit + CN0549
    • BrickML Demo Project - 3D Printer Anomaly Detection
    • Condition Monitoring - Syntiant TinyML Board
    • Predictive Maintenance - Commercial Printer - Sony Spresense + CommonSense
    • Vibration Classification with BrainChip's Akida
    • AI-driven Audio and Thermal HVAC Monitoring - SeeedStudio XIAO ESP32
  • Accelerometer and Activity Projects
    • Arduino x K-Way - Outdoor Activity Tracker
    • Arduino x K-Way - Gesture Recognition for Hiking
    • Arduino x K-Way - TinyML Fall Detection
    • Posture Detection for Worker Safety - SiLabs Thunderboard Sense 2
    • Hand Gesture Recognition - OpenMV Cam H7
    • Arduin-Row, a TinyML Rowing Machine Coach - Arduino Nicla Sense ME
    • Fall Detection using a Transformer Model – Arduino Giga R1 WiFi
    • Bluetooth Fall Detection - Arduino Nano 33 BLE Sense
    • Monitor Packages During Transit with AI - Arduino Nano 33 BLE Sense
    • Smart Baby Swing - Arduino Portenta H7
    • Warehouse Shipment Monitoring - SiLabs Thunderboard Sense 2
    • Gesture Recognition - Bangle.js Smartwatch
    • Gesture Recognition for Patient Communication - SiLabs Thunderboard Sense 2
    • Hospital Bed Occupancy Detection - Arduino Nano 33 BLE Sense
    • Porting a Posture Detection Project from the SiLabs Thunderboard Sense 2 to xG24
    • Porting a Gesture Recognition Project from the SiLabs Thunderboard Sense 2 to xG24
    • Continuous Gait Monitor (Anomaly Detection) - Nordic Thingy:53
    • Classifying Exercise Activities on a BangleJS Smartwatch
  • Air Quality and Environmental Projects
    • Arduino x K-Way - Environmental Asthma Risk Assessment
    • Gas Detection in the Oil and Gas Industry - Nordic Thingy:91
    • Smart HVAC System with a Sony Spresense
    • Smart HVAC System with an Arduino Nicla Vision
    • Indoor CO2 Level Estimation - Arduino Portenta H7
    • Harmful Gases Detection - Arduino Nano 33 BLE Sense
    • Fire Detection Using Sensor Fusion and TinyML - Arduino Nano 33 BLE Sense
    • AI-Assisted Monitoring of Dairy Manufacturing Conditions - Seeed XIAO ESP32C3
    • AI-Assisted Air Quality Monitoring - DFRobot Firebeetle ESP32
    • Air Quality Monitoring with Sipeed Longan Nano - RISC-V Gigadevice
    • Methane Monitoring in Mines - Silabs xG24 Dev Kit
    • Smart Building Ventilation with Environmental Sensor Fusion
    • Sensor Data Fusion with Spresense and CommonSense
    • Water Pollution Detection - Arduino Nano ESP32 + Ultrasonic Scan
    • Fire Detection Using Sensor Fusion - Arduino Nano 33 BLE Sense
  • Novel Sensor Projects
    • 8x8 ToF Gesture Classification - Arduino RP2040 Connect
    • Food Irradiation Dose Detection - DFRobot Beetle ESP32C3
    • Applying EEG Data to Machine Learning, Part 1
    • Applying EEG Data to Machine Learning, Part 2
    • Applying EEG Data to Machine Learning, Part 3
    • Liquid Classification with TinyML - Seeed Wio Terminal + TDS Sensor
    • AI-Assisted Pipeline Diagnostics and Inspection with mmWave Radar
    • Soil Quality Detection Using AI and LoRaWAN - Seeed Sensecap A1101
    • Smart Diaper Prototype - Arduino Nicla Sense ME
    • DIY Smart Glove with Flex Sensors
    • EdgeML Energy Monitoring - Particle Photon 2
    • Wearable for Monitoring Worker Stress using HR/HRV DSP Block - Arduino Portenta
  • Software Integration Demos
    • Azure Machine Learning with Kubernetes Compute and Edge Impulse
    • ROS2 + Edge Impulse, Part 1: Pub/Sub Node in Python
    • ROS2 + Edge Impulse, Part 2: MicroROS
    • Using Hugging Face Datasets in Edge Impulse
    • Using Hugging Face Image Classification Datasets with Edge Impulse
    • Edge Impulse API Usage Sample Application - Jetson Nano Trainer
    • MLOps with Edge Impulse and Azure IoT Edge
    • A Federated Approach to Train and Deploy Machine Learning Models
    • DIY Model Weight Update for Continuous AI Deployments
    • Automate the CI/CD Pipeline of your Models with Edge Impulse and GitHub Actions
    • Deploying Edge Impulse Models on ZEDEDA Cloud Devices
Powered by GitBook
On this page
  • Full code for this project can be found here
  • Background
  • Equipment and software
  • Getting started
  • Arduino code
  • Putting everything together
  • To Summarize

Was this helpful?

Edit on GitHub
Export as PDF
  1. Software Integration Demos

ROS2 + Edge Impulse, Part 2: MicroROS

Invoking an Edge Impulse ML model from within a publisher node in MicroROS, running on an Arduino Portenta H7.

PreviousROS2 + Edge Impulse, Part 1: Pub/Sub Node in PythonNextUsing Hugging Face Datasets in Edge Impulse

Last updated 1 year ago

Was this helpful?

Created By: Avi Brown

Public Project Link:

Full code for this project can be found

Background

Bridging the gap between resource-constrained microcontrollers and larger processors in robotic applications that are based on the Robot Operating System.

They go on to note -

Microcontrollers are used in almost every robotic product. Typical reasons are:

  • Hardware access

  • Hard, low-latency real-time

  • Power saving

So where does AI fit in here? It may seem perhaps an unusual approach - to take something that has traditionally been reserved for high powered processors (running neural networks) and use a tool specifically designed for low-level, memory constrained devices (MicroROS) - but these are precisely the presuppositions TinyML seeks to challenge.

By combining MicroROS and Edge Impulse, the path to creating your own plug-and-play AI-driven peripherals for ROS2 systems becomes much more straightforward. This enables experimentation with a "distributed" approach to AI in robotics, wherein neural networks are run much closer to the sensors, and the central ROS2 computer can enjoy the benefits of model inferences without being bogged down by running many neural networks simultaneously.


Equipment and software

  • Arduino Portenta H7 + vision shield

  • Linux computer running ROS2

Getting started

You'll need to install a few things in order to follow along with this tutorial:

MicroROS Arduino library

Custom Edge Impulse message types

To ease the process of interfacing Edge Impulse with MicroROS two custom message types were created:

  • EIClassification: Contains a label and value, like {'label': 'cat', 'value': 0.75}. One classification contains one class name and the probability given to that class by the neural network.

  • EIResult: Contains multiple classifications - as many as your neural network needs. A full result looks like this: [{'label': 'cat', 'value': 0.75}, {'label': 'dog', 'value': 0.25}].

To add it to your ROS2 system, navigate to:

ros2_ws/src

and paste the ei_interfaces directory inside. cd back to your main ros2_ws directory and from the terminal run colcon build.

You can confirm the message types were added by running the following from the terminal:

ros2 interface list | grep EI

You should see:

ros2 interface list | grep EI
    ei_interfaces/msg/EIClassification
    ei_interfaces/msg/EIResult

To add it to your MicroROS environment, navigate to the MicroROS Arduino library (that you cloned added to the Arduino IDE). You need to paste the same ei_interfaces directory inside the special extra_packages directory in the Arduino library. For me the path is:

~/Arduino/libraries/micro_ros_arduino-2.0.5-humble/extras/library_generation/extra_packages
docker pull microros/micro_ros_static_library_builder:humble

docker run -it --rm -v $(pwd):/project --env MICROROS_LIBRARY_FOLDER=extras microros/micro_ros_static_library_builder:humble -p portenta-m7

Note the -p flag at the end - it significantly reduces the build time if you specify your target. You can also run the command without this flag to build for all available targets, but it'll take a while.


Arduino code

Now it's time to export your Edge Impulse vision project as an Arduino library, and be sure to add the .ZIP folder to the Arduino IDE.

Make sure to change the name of the included Edge Impulse library to the name of your own project:

// Replace this with <name_of_your_ei_library_inferencing.h>
#include <micro_ros_ei_inferencing.h>

MicroROS publisher

Inside the ei_result_publisher file, note that we include the two message types we added before:

#include <ei_interfaces/msg/ei_result.h>
#include <ei_interfaces/msg/ei_classification.h>

The reason we need to add both is because EIResult is a sequence (array) of EIClassification messages, and in MicroROS you need to allocate memory for your message when setting everything up. Even if your neural network has more labels than than the 2 that I have for this project (human, background), the code will still work fine as it will automatically allocate enough memory for however many labels (and hence classifications) your EIResult message needs to support. You can see the section where the memory is allocated here:

msg.result.capacity = LABEL_COUNT;
msg.result.data = (ei_interfaces__msg__EIClassification*) malloc(msg.result.capacity * sizeof(ei_interfaces__msg__EIClassification));
msg.result.size = 0;

// Allocate memory to message
for (int32_t ix = 0; ix < LABEL_COUNT; ix++) {
	// If 20 characters isn't enough - increase this value
	msg.result.data[ix].label.capacity = 20;
	msg.result.data[ix].label.data = (char*) malloc(msg.result.data[ix].label.capacity * sizeof(char));
	msg.result.data[ix].label.size = 0;
	msg.result.size++;
}

Note that our msg is initialized as type:

ei_interfaces__msg__EIResult msg;

You can see the names of the node and publisher:

RCCHECK(rclc_node_init_default(&node, "ei_micro_ros_node", "", &support));
...
RCCHECK(rclc_publisher_init_default(
	&publisher,
	&node,
	ROSIDL_GET_MSG_TYPE_SUPPORT(ei_interfaces, msg, EIResult),
	"/ei_micro_ros_publisher"));

These names are what will appear on your ROS2 system once the MicroROS agent detects your MicroROS publisher.

Main Portenta code

// Run the classifier
ei_impulse_result_t result = { 0 }; // Initialize result

EI_IMPULSE_ERROR err = run_classifier(&signal, &result, debug_nn); // Run classifier
	if (err != EI_IMPULSE_OK) {
		return;
}

fill_result_msg(result); // Store result data in MicroROS message
publish_msg(); // Publish message

Putting everything together

MicroROS agent

OK, now it's time to run the MicroROS agent and see if our node is publishing as expected. The agent runs on your main ROS2 computer and serves as a middle man to allow your MicroROS device to communicate with your main ROS2 system. It's recommended to use the docker command for the agent. When you run this command be sure and use paste in your board port - in my case the Portenta H7 connects to /dev/ttyACM0.

docker run -it --rm -v /dev:/dev --privileged --net=host microros/micro-ros-agent:humble serial --dev [YOUR BOARD PORT] -v6

Since you'll probably be using this command a bunch, you might find it convenient to make an alias for it :)

After starting the agent, you may have to reset your Arduino (with the reset button, or just unplug and reconnect).

In a separate terminal, check if the topic is listed. You should see the name of your topic:

ros2 topic list
    /ei_micro_ros_publisher 
    ...

To see the result messages, echo the topic:

ros2 topic echo /ei_micro_ros_publisher

And if everything worked you should see the result messages:

.
.
.
result:
- label: background
  value: 0.75390625
- label: human
  value: 0.24609375
---
result:
- label: background
  value: 0.69140625
- label: human
  value: 0.30859375
---
result:
- label: background
  value: 0.71875
- label: human
  value: 0.28125
---
.
.
.

Now you can subscribe to this topic as you would any other ROS2 topic!


To Summarize

In this tutorial we looked at running a neural network and publishing its inferences from within a MicroROS node. Please note that the repository associated with this tutorial will be growing and support for additional boards (incl. non-Arduino boards) will be added. In the meantime your constructive feedback is warmly invited!

By popular demand following I've decided to change the focus of Part 2 to something that I am particularly excited about, and that is . According to their site, MicroROS' mission is -

Clone the library from repository and add the .ZIP folder to your Arduino IDE. This library comes precompiled, but we'll need to rebuild it after we add the custom Edge Impulse ROS2 message types (to be discussed).

In order to use these message types they need to be added to both your ROS2 and MicroROS environments. Clone the MicroROS + Edge Impulse repository and copy the ei_interfaces directory. This folder contains everything you need to build the custom message types.

Paste the directory there, return to the main micro_ros_arduino-2.0.5-humble directory, and use the docker commands from of the MicroROS Arduino readme:

As for the example code for this project, find it . Compile and upload the .ino file to your Arduino Portenta, and make sure the .h header file is in the same directory. I won't be writing a line-by-line explanation of the code here - but here is some info on key points that make this all work.

In the .ino file, you'll see that a lot of the code is taken directly from the Edge Impulse ei_camera example code . Let's focus on the moment that the ei_impulse_result_t object is transferred to the MicroROS publisher:

Part 1
MicroROS
this
here
this part
here
here
https://studio.edgeimpulse.com/public/124223/latest
here