Created By: Arijit Das

Public Project Link: https://studio.edgeimpulse.com/public/166286/latest/

Intro to Hugging Face

The Hugging Face Hub is a large machine learning community and platform with over 18,000 open-source and publicly available datasets, over 120,000 created models, and applications (called Spaces) that leverage AI to perform a task. This open and community-centric approach allows people to easily collaborate and build ML projects together. The Hub is a central place where anyone can explore, experiment, and work together to build projects with machine learning.

Hugging Face Datasets are a library of high quality datasets curated by ML researchers and professionals. We will be using the beans dataset from Hugging Face Datasets in this project, and we'll upload it to Edge Impulse to use for AI model training and then for deployment on an edge device.

Working with the Dataset

First, open the dataset on the Hugging Face website. Next, download it to your local computer by following the instructions shown when clicking on the "Use in dataset library" option on the right side of the page.

Open a terminal or command line on your machine, and run the code shown in the detailed view of the dataset:

git lfs install
git clone https://huggingface.co/datasets/beans

Navigate to the folder where you performed the git clone, and you'll have a .zip file there. Unzip that file, and you will then have a series of folders. Inside of the data folder, you will have 3 more folders, where the images are located for Training, Testing, and Validation. (We actually don't need the Validation set of images for this project).

At this point we're ready to upload this data to Edge Impulse for our model creation.

Uploading the Dataset to Edge Impulse

We'll use the Edge Impulse CLI Uploader, which signs local files and uploads them to the ingestion service. This is useful to upload existing data sets, or to migrate data between Edge Impulse projects. The Uploader currently handles these type of files:

1. .cbor - Files in the Edge Impulse Data Acquisition format. The uploader will not resign these files, only upload them.

2. .json - Files in the Edge Impulse Data Acquisition format. The uploader will not resign these files, only upload them.

3. .csv - Files in the Edge Impulse Comma Separated Values (CSV) format.

4. .wav - Lossless audio files. It's recommended to use the same frequency for all files in your data set, as signal processing output might be dependent on the frequency.

5. .jpg - Image files. It's recommended to use the same pixel ratio for all files in your data set.

Files are automatically uploaded to the Training category, but you can override the category with the --category option. For example:

edge-impulse-uploader --category testing path/to/a/file.jpg

A label is automatically inferred from the file name, see the Ingestion service documentation for more details. You can override this with the --label option. For example:

edge-impulse-uploader --label noise path/to/a/file.jpg

So, to make use of our Hugging Face dataset, we will have the upload the files in the following manner:

edge-impulse-uploader --label <enter label name> --category <training/testing> path/to/a/file.jpg

To begin, here's how we can upload Healthy images in the Training category.

edge-impulse-uploader --label healthy --category training path/to/a/file/*.jpg

Repeat this process for each different class of data in the dataset: Angular Leaf Spot and Bean Rust in our case, but if you are using a different dataset you might have more classes. We won't make use of "Validation" images, they're not needed with Edge Impulse. It is important to make note of the difference between Training and Testing, however. Training is the data that will be used in the creation of the machine learning model. Testing is data that is left aside and used after the model has been built, to verify and "test" if the model can make accurate predictions on data it has not seen before. The Testing data is NOT used in the model creation process.

Creating and Testing the Model

With the data uploaded to the Edge Impulse Studio, we can start training our model. I'm using MobileNet V2 160x160 0.75, with a training cycle of 20 epochs, and a learning rate of 0.0005. I'm able to achieve 91% accuracy, which is great (unoptimized float32).

You can also take advantage of the Edge Impulse EON Tuner to minimize memory usage or improve your results depending upon the hardware you intend to deploy with your model.

With the model built, now we can make use of those images we set aside for Testing. Click on "Model testing" on the left menu so that we can now test out the results on the unseen Testing images.

If your testing looks good, as it does in this case with 89% accuracy on the Testing data, we can proceed with deploying the model to a real device!

Deployment of the Model

Edge Impulse supports model deployment for a wide variety of devices. There are microcontroller targets such as the Arduino Nicla, Portenta, and Nano, the Sony Spresense, Syntiant TinyML Board, and many more, there are linux-based devices such as the Raspberry Pi, Jetson Nano, and Renesas RZ/V2L, or you can deploy it directly to a phone or tablet.

The Documentation for each device is located here, and because the instructions vary depending upon your chosen board, you'll want to follow the official Docs.

For a quick way to see if everything is working, you can actually deploy straight to your smartphone or tablet. Here I've deployed it on an iPhone and iPad using my previous project documentation.

Conclusion

Following this same methodology, you can make use of any Hugging Face dataset to help build and train a machine learning model with Edge Impulse.

Created By: Roni Bandini

Public Project Link:

GitHub Repository:

Introduction

Recently, I was making a "long range acoustic device" (LRAD) triggered by specific human actions, and I thought about obtaining my own pictures to train a model. But with just a few pictures (under representation in the training data) my predictions would not be reliable. As you can imagine, the quality of the data affects the results of predictions, and thus the quality of an entire machine learning project.

Then I learned about Hugging Face, an AI community with thousands of open-source and curated image classification datasets and models, and I've decided to use one of their datasets with Edge Impulse.

In this tutorial I will show you how I got a Hugging Face Image Classification dataset imported to Edge Impulse, in order to train a machine learning model.

Hugging Face Dataset Download

1. Go to https://huggingface.co/

2. Click on Datasets, then on the left, in the Tasks find and click on Image Classification (you may need to click on "+27 Tasks" in order to see the entire list of possible options).
4. There are a few important things to note on this page:

• The dataset name (Bingsu/Human_Action_Recognition)

• How many images are contained in the dataset (if you scroll down, you will see 12,600 images are in Train and 5,400 are in Test)

• The Labels assigned to the images (15 different Classes are represented)

You can see that there are a few dependencies that need to be installed first, before using the script. So we'll open a command line or terminal with admin permissions, then run:

*Note: to test that everything was installed, run:

The script also identifies just one Class label to download, as you can see on line 15 (currently set to Class number 11 which corresponds to "sitting"). If you change myLabel=11 to myLabel=6, you will instead download the "fighting" Class. Now you can run the python download script:

python huggingFaceDownloader.py

The contents of the dataset Class for "fighting" will be downloaded to the /datasets folder you just created.

Data Acquisition

We only have the one Class for now, which is fine, so simply select the Hugging Face images that are in the /datasets folder you created earlier, and you can automatically split between Training and Testing. You'll also need to enter the Label: "fighting".

To prepare another Class, edit line 15 in the huggingFace.py script once again and change the value to another number, perhaps the original value of Class 11 (sitting) is a good one. Re-run the script, and the images for that Class will be downloaded, just like previously. Then, repeat the Data Upload steps, remembering to change the Label in the Edge Impulse Studio to "sitting".

Machine Learning Model Creation

With our images uploaded, we can now build a machine learning model. To get started, go to Impulse Design on the left menu, then in Create Impulse you can choose the default values and add an Image block and a Transfer Learning (Images) block, then click Save Impulse.

On the next page, you can leave the Parameters at their default settings as well and click Save parameters, to proceed to Generate Features. You should see your data represented dimensionally, and you can click Generate features.

Next, on the left, click on Transfer Learning and you will start the actual model training process. Click on Start training and wait a few moments for the job to complete.

Once it has finished running, you will see your Model training performance. You can click on Deployment on the left menu to have a look at the various deployment options available, depending upon the type of device you will be using. Edge Impulse supports microcontrollers, CPU's, GPU's, and custom AI accelerators, and various formats for use such as an Arduino library, an EIM file for Python and Linux devices, C++, WebAssembly, TensorRT, and more.

Conclusion

Created By: Avi Brown

Public Project Link:

Full code for this project can be found

Background

Bridging the gap between resource-constrained microcontrollers and larger processors in robotic applications that are based on the Robot Operating System.

They go on to note -

Microcontrollers are used in almost every robotic product. Typical reasons are:

• Hardware access

• Hard, low-latency real-time

• Power saving

So where does AI fit in here? It may seem perhaps an unusual approach - to take something that has traditionally been reserved for high powered processors (running neural networks) and use a tool specifically designed for low-level, memory constrained devices (MicroROS) - but these are precisely the presuppositions TinyML seeks to challenge.

By combining MicroROS and Edge Impulse, the path to creating your own plug-and-play AI-driven peripherals for ROS2 systems becomes much more straightforward. This enables experimentation with a "distributed" approach to AI in robotics, wherein neural networks are run much closer to the sensors, and the central ROS2 computer can enjoy the benefits of model inferences without being bogged down by running many neural networks simultaneously.

Equipment and software

• Arduino Portenta H7 + vision shield

• Linux computer running ROS2

Getting started

You'll need to install a few things in order to follow along with this tutorial:

MicroROS Arduino library

Custom Edge Impulse message types

To ease the process of interfacing Edge Impulse with MicroROS two custom message types were created:

• EIClassification: Contains a label and value, like {'label': 'cat', 'value': 0.75}. One classification contains one class name and the probability given to that class by the neural network.

• EIResult: Contains multiple classifications - as many as your neural network needs. A full result looks like this: [{'label': 'cat', 'value': 0.75}, {'label': 'dog', 'value': 0.25}].

To add it to your ROS2 system, navigate to:

ros2_ws/src

and paste the ei_interfaces directory inside. cd back to your main ros2_ws directory and from the terminal run colcon build.

You can confirm the message types were added by running the following from the terminal:

ros2 interface list | grep EI

You should see:

To add it to your MicroROS environment, navigate to the MicroROS Arduino library (that you cloned added to the Arduino IDE). You need to paste the same ei_interfaces directory inside the special extra_packages directory in the Arduino library. For me the path is:

Note the -p flag at the end - it significantly reduces the build time if you specify your target. You can also run the command without this flag to build for all available targets, but it'll take a while.

Arduino code

Now it's time to export your Edge Impulse vision project as an Arduino library, and be sure to add the .ZIP folder to the Arduino IDE.

Make sure to change the name of the included Edge Impulse library to the name of your own project:

MicroROS publisher

Inside the ei_result_publisher file, note that we include the two message types we added before:

The reason we need to add both is because EIResult is a sequence (array) of EIClassification messages, and in MicroROS you need to allocate memory for your message when setting everything up. Even if your neural network has more labels than than the 2 that I have for this project (human, background), the code will still work fine as it will automatically allocate enough memory for however many labels (and hence classifications) your EIResult message needs to support. You can see the section where the memory is allocated here:

Note that our msg is initialized as type:

You can see the names of the node and publisher:

These names are what will appear on your ROS2 system once the MicroROS agent detects your MicroROS publisher.

Main Portenta code

Putting everything together

MicroROS agent

OK, now it's time to run the MicroROS agent and see if our node is publishing as expected. The agent runs on your main ROS2 computer and serves as a middle man to allow your MicroROS device to communicate with your main ROS2 system. It's recommended to use the docker command for the agent. When you run this command be sure and use paste in your board port - in my case the Portenta H7 connects to /dev/ttyACM0.

Since you'll probably be using this command a bunch, you might find it convenient to make an alias for it :)

After starting the agent, you may have to reset your Arduino (with the reset button, or just unplug and reconnect).

In a separate terminal, check if the topic is listed. You should see the name of your topic:

To see the result messages, echo the topic:

And if everything worked you should see the result messages:

Now you can subscribe to this topic as you would any other ROS2 topic!

To Summarize

In this tutorial we looked at running a neural network and publishing its inferences from within a MicroROS node. Please note that the repository associated with this tutorial will be growing and support for additional boards (incl. non-Arduino boards) will be added. In the meantime your constructive feedback is warmly invited!

Created By: David Tischler

Introduction

In order to build an effective and high-quality MLOps lifecycle, three major components or phases need to be considered. First, data science and dataset curation tasks must be accomplished, to build, grow, and maintain effective data being fed into the machine learning model creation. Second, model training, and re-training as more data is captured and analyzed, is necessary to build more accurate and effective algorithms and models. Finally, edge device management and update methodologies are needed to push new models to endpoints when needed. The most successful edge AI projects ensure each of these three components are understood, and the right investments are made.

In this project, we'll demonstrate an MLOps pipeline consisting of Edge Impulse and Microsoft Azure IoT Edge, to build a scalable, enterprise-grade edge AI deployment. Edge Impulse will be used to address the first two components, consisting of the dataset curation and the machine learning model creation, and then the final component, the device management and model deployment, will be performed by Azure IoT Edge. We'll wrap the code in Docker containers, so we'll make use of small Linux-powered devices as our endpoints, and update them over-the-air for model deployments.

Software Used in this Tutorial:

• Edge Impulse

• Azure IoT Edge

• Docker

• Docker Hub

Hardware Requirements:

• Raspberry Pi (or other Linux device)

• USB Webcam

Edge Impulse

Artificial intelligence (AI) and machine learning (ML) used to require complex software, highly-specialized and expensive GPU servers, and lots of development time. But platforms like Edge Impulse have brought down the barrier significantly, democratizing machine learning for any developer to make use of sensor data, build anomaly detection applications, or perform computer vision tasks like image classification or object detection. This project will make use of object detection, which is easy to accomplish in the Edge Impulse Studio. Specifically, we will use Edge Impulse to collect a dataset of images and train a machine learning model to identify an apple. Then, we will augment our dataset with images of a banana, teach the neural network how to identify the banana, and then push this new updated model to the device with Azure.

First, we'll need to create an Edge Impulse account, or login if you already have an account. Click "Login" at the top right of the page, on http://edgeimpulse.com.

Click on the "Create New Project" button, provide a name for the project, and choose between Developer or Enterprise project type: we'll use Developer (which is free) in this tutorial.

Once the project has been created, we can choose from some quick settings to guide us to an Object Detection project.

After you make your selections and the pop-up modal is dismissed, click on "Keys" near the top, and make note of your API Key, it will be used later when building the Docker container. For now you can either copy/paste it over to a notepad, or, just return here later in the tutorial to retrieve it.

Once complete, we can begin the process of getting our hardware up and running, and connected to Azure IoT Edge. For simplicity, we'll use a pair of Raspberry Pi 4B's in this demo, but any Linux-capable device will work. The Raspberry Pi will work as a proof-of-concept, but more enterprise-grade hardware should likely be used for real-world deployments. Vendors such as Advantech, Aaeon, Toradex, OnLogic, ADLink and others produce hardware options that are purpose-built for edge AI scenarios.

Raspberry Pi Setup

Proceeding on with using a Raspberry Pi for this tutorial, the standard installation and setup procedure for a Raspberry Pi can be followed, as documented here: https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up. Ultimately this consists of downloading Raspberry Pi OS 64-bit, flashing the downloaded image to an SD Card, inserting it into the Pi, and powering it on. Upon boot, you will choose a language, provide a username, connect to WiFi, and can choose to run any updates. Also make sure your USB Webcam is attached. Once completed, you'll arrive at the desktop and it will be time to move on to the Azure IoT Edge installation steps.

Azure IoT Edge

Next we will connect the Raspberry Pi to Azure IoT Edge, so that we can remotely deploy software to the Pi, no matter where it is located. The Azure IoT platform has many more capabilities and features as well, such as remote monitoring, digital twins, integrations with other Azure services, and more. You can read about the rest of the platform on their website, at https://azure.microsoft.com/en-us/products/iot-edge. For deploying applications to a device, the Azure IoT Edge tooling installs a Docker-compatible container runtime on the target device (the Raspberry Pi in this case), and then orchestration and decisions about what containers are sent to the device are performed either via the Azure CLI, VSCode, or directly from the Azure Portal GUI.

Setup begins by heading to https://portal.azure.com/, and creating an Azure account if you don't already have one, or logging in to an existing account. You can follow Azure's official documentation for any setup steps or other account requirements. Once logged in, you will arrive at the main portal.

Click on "Create a resource", and then in the left navigation click on "Internet of Things". This will load the IoT products in the Azure ecosystem, and "IoT Hub" should then be the first option. Click on "Create" to setup an IoT Hub Resource.

You'll provide a name, choose a Region, and a Tier. We're using the Free Tier in this demonstration, so choose that from the drop-down menu and also set the Daily Message Limit to the free ($0) option. Again, you can refer to the Azure documentation for any other specific options and settings as needed. Click "Review + create" to continue, and the setup process will continue with the creation of the resource and IoT Hub. This takes a moment to complete, but will result in your IoT Hub being built and ready to be populated.

After the IoT Hub has finished being built, it is time to add Devices. This will let Azure know that a device exists, and should be onboarded and managed. This can actually be done in bulk for scale-out deployments, though we will only add two devices at the moment, so we will use the GUI. On the left navigation, click on "Devices" (you might have to first refresh the page or navigate again to the IoT Hub, once the Resource finishes being created in the previous step).

Click on "Add Device", and you will be asked to provide a name ("Device ID"), and be sure to check the box below that labeled "IoT Edge Device" which will let Azure know this is an edge device running Linux, ready for containers (known as Modules in the Azure terminology). For this demonstration, "Symmetric key" is fine for authentication, but real production systems should use certificates for increased security. See Azure's documentation for information on provisioning keys and certificates. Click "Save" and the device will be created in the IoT Hub portal. You can repeat the process, to add additional devices.

After the devices have been added, click on one of them to reveal some detailed information that Azure has generated. Because we used Symmetric Keys, Azure has created some random strings for us to use to then link the Raspberry Pi to Azure, so that it can be managed and workloads pushed to the device. Of interest is the "Primary connection string", which will be needed in a moment on the Raspberry Pi.

Back on the Raspberry Pi, we can now install the Azure IoT Edge tooling. For ease of use and copy/paste ability, SSH is helpful, though you could type these commands locally on the Raspberry Pi if you have a monitor, keyboard, and mouse connected and you'll end up with the same result.

These next steps all come directly from the Azure Documentation, so refer to their official docs if you receive any errors. This tutorial uses a Raspberry Pi, which is based upon Debian Linux, so the Debian steps are used. Options exist for Ubuntu, RedHat, and Windows devices as well. First, grab the repository setup file and install it:

curl https://packages.microsoft.com/config/debian/11/packages-microsoft-prod.deb > ./packages-microsoft-prod.deb

sudo apt install ./packages-microsoft-prod.deb

Next, install Moby, which is a container runtime:

sudo apt-get update; \

  sudo apt-get install moby-engine

Then run the IoT Edge installation script:

sudo apt-get update; \

  sudo apt-get install aziot-edge defender-iot-micro-agent-edge

At the end of the installation, the IoT Edge package will alert you that the next step is to provide your connection string, which we generated a moment ago in the Azure Portal when adding the Device.

sudo iotedge config mp –connection-string ‘HostName=EdgeImpulse.azure-devices.net;DeviceId=RaspberryPi-1;SharedAccessKey=abc123def456xxxxxxxxxx'

Simply fill in your connection string in place of that sample, placed between the single quotes, that comes from the Portal.

Lastly, apply this change and save it with:

sudo iotedge config apply

The Raspberry Pi, or whichever type of device you chose to use, is now fully setup and linked to Azure IoT Edge. If you refresh the Azure Portal, you should see the device is now connected, though no Modules (workload) exists on the device yet.

Step 1 - Initial Data Collection

The first step in our MLOps loop is going to be data collection and building a high quality dataset to train our model with. Now that Edge Impulse, Azure IoT Edge, and the hardware are setup, we can begin the process and enter this feedback loop.

The Edge Impulse project that we created earlier is still empty, but is ready to accept data. There are lots of ways to connect devices to Edge Impulse, and many ways to capture data. Some of the very easiest methods involve connecting supported devices directly to your computer via USB, and capturing data directly inside the Studio. Smartphones are another great way to easily upload pictures for image classification and object detection computer vision projects. You can refer to the Edge Impulse documentation for more information. In this tutorial we'll take a less direct approach, but with the benefit of bulk deployment at scale and pushing new models over-the-air later, thanks to Azure.

On your development machine, you will need to install Docker. The official documentation is located at https://docs.docker.com/engine/install/, so follow their guidance to reach a point that Docker is up and running on your machine. You should be able to do a docker run hello-world and get confirmation that everything is working, then you're ready to proceed.

Next, we will write a Dockerfile. If you are new to Docker, you'll want to read and learn about how to craft containers, the Dockerfile syntax, best practices, and more. That type of info can all be found in their Docs, and there are many other great resources online for learning Docker as well. When you are ready, make a new directory, create a new file, and copy in this code:

FROM arm64v8/ubuntu:22.04

RUN apt update && apt-get install -y python3 v4l-utils curl sudo libcamera-dev

# Edge Impulse Linux

RUN curl -sL https://deb.nodesource.com/setup_14.x | sudo bash -

ENV DEBIAN_FRONTEND noninteractive

RUN apt update && apt-get install -y gcc g++ make build-essential nodejs sox \

    gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-base gstreamer1.0-plugins-base-apps

RUN npm config set user root && sudo npm install edge-impulse-linux -g --unsafe-perm    

# Edge Impulse SDK (optional)

RUN apt update && apt-get install -y libatlas-base-dev libportaudio2 libportaudiocpp0 portaudio19-dev \

    python3-pyaudio python3-psutil python3-pip ffmpeg libsm6 libxext6 udev usbutils pulseaudio

# (See https://exerror.com/importerror-libgl-so-1-cannot-open-shared-object-file-no-such-file-or-directory/)  

RUN pip3 install edge_impulse_linux -i https://pypi.python.org/simple

RUN pip3 install six

WORKDIR /usr/src/app

COPY start.sh ./

RUN chmod +x start.sh

ENV UDEV=1

EXPOSE 4912

CMD ["sh","./start.sh"]

This is our Dockerfile, and it will install some basic utilities for the Linux container we're building, then install NodeJS, install the Edge Impulse tooling, open up a port, and run a small script we'll create in the next step, called start.sh. This Docker file can be saved, call it literally dockerfile when you save it, and we'll move on to creating the start.sh script.

Again make a new file, and copy / paste in this code:

#!/bin/sh

/lib/systemd/systemd-udevd --daemon

sleep 5

udevadm trigger

sleep 5

edge-impulse-linux --api-key ei_1234567890abcdefghijkl --disable-microphone

This is where we need our API Key that we made note of near the beginning of the tutorial. You can easily retrieve it by simply clicking on "Dashboard", then on "Keys" in the Edge Impulse Studio, and it's displayed for you. Copy / paste the key, and place your key into the last line of the script where the sample one is currently. We need to also make a note here, that this key should be kept secure, and here in this tutorial we are placing it directly into the start.sh file, and are going to place it into the Docker Hub in a Public repository. This is not secure, nor a best practice. However, if you use a Private repository, that would be fine. Or, even better, is to use a variable here and then provide that variable as an input to the Docker container creation, over in Azure. That methodology has the added advantage of quickly being able to switch among Edge Impulse projects simply by altering the variable. However, for demonstration purposes, we'll leave this key in the start.sh script, and proceed.

Save the file, calling it start.sh. With our Dockerfile and the startup script, this container will connect to the Edge Impulse Studio as a camera device, so that we can begin taking pictures of apples, or for more enterprise deployments, collect data from the field. The goal at this point is still go collect data and build a high-quality dataset, and this container will start us on that path. We're now ready to build the container, and then place it somewhere that Azure can reach it.

Docker Hub

Depending on your experience with Docker, or as you may have seen while reading their documentation, containers get built and placed into container registries. You can host a container registry yourself, and store all of your containers on a private server, or even your own local desktop or laptop. However, many developers choose to use existing container registries like Docker Hub, or the Azure Container Registry.

We'll choose Docker Hub here, as it's a popular platform that's easy to use. If you don't already have an account at https://hub.docker.com/, create one (again, a Free account works perfectly fine for this tutorial), log in, and click on your username at the top-right to view the drop-down menu. Click on Account Settings, then click on Security on the left, and then click the "New Access Token" button. This will be used to login to Docker Hub from the command line on your development machine.

In the New Access Token window, provide a name and click "Generate". You will receive a randomly-generated password, that is only shown once. Let's use this to login immediately, then.

In a terminal, type:

docker login -u YourUsernameThatYouCreated

You will be prompted for a password, use the one shown in the New Access Token window. Once logged in, you are ready to build and upload your containers.

Start by first building your container. Be sure to make note of the trailing dot on the end of the line, indicating the current directory. The first build might take a while, but subsequent builds go quicker as layers get cached:

docker build -t edge-impulse-data-collection-container .

Next, tag the image with:

docker image tag edge-impulse-data-collection-container YourUsernameThatYouCreated/edge-impulse-data-collection-container:v1.0

And finally it can be uploaded, by running:

docker image push YourUsernameThatYouCreated/edge-impulse-data-collection-container:v1.0

Similarly, the first upload could take a while, but later uploads are quicker as layers are cached.

Refreshing the Docker Hub, you will see the new container repository that was just created, and you can click on it to see some details about it:

The Container is hosted and ready for deployment at this point. To push it to the Raspberry Pi, it is time to return to the Azure Portal.

Azure IoT Edge Modules

Azure IoT Edge uses the term "Modules" to refer to the containers and services that are orchestrated and run on devices. Modules can be pushed over-the-air to one device, or many devices, and there are very detailed methods for controlling the creation and running of services. We will keep things rather simple in this tutorial, but refer to the documentation for extremely granular deployment options and advanced capabilities of Azure IoT Edge.

In the Azure Portal, click once again on Devices, then click on the name of one of your devices. We'll start off deploying to only one of the Raspberry Pi's, to ensure everything is working. Click on "Set Modules" near the top:

Then, near the middle of the page, click on the "+ Add" drop down menu, and choose "IoT Edge Module":

This is where we will instruct Azure to look for the container we pushed to Docker Hub, and we'll add a few extra instructions to open up a port, set the container to "Privileged" so that it can access the USB Webcam (the are more secure methods to expose only specified pieces of hardware from the host system, so be sure to read the Docker documentation on the topic for enterprise deployments), and give it a friendly name to identify the service. Make note that the URL to enter into the "Image URI" field is slightly different: docker.io is used here, as opposed to hub.docker.com. Thus, you will use docker.io/YourUsernameThatYouCreated/edge-impulse-data-collection-container:v1.0:

Next click on "Container Create Options", in the middle of the page, and copy / paste in this JSON to add the features we need:

{

    "Env": [

        "UDEV=1"

    ],

    "HostConfig": {

        "Privileged": true,

        "PortBindings": {

            "4912/tcp": [

                {

                    "HostPort": "4912"

                }

            ],

            "4912/udp": [

                {

                    "HostPort": "4912"

                }

            ]

        }

    }

}

Finally, click "Add", then back on the Set Modules page click "Review + Create". You will be presented with a summary of the deployment, and you can click "Create" to start our container deployment. After a moment, you can refresh the Device Details page, and see that the Module is now "Running". (The first container download may take a few minutes, later downloads are quicker again due to layer caching).

The dashboard says that the Module is "running", so, we should have our data pipeline created and we should be able to start collecting data over-the-air from the Raspberry Pi. Data in this project consists of images of apples, but your data could of course be anything: images, video, sensor data, audio, IMU, or any other information collected at the edge.

To determine if the process did indeed work, in the Edge Impulse Studio navigate to "Devices", and the Raspberry Pi should have appeared in the list:

Next, click on "Data Acquisition". You should see a preview of the camera feed, and type in a Label for the type of data that you are collecting, in this case "apple". When you are ready, click on "Start Sampling" and the picture will be taken, and placed into your dataset.

Having one picture of an apple is a nice start, but a high-quality dataset consists of hundreds, or even thousands of samples. There should also be adequate variation in the data, for example different angles and movement of the apple, different levels of lighting, pictures that are taken closer, and some that are taken farther away. There should also be variation in the apples, so using many different apples is helpful, as their patterns, colors, and shapes will vary. The background should also be varied, so that the neural network doesn't start to believe that all objects in a specific setting or backdrop are an apple (or whatever object you are using).

Thus, this data collection process needs to be treated with care, and attention should be paid to the quantity and quality of the data; it will take time to build a robust dataset that produces a high-quality model. In the field, it may be necessary to collect a few weeks worth of sensor data, depending upon the frequency of collection and variation in the data.

For this exercise, go ahead and collect approximately 100 to 150 pictures of the apple, rotating it, moving it, and changing the angle and lighting a bit if possible as well.

Once the pictures are collected, we need to "Label" the apple, and identify the location of the apple within the frame. This information will be used later in when the neural network is created and model is built. Click on "Labeling Queue" at the top, to begin this process. The first image is loaded, and you can click and drag a bounding box around the apple in the image.

Click on "Save labels" once the box is drawn, and the next image in the dataset will automatically load, with the bounding box retained. You can move the box a bit if you need to, and then click "Save labels" once again. Repeat this until all of the pictures have been labeled, it will go quickly with the help of the bounding box following the apple from image to image.

When you reach the end of the Labeling Queue, and all of the pictures have a bounding box, click on "Impulse Design" on the left menu, to begin constructing a neural network.

On the "Impulse Design" page, the first item is already pre-populated, "Image Data". You can bump up the Input Resolution from 96 pixels x 96 pixels, and instead enter 320 x 320 pixels, which will give us better accuracy, at the cost of performance. However, the Raspberry Pi is strong enough to still run this; it is more critical to evaluate performance versus power consumption and hardware capability when using microcontrollers, or when environmental considerations need to be accounted for (limited power, solar and battery scenarios, heat produced by the device, etc.)

With the resolution increased to 320 pixels by 320 pixels, click on "Add a processing block".

The Studio will only offer one selection here, "Image", so go ahead and click "Add" to add it into the pipeline. Next, in the Learning Block, click to add a Block, and then select "Object Detection (Images)". You may see a few other options for hardware specific accelerators, and if you are using one of those you might see increased performance on that hardware, but for this Raspberry Pi the standard selection is what is needed. In the end your pipeline will be ready, and you can click on "Save Impulse".

Next, on the left-hand navigation, click on "Image", to configure the Block and set a few options. On the first panel, you can choose whether to use color (RGB) or Grayscale, again having enough computer power with the Raspberry Pi, we will choose RGB. Click "Save parameters".

Once saved, click on "Generate features" near the top, and then click the green "Generate features" button to start the process.

Upon completion, we'll receive a visual representation of the dataset, in this particular case there is only one class (apple), so it's not terribly interesting, though this feature is very useful to visually check for data clustering on larger and more diverse datasets. When ready, you can click on "Object Detection" on the left, to begin the model setup and training.

On the "Object Detection" page, default values will be entered for Number of Training Cycles (epochs), Learning Rate, and Validation set size. Leave them alone for now, but if the model accuracy is too low, we can come back and alter them to improve our model. In the "Neural network architecture" section, FOMO is automatically selected. However, FOMO is designed for more resource-constrained devices like MCU's, so for this demonstration we will increase to the larger MobileNetV2 SSD model. Click on "Choose a different model" and select "MobileNetV2 SSD FPN-Lite 320x320". Then click the "Start Training" button.

It will take a few minutes for the model to be built, but at the end of the process you should see "Job completed" and receive an F1 Score, which is an estimation of the model's accuracy.

This model resulted in an 87.2% accuracy estimation, which is not too bad and definitely sufficient for this demonstration. With all of the data collected, labeled, and a model built, the first part of the MLOps lifecycle is complete, and we can move on to the next part of the loop, deploying our model.

Step 2 - Model Deployment

At the moment, our Raspberry Pi is setup to collect data and upload results into the Edge Impulse Studio. So, we'll need to make a change to the workload running on the Raspberry Pi, and instead direct the device to perform local inferencing using the Edge Impulse object detection model we just built in the previous step.

The steps to make this change are quite similar to what we've already done: We will create a Docker container, upload that container to Docker Hub, and then provision it over-the-air using Azure Iot Edge. These steps will actually be very easy, thanks to the work we've already done.

To begin, make a new folder on your development machine, and copy / paste the existing dockerfile and start.sh files we used in the last step, into the new folder. Open up the start.sh script, and make one small (but important!) change. On the last time, change edge-impulse-linux to edge-impulse-linux-runner, like so:

Save the file, keeping in mind the same note we discussed earlier about the use of the Key directly in the start.sh file here. When going to production and scaling enterprise applications, this is fine if you use a Private container repo, or even better is to replace this with a variable. But for demonstration purposes, we'll go ahead and leave it in the script so you can see how it works. Next, we will do a similar Docker "build", "image tag", and "image push", like we did previously. Specifically, from within this new directory with newly updated start.sh, run the following commands:

docker build -t edge-impulse-runner-container .

docker image tag edge-impulse-runner-container davidtischler/edge-impulse-runner-container:v1.0

docker image push davidtischler/edge-impulse-runner-container:v1.0

Once this completes, in Docker Hub, you will have the new container ready for use:

And then back in Azure, we can push the container to the Raspberry Pi (or any number of Raspberry Pi's or your selected device type), by heading back to the device details page and once again clicking on "Set modules", clicking the drop-down menu called "+ Add", and choosing "IoT Edge Module".

In the container creation details, we will again use very similar settings as used during the data-collection container setup. First, provide a Module name that identifies the container, then provide the Image URI, which will be docker.io/YourUsernameThatYouCreated/edge-impulse-runner-container:v1.0. Then, click on "Container Create Options" and insert the same snippet we used earlier, which opens the port and sets the container to "Privileged" (again, recall, there are more secure ways of exposing only specific pieces of hardware, but for simplicity in this demo we'll give it this access).

{

    "Env": [

        "UDEV=1"

    ],

    "HostConfig": {

        "Privileged": true,

        "PortBindings": {

            "4912/tcp": [

                {

                    "HostPort": "4912"

                }

            ],

            "4912/udp": [

                {

                    "HostPort": "4912"

                }

            ]

        }

    }

}

Click the "Add" button at the bottom of the page,, to return to the "Set modules" page. You'll notice that both the "data-collection" and "inference-runner" containers are displayed, but we no longer need the "data-collection" container and intend to replace it. To the right, you can click the "trash can" icon, to remove the "data-collection" container from our deployment.

Finally, click "Review + Create", then confirm the details by clicking "Create". Within a few minutes, Azure will instruct the device to delete the existing container, and will download the new workload from Docker Hub. This could also take a few minutes, but then refresh the device details page and you will see the new Module has replaced the previous Module:

With this new service running, our inferencing should be occurring. Check to see if this is the case by going to the IP address or hostname of the Raspberry Pi (assuming you are on the same network, or a fully qualified domain name if your device is remote), followed by port 4912. In this example, the device is on the same network, so http://192.168.0.128:4912 is the URL to use.

Sure enough, our object detection model is running, and we are detecting apples with about 95 to 97 percent accuracy!

This completes the first iteration of the loop, and we've now fully demonstrated a data collection, model creation, and model deployment pipeline or pass through an MLOps loop.

However, running this model indefinitely is not feasible, as data can continue to be collected, and environmental conditions might change. This is why the ability to update devices and add improved models, added features, or new capabilities is critical. To demonstrate the need to adapt, let's now imagine some new, previously unseen data has been identified: a banana.

Step 3 - Model Retraining, and Redeployment

Introducing a banana exposes a flaw of our existing model. It thinks nearly anything placed in front of the camera, is an apple.

Thus, we need to provide more and varied data to build a stronger neural network, and ultimately a better model. With Edge Impulse, Azure IoT Edge, and Docker, you simply pass through your MLOps loop again to mitigate this issue. We'll collect new data (and label it), build a new model, and push it once again over-the-air to the device, increasing the intelligence and adding the ability to identify and locate the new object, a banana in this case.

First, we can revert our running inference container to our "data-collection" container, to place our device back into a state where it collects images and uploads them to the Edge Impulse Studio. In Azure, click on the device, click on "Set Modules", click on the drop-down menu called "+ Add", and choose "IoT Edge Module", and then on the "Add module" page enter the same URI used in Step 1: docker.io/YourUsernameThatYouCreated/edge-impulse-data-collection-container:v1.0. Also as usual, click on "Container Create Options" and of course enter the same JSON snippet to open ports and set Privileged:

{

    "Env": [

        "UDEV=1"

    ],

    "HostConfig": {

        "Privileged": true,

        "PortBindings": {

            "4912/tcp": [

                {

                    "HostPort": "4912"

                }

            ],

            "4912/udp": [

                {

                    "HostPort": "4912"

                }

            ]

        }

    }

}

Click the "Add" button at the bottom, then back on the "Set modules" page click the trash can icon next to the "inference-runner" container, to remove that one from the deployment. Click "Review + Create", and confirm the choices with the "Create" button. As usual, give it a few minutes for the device to update.

This should once again give us access to the device inside of the Edge Impulse Studio, for image acquisition. Head back to the Studio, click on Data Acquisition, and sure enough you can see the camera feed. Click "Start Sampling" to take pictures of the banana, preferably in varying positions, with varying lighting, and zooming in and out to get closer and further. Like before, a high-quality dataset, leads to a high-quality model.

Once you have enough images collected, click on Labeling Queue at the top, and again draw bounding boxes around the items of interest, and then click "Save labels", like so:

Repeat the process for all the images, like last time the bounding box will attempt to follow the object that you are labeling, so it should move along quickly. Once finished and there are no more images in the queue,, click on "Create Impulse" on the left.

When the Impulse page loads, you will notice that the right-hand column now reflects two classes, "apple" and "banana", as opposed to only apple previously.

Click on "Image" on the left, to load the details of the Image Processing block. There is no real difference here, once again we will use RGB, so you can click the "Save Parameters" button and then click on the "Generate Features" button on the next page.

When this is done, you can proceed to building the model, by clicking on "Object detection" on the left-hand navigation. The settings here will be the same as what was used on the last training run, and if the defaults worked well for you the first time around, there is no need to change them. Be sure that "MobileNetV2 SSD FPN-Lite 320x320" is still selected for the "Neural network architecture", and click on "Start Training". Like before, this will take some time to complete, and you may need to increase the number of epochs, or alter the settings a bit to improve accuracy if your model is not working well. These are all documented in the Edge Impulse docs at https://docs.edgeimpulse.com/docs/.

Upon completion, this model is scoring 93.7%, which will be fine for demonstration purposes, so we'll proceed to deploying this new model to the Raspberry Pi. Back in Azure, we will follow the same steps as previously, of removing the existing container and adding back our inferencing container instead. In Azure, click on your device, click on "Set modules", click on the trash can icon next to "data-collection", click the "+ Add" drop-down, click "IoT Edge Module", and once again provide a name, insert the URI docker.io/YourUsernameThatYouCreated/edge-impulse-runner-container:v1.0, click "Container Create Options", and add the same JSON snippet we've been using:

{

    "Env": [

        "UDEV=1"

    ],

    "HostConfig": {

        "Privileged": true,

        "PortBindings": {

            "4912/tcp": [

                {

                    "HostPort": "4912"

                }

            ],

            "4912/udp": [

                {

                    "HostPort": "4912"

                }

            ]

        }

    }

}

Then click on the "Add" button, then "Review + Create", then "Create" to redistribute our existing inferencing container back to the Raspberry Pi.

This time, once the container loads (in a few minutes), it will download the newer version of the model that we just created. This newer model should have the ability to detect bananas, if everything goes according to plan. To check, again visit the IP address or hostname of the Raspberry Pi, followed by port 4912, like this as an example: http://192.168.0.128:4912

Sure enough, the new model is running, and we have successfully added net-new capability via an over-the-air deployment of an updated computer vision model.

We have also completed another loop in the MLOps lifecycle, and this process can be repeated continually as new data is gathered, model accuracy improves with additional training, or new application features are developed. Azure IoT Edge gives you the ability to easily update entire fleets of devices, no matter where they are located.

Conclusion

This project is an example of how to build and utilize an MLOps workflow to continually improve and iterate a computer vision application and distribute it to a fleet of edge AI devices. We set up a device (Raspberry Pi), installed Azure IoT Edge, and then used Docker containers and the Docker Hub container registry to install both an Edge Impulse data collection utility, as well as an Edge Impulse inferencing application. We demonstrated how to successfully collect images, build a high-quality dataset, discussed best practices, and walked through the object detection model creation process in Edge Impulse. We showed how to deploy that model via Azure, showed how to then collect more data, retrain the neural network, and finally redeploy the new model to the device, completing a second loop around the MLOps lifecycle. There are many more features and capabilities available within both Edge Impulse and Azure IoT Edge, to allow for enterprise edge AI solutions to be built easily at scale.

Created By: Simone Salerno

Public Project Link: https://studio.edgeimpulse.com/public/508852/latest

GitHub Repo: https://github.com/eloquentarduino/python-edgeimpulse-ota/blob/main/examples/ei_model_ota

Introduction

Continuous deployment in the context of the software development lifecycle refers to the practice of periodically updating your source code, and shipping the latest release to your users without service interruption.

In the context of machine learning, this same concept extends (and is more often put in practice) to the model and its weights. Your first iteration of a model is, actually, the best iteration so far, in the sense that it achieved the best metric (e.g. accuracy) on the data you had available at the time of training.

After this first deployment however, you should be monitoring your model's performance in the field, and keep track of how well it performs on new, unseen data. It is usually the case that model performance degrades over time (so called "model drift") and you need to update the model to remediate this degradation.

This article will show you how you can perform over-the-air (OTA) updates to an Edge Impulse model's weights deployed on a microcontroller in a vendor-agnostic way. You can swap the model weights by loading them from the internet, or from an SD card, or any other storage medium you prefer. The key concepts are that:

1. you'll only update the model's weights, not the entire board firmware (as opposed to most vendor-specific OTA strategies)

2. the process works with any board

Here's the roadmap for the rest of the article that highlights the steps required to make the entire process work:

1. Train a model on Edge Impulse as usual and export as "genuine" Arduino library

2. Patch the library to include the OTA mechanism and deploy the patched version to your board

3. Collect more data, fine tune parameters, improve labelling to generate a better model (not included in this article, specific for your project)

4. Export the updated "genuine" model and convert it into an OTA binary payload

5. Trigger the update process from the board (by periodically pinging a server for updates, by user interaction, etc.)

I created a Public Project available for you to follow along at https://studio.edgeimpulse.com/public/508852/latest. It is a FOMO classifier that recognizes a penguin toy from camera frames. The project has two versions: "bad" is a (deliberately) low accuracy model trained for 1 epoch which can't recognize anything. A second model named "smart" is trained for 50 epochs and has a pretty high accuracy. We will confirm that our OTA process is working by observing our bad model failing on a frame containing a penguin, then being replaced by the smart one that correctly locates the object.

Train First Iteration of a Model

This part is project-specific. I assume you already have a model in Edge Impulse that you want to deploy to your microcontroller. It can be of any type (audio, image, sensor), it makes no difference from the OTA perspective. If you want to clearly see the weights' swapping happening, I suggest you create a bad version of your model which achieves 0% accuracy (e.g. by training the model for just 1 epoch). Then go to the Deployment page and export the model as an Arduino library zip.

Patch the Model with OTA Routines

As stated earlier, the genuine Edge Impulse generated library doesn't support model weight updates natively for microcontrollers. So we need to patch its source code. I created a Python package called edgeimpulse_ota that does the job for you, so this part is completely automated. You can refer to the Colab Notebook for the code.

The patched library is the one that needs to be deployed on your board. Here's an example sketch that runs the FOMO model on a static sample (containing a penguin). When running, you should only see the message "loop", indicating that no object has been detected.

The sketch is available on GitHub.

/**
 * Edge Impulse OTA updates.
 * Open the Serial Monitor and enter "ota"
 * to perform the update.
 * Enter "restore" to restore the original weights.
 * 
 * This project ships with a "dumb" model (default)
 * which can't predict anything. When you enter "ota",
 * a "smart" model is loaded and you will see it detects
 * the penguin in the input data.
 * 
 * When you enter "restore", the dumb model is loaded back
 * and the models makes no more predictions.
 */

#define WIFI_SSID "SSID"
#define WIFI_PASS "PASSWORD"
#define EI_API_KEY "ei_59e01e8cdef245c00a1e5f654f978f806ecebe5f2c58d68b"
#define EI_PROJECT_ID "508852"
#define OTA_SERVER "eloquentarduino.com"
#define OTA_PATH "/edgeimpulse-ota/serve/" EI_API_KEY "/" EI_PROJECT_ID ".bin"

#include <WiFi.h>
#include <fomo-dumb_inferencing.h>


signal_t input;
ei_impulse_result_t result;
WiFiClient http;

float penguin[] = {0x4c584d, 0x4c5750 ... 0x90f0d, 0x90f0d};


bool makeRequest();
void updateWeights(Stream& updateSource);
void restoreWeights();


void setup() {
  Serial.begin(115200);
  while (!Serial);
  delay(3000);
  Serial.println("Edge Impulse OTA over HTTP");

  // connect to WiFi
  WiFi.begin(WIFI_SSID, WIFI_PASS);
  
  while (WiFi.status() != WL_CONNECTED) {
    Serial.println("Attempting to connect to WiFi...");
    delay(2000);
  }
}


void loop() {  
  Serial.println("loop");
  String userInput = Serial.available() ? Serial.readStringUntil('\n') : "";


  // update / restore weights from user input
  if (userInput.startsWith("ota")) {
    if (makeRequest())
      updateWeights(http);
  }

  if (userInput.startsWith("restore")) {
    restoreWeights();
  }


  // run impulse
  if (numpy::signal_from_buffer(penguin, EI_CLASSIFIER_DSP_INPUT_FRAME_SIZE, &input) != 0) {
    Serial.println("numpy::signal_from_buffer failed");
    return;
  }

  if (run_classifier(&input, &result, false) != 0) {
    Serial.println("run_classifier failed");
    return;
  }


  // print results
  for (size_t ix = 0; ix < result.bounding_boxes_count; ix++) {
    auto bb = result.bounding_boxes[ix];

    if (bb.value > 0) {
        Serial.print(" > Found object ");
        Serial.print(bb.label);
        Serial.print(" at x, y = ");
        Serial.print(bb.x);
        Serial.print(", ");
        Serial.print(bb.y);
        Serial.print(" with confidence ");
        Serial.println(bb.value);
    }
  }

  delay(1000);
}


bool makeRequest() {
  // flush request, if necessary
  while (http.connected() && http.available())
    http.read();
    
  //http.stop();
  
  if (http.connect(OTA_SERVER, 80)) {
    Serial.println("connected to server");
    // Make a HTTP request:
    http.print("GET ");
    http.print(OTA_PATH);
    http.println(" HTTP/1.1");
    http.print("Host: ");
    http.println(OTA_SERVER);
    http.println("User-Agent: Arduino");
    http.println("Connection: close");
    http.println();
    delay(3000);

    size_t timeout = millis() + 10000L;
    String response("");

    while (millis() < timeout) {
      if (http.available()) {
        response += (char) http.read();

        // detect start of body
        if (response.endsWith("edgeimpulse")) {
          Serial.println("Response:");
          Serial.println(response);
          
          return true;
        }
      }
    }

    Serial.println("Got response, but cannot detect body");
    return false;
  }
  else {
    Serial.println("Can't connect to OTA server");
    return false;
  }
}


void updateWeights(Stream& updateSource) {
  Serial.println("Updating weights...");
  String updateError = ei_update_weights(updateSource);

  if (updateError != "") {
    Serial.print("Update error: ");
    Serial.println(updateError);
    restoreWeights();
  }
  else {
    Serial.println("Update ok!");
  }
}


void restoreWeights() {
  Serial.println("Restoring original weights...");

  String restoreError = ei_restore_weights();

  if (restoreError != "") {
    Serial.print("Restore error: ");
    Serial.println(restoreError);
  }
  else {
    Serial.println("Restore ok!");
  }
}

The example sketch triggers the model's update when you enter ota into the serial monitor. It is up to you to replace this part with your custom logic. The new model is fetched from the internet in this case, but you can load the binary data from any instance of Arduino's stream class.

The public endpoint at eloquentarduino.com prepends the string edgeimpulse to the OTA payload so that you can easily seek the request at the correct position. The ei_update_weights will start reading the stream from its current position, so be sure you seek it correctly in case you are going to adapt this sketch to your own needs!

Train a New Model

In a real world scenario, after some time you may have collected more data, or improved the label quality, or trained the model for more epochs. No matter how, you have generated an improved version of your model.

What is mandatory here is that the architecture of the model must stay the same!. Let's take the case of a FOMO model, for instance. If your original model is trained on 96x96 pixel, grayscale images using MobileNetV2 0.1, the improved model cannot be trained on 128x128 pixel images, or in RGB mode, or with MobileNetV2 0.3. Since we're only updating the weights, we require that their type, number and layout exactly match the original ones.

If you need to make changes to the model's architecture, you'll need to go through your vendor-specific full-firmware OTA update process instead.

Convert Model to OTA Binary Format

Now that we have a new model, we have to convert it into a format suitable for OTA update. Edge Impulse export options always include the full TensorFlow runtime + Edge Impulse SDK source code to make it self-contained for the target environment. In our case though, the SDK and any additional support code is already flashed on the board: we're only interested in the new weights. Again, we'll leverage the edgeimpulse_ota Python package to do the conversion for us. Refer to the Colab Notebook here for the code. You will produce a .bin file that can be read by the Arduino OTA code to update the model's weights on the fly. How you will serve this file is up to you. The 2 most common use cases are by hosting it on a web server accessible at, for example http://your-domain.com/ei-ota.bin, or saving it to an SD card accessible from the microcontroller.

As a convenience for you, I made a public API endpoint available for free that leverages the Edge Impulse API to serve the model's latest version binary data available at https://eloquentarduino.com/edgeimpulse-ota/serve/<api-key>/<project-id>.bin.

To generate an API key, head to the Project Dashboard in the Studio and navigate to the Keys tab at the top, then click Add a new api key in the right corner. A read-only key will suffice for the purpose.

To get the Project ID, inspect the URL: it will look like https://studio.edgeimpulse.com/studio/123456: those digits are the ID you need.

Since the whole point of OTA updates is to not physically touch the device after deployment, the SD card approach has an interesting collateral use case that is model swapping. You can pre-load multiple models into your SD card and allow the user to choose which one they want. Working with a camera, for example, you can ask the user if they want to recognize dogs, or cats, or penguins... Working with audio, the user could choose their preferred wake word from many available, or swap between voice commands for home automation (e.g. light on, light off) vs media control (e.g. play, pause, next), which would be too many to predict accurately for a single model.

Caveats

Edge Impulse model weights are declared as constant, by default. This means they can't be updated later, on the fly. To overcome this limitation, the edgeimpulse_ota library strips the const modifier and makes them editable. The downside of this process is that now those weights will be stored in the RAM instead of the FLASH memory, and RAM is usually more limited.

Then, there exists an ei_restore_weights() function that allows you to restore the original weights if something goes wrong during the update process (e.g. corrupted OTA payload, or broken internet connection). This is not called by default because it essentially doubles the memory needed to store a model. Given the update payload is read as a unidirectional stream and that the weights are updated inplace, there's no mechanism to selectively revert the weights. Hence, we need to store a full copy of the original weights that won't be altered.

Conclusion

Now you have all the pieces required to perform model weights updates over-the-air for your project:

• a method to patch a genuine Edge Impulse library that allows for weights' replacement

• a method to convert an Edge Impulse model's weights into OTA format

How and when you will improve the model or trigger the update is up to you, and depends on your specific deployment environment. You may have an MLOps pipeline already created that captures data from the field, you can collect more data yourself, or you may even leverage synthetic data. No matter how, you can now have a feedback loop to continuously improve your model's performance.

Created By: Attila Tokes

GitHub Repo: https://github.com/attila-tokes/edge-impulse-expert-projects/tree/main/azure-ml-voice-to-text

Intro

Edge ML enables developers to run Machine Learning (ML) on Internet of Things (IoT) and Edge devices. This offers many advantages, such as reduced power consumption, low latency, reduced bandwidth, and increased privacy.

On the other hand, Edge ML can also be limited in functionality, given the reduced hardware capability of the edge devices. In these cases, it can be a good idea to combine Edge ML with Cloud ML functionality. This is usually done by running an ML model on the Edge device continuously, combined with a Cloud endpoint which is only called by the edge device when advanced functionality is needed.

In this project, I will demonstrate how to create a solution using Edge ML functionality provided by Edge Impulse, in combination with a Cloud ML endpoint implemented with Azure ML.

In this project we will implement a Voice-to-Text solution running on a low power edge device like the Raspberry Pi.

The device will be able to detect a keyword like "Listen!" and then record and translate voice to written text. The keyword detection will be implemented locally using an Edge Impulse model, while the voice-to-text transformation will use a model running in an Azure ML endpoint.

Below is short video showing the project in action:

In the following sections I will describe how such an application can be implemented. We will start with the voice-to-text endpoint implemented with Azure ML, and then we will integrate this into an Edge Impulse application running on the Raspberry Pi.

Cloud ML with Azure ML

Azure Machine Learning is Microsoft's cloud offering of machine learning services, covering the machine learning project lifecycle, including training and inference. It supports all the popular open-source machine learning frameworks such as TensorFlow, PyTorch and others.

In this section I will show how to implement a voice-to-text translation endpoint with Azure ML.

The Model

The machine learning model we will use for voice-to-text transformation is the Wav2vec 2.0 NLP framework, more specifically a pre-trained version of it. Wav2vec 2.0 is the second version of a speech recognition model developed by Facebook / Meta researchers:

Wav2vec 2.0: Learning the structure of speech from raw audio https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/

A pre-trained version of Wav2vec 2.0 is available through the 🤗 Transformers library. The pre-trained model supports both PyTorch and TensorFlow libraries. We will use it with PyTorch.

Getting Started with Azure ML

The functionality offered by Azure Machine Learning is accessed via the Azure ML Studio.

As a prerequisite to accessing Azure ML Studio, we will need an Azure account and an active Subscription. Users whoa are new to Azure can also create a free account, with one year of free services and some credits for experimentation.

Opening the Azure ML Studio brings us to a welcome page:

Here we can create a new Workspace, if we don't already have one:

When we enter the workspace we want to use, a page with an overview and quick actions is shown:

Jupyter Notebooks

On the workspace overview page there are a couple of quick actions to choose from. I think Notebooks can be a good starting point. Notebooks allows us to work with a custom version of Jupyter Notebook, a tool which should be familiar for most people involved with ML projects.

On the Notebooks page, we can either choose to create a new notebook, or to an upload existing one. I went ahead and created a new notebook, as I wanted to experiment with the Wave2Vec 2.0 model.

The "Wave2vec 2.0 Demo" notebook I used can be found here.

The Notebooks interface is similar to that of a standard Jupyter install, but to run code we need an Azure Compute Instance:

The compute instance can be created on-the-fly when we try to run the notebook:

(note: choosing the smallest and cheapest options should be sufficient)

It takes a couple of seconds for the instance to be started, after which we should be able to run the demo. What it does is:

• downloads a sample audio file (WAV), with a person saying: "She had your duck soup and greasy washwater all year"

• downloads a pre-trained version of the Wave2Vec 2.0 model (wav2vec2-base-960h)

• runs the model on the sample audio file, and shows us the resulting transcript

ML Endpoints

Notebooks are a good way for experimenting with ML models. But, in order to make use of the functionality offered by the model, we need a way to expose the model for consumption by other components.

One way to do this is by using Azure Machine Learning Endpoints. Endpoints allows us to expose ML functionality over HTTPS endpoints, with features like SSL termination, authentication, DNS names and canary releases provided out-of-the-box.

In order to deploy an ML Endpoint we need to setup two things: a Model and an Environment.

The Model contains a machine learning model packaged in some form. Supported formats are Score Model, MLFlow and Triton. The Score Model is the easiest option to implement. All we need is a Python "scoring file" of the following form:

# The init() is called once at start-up
def init():
    # ... initialize model ...
   
# The run() method is called each time a request is made to the scoring API.
@input_schema(...)
@output_schema(...)
def run(data):    
    # ... run inference, and return the result ...

Using this file Azure ML will create a simple web server that exposes a /score endpoint. This endpoint can be accessed using a simple HTTP call.

The scoring file for our voice-to-text application can be found in the scoring-func/score_audio.py file.

We can upload this to Azure ML from the Models page:

• first we need to select the "Custom" model type, and upload the scoring-func folder

• then we choose a name

• and register the model

Next we need an Environment in which the model can run. The Environment will define aspects such as the OS, Python version, and libraries installed in the Docker container where the Model will run.

There are two types of models we can use:

• Curated Environments - these are ready-to-use environments created by Microsoft, and they have popular ML frameworks like TensorFlow or PyTorch pre-installed

• Custom Environments - can be used we need custom libraries, or something that is not already present in the curated environments

As our model uses custom Python libraries like transformers, we need a Custom Environment. This can be created from the Environments page. We can choose to start from a Curated Environment, or we can use our own Dockerfile. After multiple tries, I ended up creating a Custom Environment based on the mcr.microsoft.com/azureml/pytorch-1.10-ubuntu18.04-py37-cpu-inference image.

This is PyTorch based image, supporting only CPU inference. It also has Python 3.7 and the transformers library installed.

After this we should be ready to create an Endpoint. In the Endpoints page we need to do the following:

• choose a name, the compute type "Managed", and "Key-based authentication"

• select the model we created earlier

• on the Environment page we select our Custom Environment

• choose a VM type, and set the Instance count to 1

• review and confirm the settings, and click Create

The provisioning of our endpoint will take a couple of minutes. When the endpoint is ready it should like something like:

In order to consume the endpoint, we need to take note of the "REST endpoint" listed above, and as well the API Key from the Consume page:

Using these two pieces, we should now be able to make HTTP calls to our ML endpoint:

POST /score HTTP/1.1
Host: <name>.<region>.inference.ml.azure.com
Authorization: Bearer <api-key>
...
<data>

The endpoint accepts audio input as a numeric array in the data field. To call it with a real audio file we can use a client side Python script like this:

from scipy.io import wavfile
import requests
import librosa

# Read audio file
file_name = 'sample.wav'
input_audio, _ = librosa.load(file_name, sr=16000)

# Make request to the Azure ML endpoint
r = requests.post('https://ml-endpoint-voice-to-text.westeurope.inference.ml.azure.com/score',
             json={"data": input_audio.tolist()}, headers={"Authorization":"Bearer 4h6ic..."})

print("Status code: %d" % r.status_code)
print("Result: %s" % r.json())

This should produce the same result as the one seen in the Jupyter notebook example:

$ python3 sample-client.py

Status code: 200
Result: {'result': 'SHE HAD YOUR DUCK SOUP AND GREASY WASHWATER ALL YEAR'}

In the Monitoring tab we can see metrics like Request count and latency:

Kubernetes Compute

Up to this point, we have used the Managed Compute Instances / Clusters with Azure ML. Managed Compute Instances are Azure VM instances with lifecycle, OS updates, and software stacks fully managed by Azure. When using Managed Compute Instances we are able to select the VM instance type and size. Clusters can have either a fixed number of VM instances, or varying number of VM instances managed by auto-scaling functionality. Virtual Machines with dedicated GPUs are also supported.

Along with Managed Compute Instances, Azure ML also supports several other instance types. The most notable is Kubernetes based compute clusters. Kubernetes is a widely used open-source container orchestration system. It supports automatic deployment, scaling and management of container based application. Thus, it is a great choice for cloud-based systems.

Azure ML supports two types of Kubernetes compute clusters:

• Azure Kubernetes Service (AKS) cluster - these are fully managed clusters offered by Azure

• Azure Arc enabled Kubernetes clusters - these are customer managed clusters connected to Azure via Arc

Running machine learning workloads on an already existing Kubernetes cluster can have many advantages, such as better resource utilization and scalability. On the other hand, setting up a Kubernetes compute cluster is not as easy, so using a managed solution can be helpful.

To set a Kubernetes compute in Azure ML, first we need to install the Azure Kubernetes ML Extension to our K8S cluster. For this project, I used a Azure Kubernetes Service (AKS cluster) which looked like this:

The Azure ML Extension can be installed using Azure CLI, by running the following command:

$ az k8s-extension create --name <cluster-name> --extension-type Microsoft.AzureML.Kubernetes --config enableTraining=True enableInference=True inferenceRouterServiceType=LoadBalancer allowInsecureConnections=True inferenceLoadBalancerHA=False --cluster-type managedClusters --cluster-name <cluster-name> --resource-group <resource-group> --scope cluster

Azure ML CLI & SDK

Azure ML Studio offers a good visual UI for creating and managing Azure ML resources. For people using Azure ML for the first time, it offers a great overview of how to get started and what features are available on the platform.

Additionally, Azure ML also has a CLI, and Python SDK for direct interaction from a console and code:

What is Azure Machine Learning CLI & Python SDK v2? https://docs.microsoft.com/en-us/azure/machine-learning/concept-v2

The Azure ML CLI and Python SDK enable engineers the use MLOps techniques. Similar to DevOps, MLOps is a set of practices that allows the reliable and efficient management of AI / ML application lifecycle. It enables processes like:

• deployment automation

• consistent and repeatable deployment

• ability to create / manage / deploy resources programmatically

• continuous integration and development (CI/CD)

Edge ML with Edge Impulse

Edge Impulse is the leading development platform for Edge Machine Learning (Edge ML). It enables the creation of smart solutions via efficient machine learning models running on edge devices.

As a demonstration we will implement a voice-to-text application on a Raspberry Pi. The solution will feature a keyword spotting model implemented with Edge Impulse, as as well the Cloud ML endpoint we created in the previous section.

Hardware

The hardware we will use is a Raspberry Pi 4 (2GB) development board, along with a Logitech USB headset used as the microphone input.

The Raspberry Pi 4 is a relatively low power single board computer, popular among makers. It is a fully supported Edge Impulse development board. As a note, we are using a Raspberry Pi 4 mostly for convenience. The project probably could be implemented on any of the supported development boards with a microphone and Internet connectivity. The tools / programming languages may differ.

The Raspberry Pi 4 can be set up the standard way. The Raspberry Pi OS is flashed to an SD Card, then we set up network connectivity / Wifi and SSH access. The official documentation describes in great details how to do this:

Setting up your Raspberry Pi https://projects.raspberrypi.org/en/projects/raspberry-pi-setting-up/0

Next, there are couple steps to be done in order to connect the device to EdgeImpulse. The goal is to install the edge-impulse-linux utility, which can be done as follows:

$ curl -sL https://deb.nodesource.com/setup_14.x | sudo bash -
$ sudo apt install -y gcc g++ make build-essential nodejs sox gstreamer1.0-tools gstreamer1.0-plugins-good gstreamer1.0-plugins-base gstreamer1.0-plugins-base-apps
$ npm config set user root && sudo npm install edge-impulse-linux -g --unsafe-perm

After running these commands, we should be able to connect to Edge Impulse Studio by running:

$ edge-impulse-linux --disable-camera

The full set of instructions can be found in the official guide.

Audio Project with Raspberry Pi

Next, we can login to the Edge Impulse Studio, and create a new project:

and select the Audio project template

At this point, among other things, Studio offers us to connect a device to this project.

After we select "Connect your development board", we need to launch edge-impulse-linux on the Raspberry Pi:

The tool asks us to login to Edge Impulse, select a project and a microphone to be used. After completing these steps the device should show up the in the Devices tab:

Data Collection

Now we can start building our keyword spotting model. Edge Impulse has a great tutorial on this:

Responding to your voice https://docs.edgeimpulse.com/docs/tutorials/responding-to-your-voice

The first step in training a keyword spotting model is to collect a set of samples of the word we want to detect. This can be done in the Data Acquisition tab:

In our case the word we want to detect is "Listen!", so I collected about 3 minutes of audio data, which contained about ~ 130 samples of the word "Listen!":

Initially the data collection produces a single sample. This needs to be split up, so that each sample contains one instance of the word. Fortunately, this is easily done by selecting the Split Sample option from the context menu:

As a note, I ended up re-doing the data acquisition process, as I realized the recorded audio had a 50Hz mains interference noise picked up from the power supply of the Raspberry Pi. To fix this, I switched to using a power bank instead of a wall power supply and re-did the data collection.

Along with the recorded keyword samples, we will also need some sample for other categories such as "Noise" and "Unknown" words. Luckily Edge Impulse already has a pre-built keyword spotting dataset, which contains samples for these classes.

To use these samples we can:

• download the dataset to the Raspberry Pi

$ wget https://cdn.edgeimpulse.com/datasets/keywords2.zip
$ unzip keywords2.zip

• reduce the number of samples to about ~130 per class (so that it matches the Listen! samples we have):

$ find noise/ | sort -R | tail +130 | xargs -n 1 -I % rm %
$ find unknown/ | sort -R | tail +130 | xargs -n 1 -I % rm %

• use the Edge Impulse Uploader tool to upload the samples to our project:

$ edge-impulse-uploader --label noise --category split noise/*.wav
$ edge-impulse-uploader --label unknown --category split unknown/*.wav

The samples should appear in Edge Impulse, and we should see that samples for the 3 classes (listen, noise, unknown) are evenly distributed:

Training a Keyword Spotting Model

At this point our dataset is complete, and we can start building and training an ML pipeline / Impulse. This is relatively easy, as we can create an Impulse containing:

• a Time series data input with windows size of 1 sec

• an Audio MFCC processing block, which extracts cepstral coefficients from the audio data

• a Classification (Keras) neural network based learning block

• an Output block for our 3 classes

The MFCC _(Mel Frequency Cepstral Coefficients)_ block extracts coefficients from an audio signal. For keyword spotting, training it with the default parameters usually works:

The NN Classifier block is a neural network classifier, that takes the cepstral coefficients produced by the MFCC block, and tries to predict our 3 classes from it. We can train it with the default setting, but we also have the possibility to add some noise and randomness to the inputs:

The overall accuracy I got is 96.8%, which is pretty good. In the Data Explorer section we can see that sample of our keyword (listen) are clearly separated from the unknown and noise samples.

Our Impulse at this point is ready to be used. We can try it out in the Live classification tab.

Raspberry Pi Deployment

The next step is to deploy the model as a standalone app on the Raspberry Pi. One way to do this is to use the edge-impulse-linux-runner app.

The edge-impulse-linux-runner tool automatically downloads and optimizes the model for the Raspberry Pi. Then it runs a sample app that continuously analyses the input audio, and gives the probabilities of the predicted classes:

If we want to modify / extend this application we can make use of the Edge Impulse SDKs offered for Linux development boards. I opted for the Python SDK, which can be installed on the Raspberry Pi as follows:

$ sudo apt-get install libatlas-base-dev libportaudio2 libportaudiocpp0 portaudio19-dev
$ pip3 install edge_impulse_linux -i https://pypi.python.org/simple
$ pip3 install pyaudio

We can also get a set of examples by downloading the following GitHub repository:

$ git clone https://github.com/edgeimpulse/linux-sdk-python

An audio classification example app can found in the examples/audio/classify.py file. We can launch it as follows:

$ python3 linux-sdk-python/examples/audio/classify.py  /home/pi/.ei-linux-runner/models/128115/v1/model.eim

Cloud ML Integration

Now that we have the keyword spotting working, we can develop an app that also takes advantage of the Cloud ML functionality. So, using the Python SDK I created a simple app that does the following:

• detects the "Listen!" keyword using the Edge Impulse model

• when the keyword is spotted, records a couple seconds of audio

• sends the recorded audio to the Cloud ML endpoint for voice-to-text transformation

• displays the result / decoded text

This is what the output of the app looks like:

The app is built up from the following Python classes / files:

• EdgeML / edgeml.py - responsible for running the keyword spotting model, until a given keyword is detected

• Audio / audio.py - contains the audio recording functionality, with silence detection

• CloudML / cloudml.py - responsible for talking to the Cloud ML endpoint

• main.py - the entry point of the app, with a control loop linking the above parts together

The source code of the app can be found in the edgeml/python-app/ folder.

Conclusions

Using a combination of Edge ML and Cloud ML enables the creation of smart solutions with advanced functionality on low power edge devices. Edge ML is great for simpler tasks such as audio and signal processing, while Cloud ML enables the addition of more advanced functionality that would not otherwise be possible on edge devices.

Platforms like Edge Impulse and Azure ML enable developers to create machine learning solutions, without the need for deep knowledge of machine learning architectures and frameworks.

References

1. Azure Machine Learning Documentation: https://docs.microsoft.com/en-us/azure/machine-learning/

2. Edge Impulse Documentation: https://docs.edgeimpulse.com/docs/

3. Wav2vec 2.0: Learning the structure of speech from raw audio: https://ai.facebook.com/blog/wav2vec-20-learning-the-structure-of-speech-from-raw-audio/

4. Realizing Machine Learning anywhere with Azure Kubernetes Service and Arc-enabled Machine Learning: https://techcommunity.microsoft.com/t5/azure-arc-blog/realizing-machine-learning-anywhere-with-azure-kubernetes/ba-p/3470783

Created By: Avi Brown

Public Project Link: https://studio.edgeimpulse.com/public/108508/latest

GitHub Repository: https://github.com/avielbr/edge-impulse/tree/main/ros2/ei_ros2

Background

ROS2 is the world’s most popular robotics development framework. It enables developers to build organized, modular, and scalable robotics platforms with the full force of the open source community behind it. When developing a new robot, it’s common to recycle existing ROS2 packages instead of writing new ones, saving precious time in development.

In this tutorial we will build a recyclable ROS2 node based around an Edge Impulse machine learning model. This node will draw sensor data from a sensor topic, run the data through an Edge Impulse model, and then publish the results of the machine learning to another topic, to which other nodes in the system can subscribe.

For the sake of demonstration, we’ll be using an accelerometer-based machine learning model trained to recognize “circle”, “up_down”, and “side_side” movements.

In this project we’ll learn how to:

• Build a pub/sub Edge Impulse node

• Fill a sensor data buffer using a subscriber

• Import and use a machine learning model from within a ROS2 node

• Publish the inferences made by the machine learning node to a topic

Equipment & software

• Raspberry Pi 4

• Adafruit MPU6050 accelerometer + gyroscope module (*)

• Ubuntu 20.04 server

• ROS2 Foxy Fitzroy

• Edge Impulse Linux CLI

• VSCode Remote Development extension

(*) BYO sensor For this tutorial we’ll be using a 3-axis accelerometer, but it is general enough to be adapted for just about any type of sensor.

Getting started

This tutorial is meant to be as general as possible, and as such will not go in depth on building an Edge Impulse project / model. If you need help getting started you can find many high quality examples spanning a variety of sensors, boards, and use cases here.

You should also have the Edge Impulse Linux CLI up and running on your Linux board (RPi, Jetson Nano, etc.). Read about installing the CLI here. Test that it is indeed installed by running:

edge-impulse-linux

You should see the CLI initialize.

Creating a ROS2 package

Navigate to your workspace, and from within ros2_ws/src run:

ros2 pkg create ei_ros2 --build-type ament_python --dependencies rclpy std_msgs board os adafruit_mpu6050

This will create a package called “ei_ros2”. Adjust the dependencies to suit whatever libraries your package requires (or edit them using the package.xml file).

Building a node for the accelerometer (or other sensor)

Now we need to build the node that will publish sensor data that in turn will be fed to the machine learning model. Navigate to the “Impulse design” section of your Edge Impulse project, and take a loot at the first block:

This block contains a lot of important information. Let’s break it down:

• First, we see that the block expects to receive three input axes (since it is a 3-axis accelerometer).

• Next, we see that the model expects 2000ms (2 seconds) worth of data each time it is called.

• And finally, the frequency tells us how many times per second the sensor is sampled. In this case, 98 times / second (Hz).

We’ll return to these numbers a couple of times, but for now let’s start building our sensor node!

Make a file called mpu6050_node.py (or whatever your sensor is called) under /ros2_ws/src/ei_ros2/ei_ros2.

1: Imports

Here are the libraries we need for this node. Depending on which sensor you’ll need your imports may look different.

I’m using the Float32MultiArray as the message type, because we want to send an array with three readings (accelerometer X, Y, Z) which are float values.

import board # Sensor specific
import rclpy # Always
from rclpy.node import Node # Always
from std_msgs.msg import Float32MultiArray # Sensor specific
import adafruit_mpu6050 # Sensor specific

2: Node class

Now we need a class with a publisher to handle our sensor. A lot of this code will be familiar to you if you have experience with ROS2.

• Notice the frequency: This should be the same frequency as the one in the “Impulse design” section of your EI project. This will make it so that sensor data is published at the required rate (1 / frequency).

• 98 times per second the read_mpu6050() method is called. Each time the 3 accelerometer axes are read, and sent together in an array to the mpu6050_stream topic. Once it arrives to the topic it becomes available for our machine learning node to subscribe to.

class MPU6050Node(Node):
    def __init__(self):
        super().__init__("mpu6050")
        self.i2c = board.I2C()
        self.mpu = adafruit_mpu6050.MPU6050(self.i2c)
        self.publisher_ = self.create_publisher(Float32MultiArray, "mpu6050_stream", 15)
        self.frequency = 98 # See "Impulse design" block in EI project
        self.timer_ = self.create_timer((1 / self.frequency), self.read_mpu6050)
        self.get_logger().info("MP6050 stream opened.")

    def read_mpu6050(self):
        msg = Float32MultiArray()
        msg.data = [round(self.mpu.acceleration[0], 2), # accX
                    round(self.mpu.acceleration[1], 2), # accY
                    round(self.mpu.acceleration[2], 2)] # accZ
        self.publisher_.publish(msg)

Adding the main() function, the full publisher node for the MPU6050 accelerometer is:

import board
import rclpy
from rclpy.node import Node
from std_msgs.msg import Float32MultiArray
import adafruit_mpu6050

class MPU6050Node(Node):
    def __init__(self):
        super().__init__("mpu6050")
        self.i2c = board.I2C()
        self.mpu = adafruit_mpu6050.MPU6050(self.i2c)
        self.publisher_ = self.create_publisher(Float32MultiArray, "mpu6050_stream", 15)
        self.frequency = 98
        self.timer_ = self.create_timer((1 / self.frequency), self.read_mpu6050)
        self.get_logger().info("MP6050 stream opened.")

    def read_mpu6050(self):
        msg = Float32MultiArray()
        msg.data = [round(self.mpu.acceleration[0], 2),
                    round(self.mpu.acceleration[1], 2),
                    round(self.mpu.acceleration[2], 2)]
        self.publisher_.publish(msg)

 
def main(args=None):
    rclpy.init(args=args)
    node = MPU6050Node()
    rclpy.spin(node)
    rclpy.shutdown()

Now just run chmod +x mpu6050_node.py to make your node file executable, since we're using --symlink-install

Building an AI powered Edge Impulse pub/sub node

We’ve arrived to the fun part! Effort was made to make this next chunk of code as reusable as possible. Let’s go through it piece by piece.

First let’s download our model from Edge Impulse.

• From the terminal, run:

edge-impulse-linux-runner --clean

• After entering your credentials you should see:

• Select the project you want, and hit Enter. Now you should see this message:

Take note of the file location and name. You should now copy and paste this file to the same directory where the sensor / Edge Impulse nodes are located (/ros2_ws/src/ei_ros2/ei_ros2).

Now let’s build the node!

Make a file called ei_node.py (or something) under /ros2_ws/src/ei_ros2/ei_ros2.

1: Imports:

Note that in this case we need both String and Float32MultiArray:

• Float32MultiArray for subscribing to the sensor node we just created

• And String, for publishing the machine learning results to an inference stream

import os
import rclpy
from rclpy.node import Node
from std_msgs.msg import String, Float32MultiArray
from edge_impulse_linux.runner import ImpulseRunner

2: Edge Impulse node class

There’s a lot going on here, but the important parts have numbered comments beside them, so let’s take them one by one:

1. “Change subscription” - Make sure you are subscribed to the correct topic (the same one that the sensor node is publishing to)

2. “Adjust timing” - This timer calls the function that runs the machine learning model. Basically, the function will only run the model once the buffer (which we’ll discuss in a moment) is full of data, so this timer essentially means: How often do you want to check if the buffer is full. If time isn’t an issue, you can set it to check every couple of seconds. Here I want to run the model as soon as I have a full buffer, so I set it to check every 0.01 seconds.

3. “Check file name” - The name of the .eim file here should match the name of the file downloaded via the CLI.

4. “Set max length” - The model expects to receive a specific amount of data, so we need to set the buffer (the array of sensor data that is passed to the model) to a specific length. How do we find the length? If we return to the “Impulse design” section of our Edge Impulse project, we’ll see that the window length is 2 seconds, where the sensor is sampled 98 times per second, each time giving 3 values. So: 2 * 98 * 3 = 588.

Don’t stress too much about this part — if you put the wrong number in, you’ll get an error telling you what the correct number is 🙂

class EdgeImpulseNode(Node):
    def __init__(self):
        super().__init__("edge_impulse_classifier")

        # Sensor data subscriber
        self.subscriber_ = self.create_subscription(Float32MultiArray, "mpu6050_stream", self.callback_fill_buffer, 10) # 1: Change subscription
        # Inference publisher
        self.publisher_ = self.create_publisher(String, "inference_stream", 10)
        self.timer_ = self.create_timer(0.01, self.classify) # 2: Adjust timing if desired

        # Create Classifier object based on Edge Impulse model file
        self.classifier = Classifier('model.eim') # 3: Check file name
        
        self.buffer = []
        self.buffer_full_len = 588 #4: Set max length of buffer array

        self.get_logger().info("Edge Impulse node opened.")

    # If buffer is full, classify and publish result!
    def classify(self):
        # String since we're publishing the class name
        msg = String()

        if len(self.buffer) == self.buffer_full_len:
            # Pass buffer to classifier
            msg.data = self.classifier.classify(self.buffer)
            self.publisher_.publish(msg)

            # Reset buffer
            self.buffer = []

    # Adds to buffer whenever new sensor data is published
    def callback_fill_buffer(self, msg):
        if len(self.buffer) < self.buffer_full_len:
            for val in msg.data:
                self.buffer.append(val)

3: Classifier class

This class handles passing the sensor data to the Edge Impulse model file we downloaded.

Note: The classify() method returns a string with the name of the most likely result. Machine learning inferences look like: {'dog': 0.8, 'cat': 0.2, 'fish': 0.1}. This function returns only the string with the highest probability.

class Classifier:
    def __init__(self, model_path):
        self.runner = None
        self.model_path = model_path

    def classify(self, features):
        dir_path = os.path.dirname(os.path.realpath(__file__))
        modelfile = os.path.join(dir_path, self.model_path)
        self.runner = ImpulseRunner(modelfile)

        try:
            self.runner.init()
            res = self.runner.classify(features)

            # Return classification with highest probability
            return max(res["result"]["classification"], key=res["result"]["classification"].get)

        finally:
            if (self.runner):
                self.runner.stop()

Now we can add the main() function and we’re ready to go!

Full Edge Impulse Pub/Sub code:

# Imports
import os
import rclpy
from rclpy.node import Node
from std_msgs.msg import String, Float32MultiArray
from edge_impulse_linux.runner import ImpulseRunner

# Classificiation node that subs to sensor stream and publishes to inference topic
class EdgeImpulseNode(Node):
    def __init__(self):
        super().__init__("edge_impulse_classifier")

        # Sensor data subscriber
        self.subscriber_ = self.create_subscription(Float32MultiArray, "mpu6050_stream", self.callback_fill_buffer, 10) # 1: Change subscription
        # Inference publisher
        self.publisher_ = self.create_publisher(String, "inference_stream", 10)
        self.timer_ = self.create_timer(0.01, self.classify) # 2: Adjust timing

        # Create Classifier object based on Edge Impulse model file
        self.classifier = Classifier('model.eim') # 3: Check file name
        
        self.buffer = []
        self.buffer_full_len = 588 #4: Set max length of buffer array

        self.get_logger().info("Edge Impulse node opened.")

    # If buffer is full, classify and publish result!
    def classify(self):
        # String since we're publishing the class name
        msg = String()

        if len(self.buffer) == self.buffer_full_len:
            # Pass buffer to classifier
            msg.data = self.classifier.classify(self.buffer)
            self.publisher_.publish(msg)

            # Reset buffer
            self.buffer = []

    # Adds to buffer whenever new sensor data is published
    def callback_fill_buffer(self, msg):
        if len(self.buffer) < self.buffer_full_len:
            for val in msg.data:
                self.buffer.append(val)

# Edge Impulse classification class
class Classifier:
    def __init__(self, model_path):
        self.runner = None
        self.model_path = model_path

    def classify(self, features):
        dir_path = os.path.dirname(os.path.realpath(__file__))
        modelfile = os.path.join(dir_path, self.model_path)
        self.runner = ImpulseRunner(modelfile)

        try:
            self.runner.init()
            res = self.runner.classify(features)

            # Return classification with highest probability
            return max(res["result"]["classification"], key=res["result"]["classification"].get)

        finally:
            if (self.runner):
                self.runner.stop()

def main(args=None):
    rclpy.init(args=args)
    node = EdgeImpulseNode()
    rclpy.spin(node)
    rclpy.shutdown()

if __name__ == "__main__":
    main()

Once again run chmod +x ei_node.py to make your node file executable.

Adding entry points

In your [setup.py](http://setup.py) file add entry points for your nodes:

.
.
.

entry_points={
        'console_scripts': [
            "mpu6050 = ei_ros2.mpu6050_node:main",
            "ei = ei_ros2.ei_node:main"
        ],
    }

We’re ready to build!

Navigate to ros2_ws (or your main ROS2 workspace directory) and build the package using:

colcon build --packages-select ei_ros2 --symlink-install

Once that finishes, we’re ready to for tests!

Testing our Edge Impulse node

Let’s run both the sensor node and the Edge Impulse machine learning node. From two separate terminals run:

ros2 run ei_ros2 mpu6050

ros2 run ei_ros2 ei

And now in a third terminal let’s listen on the inference_stream topic using:

ros2 topic echo /inference_stream

Now let’s move our accelerometer around and see what appears on the inference_stream topic:

Looking good!

To summarize

In this tutorial we looked at how to make a AI-powered pub/sub node in ROS2. This tutorial is part of an Edge Impulse + ROS2 series, and in Part 2 we’ll look at how to use a service / client architecture. The contents of this tutorial can be generalized to suit just about any sensor and use case, so please don’t hesitate to try it out yourself! Feel free to ask questions at forum.edgeimpulse.com, or on the YouTube video version of this tutorial here:

https://youtu.be/0SabLvJqSaM

Created By: Adam Milton-Barker

Project Repo

https://github.com/AdamMiltonBarker/edge-impulse-jetson-nano-trainer

Introduction

A Python program that utilizes the Edge Impulse API to create, train and deploy a model on Jetson Nano.

NVIDIA Jetson Nano

The NVIDIA Jetson Nano is a small yet powerful computer designed for use in embedded systems and edge computing applications. The Jetson Nano is particularly well-suited for use in applications that require machine learning or computer vision processing at the edge of a network.

Its small size and low power consumption also make it a cost-effective and efficient choice for edge computing applications in industries such as robotics, healthcare, and manufacturing.

As an NVIDIA Jetson AI Specialist and Jetson AI Ambassador, I love building projects for the Jetson Nano. I use the Jetson Nano for both my Leukaemia MedTech non-profit, my business, for projects I contribute to the Edge Impulse Experts platform, and for personal projects.

Edge Impulse

Edge Impulse is an end-to-end platform for building and deploying machine learning models on edge devices. It simplifies the process of collecting, processing, and analyzing sensor data from various sources, such as microcontrollers, and turning it into high-quality machine learning models.

The platform offers a variety of tools and resources, including a web-based IDE, a comprehensive set of libraries and APIs, and a range of pre-built models that can be customized for specific use cases.

The Problem

Whilst the Jetson Nano is a highly capable device for edge inference, it may not be the most suitable choice for AI model training, in fact NVIDIA recommend you should not train models on the Jetson Nano. However, Edge Impulse offers a compelling solution to this challenge by providing a platform for developing and deploying models on a range of edge devices, including the Jetson Nano.

That said, some researchers and developers may prefer a more hands-on approach to coding and developing solutions on the Jetson Nano, despite its limitations for AI training.

The Solution

This is where the Edge Impulse API comes in. Edge impulse have a number of APIs, which together, provide the ability to hook into most of the platforms capabilities, including the Studio. In this project we will create a new Edge Impulse project, connect a device, upload training and test data, create an Impulse, train the model, and then deploy and run the model on your Jetson Nano.

Installation

Before you can get started you need to clone the edge-impulse-jetson-nano-trainer repository to your Jetson Nano. On your Jetson Nano navigate to where you want to be and run the following command:

bash
git clone https://github.com/AdamMiltonBarker/edge-impulse-jetson-nano-trainer

Now cd into the directory:

bash
cd edge-impulse-jetson-nano-trainer

And run the following command to install the required software:

bash
sh install.sh

This will install the required software for your program.

The Configuration

You can find the configuration in the confs.json file. This file has been set up to run this program as it is, but you are able to modify it and the code to act how you like. Think of this program as a boilerplate program and introduction to using the Edge Impulse APIs.

At certain points during the program, this file will be update, this ensures that if you stop the program you will always start off from where you left off.

Edge Impulse Account

To use this program you will need an Edge Impulse account. If you do not have one, head over to the Edge Impulse website and create one, then head back here.

Data

You can use any dataset you like for this tutorial, I used the Car vs Bike Classification Dataset from Kaggle, and the Unsplash random images collection for the unknown class.

These datasets include .jpg, .jpeg, and .png files, so we need to update the configuration file to look like the following:

json
"data": {
    "file_type": [
        ".jpg",
        ".jpeg",
        ".png"
    ],
    "test_data": false,
    "test_dir": "data/test",
    "train_data": false,
    "train_dir": "data/train",
    "type": "",
    "types": [
        "local",
        "remote"
    ]
},

You will notice the test_dir and train_dir paths, this is where your data should be placed. The directory names inside of those directories will be used as the labels for your dataset. In this case, you should create car, bike, and unknown directories in both the train and test dirs.

There is a limitation on the number of files you can upload through the API, through my testing I was able to comfortably upload around 500 training per class, and 250 testing images per class.

The Program

The main bulk of the code lives in the ei_jetson_trainer.py file. Ensuring you have your Edge Impulse account set up, let's begin.

Start The Program

Navigate to the project root directory and execute the following command:

bash
python3 ei_jetson_trainer.py

Login To Edge Impulse

The first thing the program will ask you to do is login. Enter your Edge Impulse username or email, and then your password.

What is your username?

What is your password?

For security your username and password are not stored on the Jetson Nano. Each time you use the program you will have to enter them at the beginning of your session.

Project Details

Next you will be asked for a name for your new project.

What is your new project name?

Enter a name for your new project and continue by pressing enter.

Connect Your Device

The prompt will now ask you for your device ID:

What is your new device ID?

At this point you need to follow the instructions in the Jetson Nano documentation on the Edge Impulse website. The program will provide you the link so you can just copy and paste it into your browser.

Once you have installed all of the required software, head over to a new terminal and run the following command:

bash
edge-impulse-linux

Follow the steps given to you and then head to the devices tab on your new project in the Edge Impulse Studio and copy the device ID. Once you have that, head back to the Jetson Nano trainer terminal and enter it into the program.

Data

You should have followed the steps above and all of your training and testing data is in the relevant directories. The program will now loop through your data and send it to the Edge Impulse platform.

** Data ingestion type is local
** Make sure your data is in the data/train and data/test folders.
** Directory names in these folders will be used as the labels.
** 499 bike train data images
** 498 car train data images
** 500 unknown train data images
** 250 bike test data images
** 250 car test data images
** 250 unknown test data images

This may take some time. While you wait you can head over the Edge Impulse Studio and navigate to the Data Aquisition tab and you will be able to see your data being imported to the platform.

Next the program will create the Impulse for you, including all required blocks.

Feature Generation

The next step the program will take is to generate the features for your dataset. This will start a job and the platform will send socket messages to the program to let it know the job has been completed and to continue. While this is happening you can navigate to Impulse Design -> Image -> Generate Features where you will see the features being generated.

Once the platform informs the program that the features have been created, training will begin.

Training

The program will now start training. You can head over to Impulse Design -> Image -> Transfer Learning where you will be able to watch the model being trained. Once the training has finished the results will be displayed and the program will be notified via sockets.

Testing

The program will now begin testing on the test data. You can watch this happening in real-time in the Edge Impulse Studio in the Model Testing tab.

Deploy

Now that our model is trained and tested, it is time to run it on our device. Thanks to Edge Impulse, this step is easy. Make sure you have disconnected your device from the platform, and in terminal run:

bash
edge-impulse-linux-runner 

Your model will be installed on your Jetson Nano and immediately begin classifying. The Edge Impulse runner will give you a local URL you can view the real-time stream and classifications.

Created By: Solomon Githu

Public Project Link:

Introduction

In Machine Learning (ML), we create a model that is trained to do a particular task like object detection, anomaly detection, or prediction. To develop a model, we normally collect data on one computer (possibly in the cloud) and then we train the model on the computer with the centralized data. However, in some situations, using a centralized machine learning model may not be effective or efficient. In some situations, the data may be sensitive, not diverse, or too large for the available internet bandwidth making it unable to be uploaded to the central computer.

Federated Learning enables us to bring the model, to the data. For example, voice recognition and face recognition by Siri and Google Assistant are Federated Learning based solutions. In these cases, we do not want to send our voices or pictures to the cloud for training the model. Federated Learning works by training models locally on the devices using the data on the device. Once a model has been trained, a device uploads the new model updates to a server that aggregates model parameters from various devices and generates a global updated model. This global updated model can then be deployed to the devices for better Machine Learning task performance, and also continuous retraining of the model.

The approach of federated learning normally follows four major processes:

• A central server initializes a global model and its parameters are transferred to clients in each iteration

• Clients update their local model parameters by locally training a model

• The server gets model parameters from clients, aggregates them, and updates the global parameters

• The above steps are repeated until local and global parameters converge

There are several Open-Source Federated Learning frameworks that we can use. However, there are some factors that should be considered before selecting a Federate Learning framework. Some of these factors include:

• The supported Machine Learning frameworks

• Aggregation Algorithms - the most widely supported Federated Learning algorithm is Federated averaging (FedAvg). However, the specific algorithms offered by each framework may vary.

• The supported privacy methods, such as encryption

• The supported devices and operating systems

• Scalability - the complexity of adding your own model or aggregation algorithm

Demonstration

To demonstrate Federated Learning, I simulated a situation where we want to identify if workers at a construction site are wearing safety equipment (hardhats). At each construction site, we have a surveillance camera that is monitoring the workers. The camera device will be taking an image of a person and determining if it sees a head or a hardhat.

Some of the challenges in this use case are:

• how can we overcome sending sensitive photos of workers to the cloud?

• how can we overcome the need to send a lot of image data to a central server for training a model?

• how to acquire diverse data?

There are 6 Federated Learning iterations where both the Raspberry Pi and the personal computer individually train a MobileNetV2 model, send updates to the server, and the server aggregates the model parameters. During the client's training process, each client uses a dataset, different from the other, to train the model. This helps us simulate a situation where we have different devices at different locations and therefore the data is different and more diverse.

For my demonstration, I chose the MobileNetV2 architecture since it is a lightweight neural network architecture that is designed to be efficient and fast, with less computation power requirements. In my previous tests, I trained an EfficientNetB0 model and it achieved almost the same performance as the MobileNetV2 model, but at the cost of a significantly longer training and classification time.

Components and Hardware Configuration

Software components:

• Edge Impulse Studio account

• Python

• Edge Impulse for Linux

Hardware components:

• Personal Computer with Windows or Linux based Operating System

• Raspberry Pi 4 (recommended to use the 4GB RAM version) with Raspberry Pi OS

• Official Raspberry Pi 4 power adapter (recommended)

• Raspberry Pi V2 camera module

Data Collection Process

The public project has a total of 583 images of people's heads and people wearing safety hats. I then split the images according to this:

• two folders with training and test images for two client devices

• one folder with test images for the server model testing during the Federated Learning

• one folder with test images that we can give to the final global model after the Federated Learning

Training the Model, the Federated Way

First, we need computers for the server and clients. You can also use the same computer as both a server and client, provided the computer has enough resources to do that. The minimum number of required clients is two for the Federated Learning to start. This minimum number can be modified in the server.py code, but remember to also modify the client.py code to load datasets for the additional clients.

I decided to use my personal computer as the server and also as one client device. For the other client device, I decided to use a Raspberry Pi 4 with 4GB of RAM.

In my test with Raspberry Pi 3's running as the client devices, they managed to train a model but failed at the model evaluation process. This can be related to the fact that the Raspberry Pi 3 is more resource constrained than the Raspberry Pi 4, with a less powerful CPU and less RAM. Using the top command on the Raspberry Pi 3's showed that the CPU and RAM usage were at max capacity during the training process. When it reached the evaluation process, the RAM usage decreased to around 80%, CPU usage dropped to around 40%, but then the Federated Learning framework disconnected the Raspberry Pi 3 client devices. The Raspberry Pi 3's also showed 92% CPU usage and 45% RAM usage when they were connecting as the client devices.

Next, we need to install dependencies on the devices. The difference between the server and client dependencies is that the server computer uses the Edge Impulse Python SDK for profiling and deploying the model. We can install dependencies on the server computer by running the command below on a terminal or a Command Prompt (CMD):

To install the dependencies on the Raspberry Pi 4 running as a client device, we use the command below:

Next, we need to update the server_address value in both server.py and client.py with the IP address of the device running as the server. If you get an error message from server.py that says _ERROR_MESSAGE_PORT_BINDING_FAILED, change the server's port to another one that is available.

We can now run the Federated Learning system. I first start the server on my personal computer by running python server.py. The server will load the test images, initialize the global model parameters, evaluate the initial model's parameters, and then wait until at least two clients join before starting the Federated Learning.

Next, I start one client on my personal computer by running python client.py --client_number=1 in a Command Prompt (CMD). When running the client scripts we use the argument client_number to enable the script to load different datasets for each client using the two folders with the client's dataset.

I then start the second client on the Raspberry Pi 4, by running the command python client.py --client_number=2.

Once the two clients have connected, the Federated Learning will start. Each client will load a MobileNetV2 model, train the model using the train data, evaluate the model using the test data, and then send model updates to the server. In each Federated Learning iteration, the clients train a model with 20 epochs and a batch size of 8. The sever then aggregates the model's parameters from the updates sent by the clients, and then updates the initial model with the new parameters. This process continues six times, and then the Federated Learning will be completed.

Finally, when the Federated Learning is complete, I added some code on the server script to test the final global model with the test images that were not used during the Federated Learning. In the server's logs, we can see that the global model gives an accuracy of 1.0 in all the Federated Learning iterations. This, however, does not suggest that our model is perfect. Our dataset is still relatively small with only 415 images, equally divided for the two client's training dataset. Also, since this is transfer learning, our head and hardhat images are not very complex objects and the pre-trained model may require a bit of fine-tuning to make it learn the new task.

Testing the Global Model

When we go to the Edge Impulse project, we will see "Upload model" under "Impulse design". This is because our final global model was uploaded to the project during profiling.

We first need to configure some parameters on the Edge Impulse project. Click "Upload model" and a new interface will open on the right side of the page. Here, we need to select "Image (RGB)" for the model input since our model is using RGB images. Next, for the input scaling query, we select "Pixels ranging 0..255(not normalized)". Afterwards, we select "Classification" for model output since this is an image classification model. Finally, the output labels should be: head, hardhat. Click "Save model" to finish the configuration.

Afterwards, we can upload a test image to see if the selections we made are correct. In this test image, we can see that even though person is occupying a relatively small area portion of the image, the model was able to correctly determine that this is a hardhat image.

Perfect! Now we have a Federated Learning model added to Edge Impulse.

A new interface will open. Here we can first choose "Select a folder" for the upload mode. Click "Choose files" and select the dataset_test directory on your computer from where you cloned the GitHub repository to. Next, select "Testing" for the upload category since we have already trained a model and therefore there is no need to have training data. Next, for Label we select "Leave data unlabeled". Finally, click "Upload data" and the images will be uploaded to the project. The uploaded images can be seen by going to "Test" in Data acquisition.

The last thing to do is to label the images. This label information describes what each image is, head or hardhat. The label information will also be used during the model testing by comparing the models output to the correct class (label). To label the images, first click the kebab menu (three dots menu) next to each item listed in the test data. Next, select "Edit label" and type the name of the class which the image belongs to: head or hardhat. Do this until all images have been labelled.

Finally, when all the images have been labeled, we can click "Model testing" and afterwards "Classify all". This will test the model on all the test images, determine the model's performance and also create a confusion matrix. From my test, the model achieved an accuracy of 93%. However, for a more robust model, we still need to train the model on more data, and more times. For my demonstration, I chose this result as an acceptable performance.

Result

First, we need to attach the Raspberry Pi camera to the board.

Next, we need to install Edge Impulse for Linux dependencies on the Raspberry Pi 4. To do this, we can run the commands below on the Raspberry Pi:

Afterwards, we need to activate the camera interface on the Raspberry Pi 4 for the camera module. We can run the command sudo raspi-config and use the cursor keys to select and open Interfacing Options, then select Camera, and follow the prompt to enable the camera. Finally, reboot the Raspberry Pi by running the command sudo reboot.

Once rebooted, we can download the final global model from the Edge Impulse project by running the command below. You will be prompted to input your username and password for your Edge Impulse account, followed by a prompt to select the Edge Impulse project.

In the command, we pass the name of the downloaded .eim file, modelfile.

We can go to the provided URL (Raspberry Pi's IP address at port 4912) and we will see the feed being captured by the camera as well as the model's predictions. At this point I used a 3D printed support to hold the Raspberry Pi camera upright and then projected the test images to the camera.

Below is a demo video of live classification on the Raspberry Pi 4. We can see that the model predicts the correct class for each image.

Conclusion

From the demonstration, we have seen that we can obtain more accurate and generalizable models through Federated Learning, without requiring the data leave the client devices. Federated Learning has a lot of potential. It prevents sending sensitive information like healthcare records, financial records, or similar across the internet. Since the training occurs from multiple data sources, we can also get more diverse data enabling us to come up with more robust models, that perform better at their tasks.

An excellent progression of this demonstration would be to implement the Federated Learning system with a different Machine Learning model framework, and adding more clients and data to the system. Additionally, we can also reinforce the system by implementing automated deployments, whereby a final global model is automatically deployed on edge devices from an Edge Impulse project.