Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Welcome to Edge Impulse! If you're new to the world of edge machine learning, you've come to the right place. This guide will walk you through the essential steps to get started with Edge Impulse, a suite of engineering tools for building, training, and deploying machine learning models on edge devices.
Check out our Edge AI Fundamentals course to learn more about edge computing, machine learning, and edge MLOps.
Edge Impulse empowers you to bring intelligence to your embedded projects by enabling devices to understand and respond to their environment. Whether you want to recognize sounds, identify objects, or detect motion, Edge Impulse makes it accessible and straightforward. Here's why beginners like you are diving into Edge Impulse:
No Coding Required: You don't need to be a coding expert to use Edge Impulse. Our platform provides a user-friendly interface that guides you through the process - this includes many optimized preprocessing and learning blocks, various neural network architectures, and pre-trained models and can generate ready-to-flash binaries to test your models on real devices.
Edge Computing: Your machine learning models are optimized to run directly on your edge devices, ensuring low latency and real-time processing.
Support for Various Sensors: Edge Impulse supports a wide range of sensors, from accelerometers and microphones to cameras, making it versatile for different projects.
Community and Resources: You're not alone on this journey. Edge Impulse offers a supportive community and extensive documentation to help you succeed.
Ready to begin? Follow these simple steps to embark on your Edge Impulse journey:
Start by creating an Edge Impulse account. It's free to get started, and you'll gain access to all the tools and resources you need.
Once you're logged in, create your first project. Give it a name that reflects your project's goal, whether it's recognizing sounds, detecting objects, or something entirely unique.
To teach your device, you need data. Edge Impulse provides user-friendly tools for collecting data from your sensors, such as recording audio, capturing images, or reading sensor values. We recommend using a hardware target from this list or your smartphone to start collecting data when you begin with Edge Impulse.
You can also import existing datasets or clone a public project to get familiar with the platform.
Edge Impulse Datasets
Need inspiration? Check out our Edge Impulse datasets collection that contains publicly available datasets collected, generated or curated by Edge Impulse or its partners.
These datasets highlight specific use cases, helping you understand the types of data commonly encountered in projects like object detection, audio classification, and visual anomaly detection.
Organize your data by labeling it. For example, if you're working on sound recognition, label audio clips with descriptions like "dog barking" or "car horn." You can label your data as you collect it or add labels later, our data explorer is also particularly useful to understand your data.
This is where the magic happens. Edge Impulse offers an intuitive model training process through processing blocks and learning blocks. You don't need to write complex code; the platform guides you through feature extraction, model creation, and training.
After training your model, you can easily export your model to run in a web browser or on your smartphone, but you can also run it on a wide variety of edge devices, whether it's a Raspberry Pi, Arduino, or other compatible hardware. We also provide ready-to-flash binaries for all the officially supported hardware targets. You don't even need to write embedded code to test your model on real devices!
If you have a device that is not supported, no problem, you can export your model as a C++ library that runs on any embedded device. See Running your impulse locally for more information.
Building Edge AI solutions is an iterative process. Feel free to try our organization hub to automate your machine-learning pipelines, collaborate with your colleagues, and create custom blocks.
The end-to-end tutorials are perfect for learning how to use Edge Impulse Studio. Try the tutorials:
These will let you build machine-learning models that detect things in your home or office.
Remember, you're not alone on your journey. Join the Edge Impulse community to connect with other beginners, experts, and enthusiasts. Share your experiences, ask questions, and learn from others who are passionate about embedded machine learning.
Now that you have a roadmap, it's time to explore Edge Impulse and discover the exciting possibilities of embedded machine learning. Let's get started!
Welcome to Edge Impulse! Whether you are a machine learning engineer, MLOps engineer, data scientist, or researcher, we have developed professional tools to help you build and optimize models to run efficiently on any edge device.
In this guide, we'll explore how Edge Impulse empowers you to bring your expertise and your own models to the world of edge AI using either the Edge Impulse Studio, our visual interface, and the Edge Impulse Python SDK, available as a pip package.
Flexibility: You can choose to work with the tools they are already familiar with and import your models, architecture, and feature processing algorithms into the platform. This means that you can leverage your existing knowledge and workflows seamlessly. Or, for those who prefer an all-in-one solution, Edge Impulse provides enterprise-grade tools for your entire machine-learning pipeline.
Optimized for edge devices: Edge Impulse is designed specifically for deploying machine learning models on edge devices, which are typically resource-constrained, from low-power MCUs up to powerful edge GPUs. We provide tools to optimize your models for edge deployment, ensuring efficient resource usage and peak performance. Focus on developing the best models, we will provide feedback on whether they can run on your hardware target!
Data pipelines: We developed a strong expertise in complex data pipelines (including clinical data) while working with our customers. We support data coming from multiple sources, in any format, and provide tools to perform data alignment and validation checks. All of this in customizable multi-stage pipelines. This means you can build gold-standard labeled datasets that can then be imported into your project to train your models.
In this getting started guide, we'll walk you through the two different approaches to bringing your expertise to edge devices. Either starting from your dataset or from an existing model.
Start with existing data
We currently accept various file types, including .cbor
, .json
, .csv
, .wav
, .jpg
, .png
, .mp4
, and .avi
.
Organization data
Since the creation of Edge Impulse, we have been helping our customers deal with complex data pipelines, complex data transformation methods and complex clinical validation studies.
The organizational data gives you tools to centralize, validate and transform datasets so they can be easily imported into your projects.
Each block will provide on-device performance information showing you the estimated RAM, flash, and latency.
Start with an existing model
If you already have been working on different models for your Edge AI applications, Edge Impulse offers an easy way to upload your models and profile them. This way, in just a few minutes, you will know if your model can run on real devices and what will be the on-device performances (RAM, flash usage, and latency).
Edge Impulse Python SDK is available as a pip
package:
From there, you can profile your existing models:
If you target MCU-based devices, you can generate ready-to-flash binaries for all the officially supported hardware targets. This method will let you test your model on real hardware very quickly.
In both cases, we will provide profiling information about your models so you can make sure your model will fit your edge device constraints.
Welcome to Edge Impulse! We enable professional developers and researchers to create the next generation of intelligent products with Edge AI. In this documentation, you'll find user guides, tutorials, and API documentation. If at any point you have questions, visit our .
If you are a beginner, an advanced embedded engineer, an ML engineer, or a data scientist, you may want to use Edge Impulse differently. We have tailored Edge Impulse to suit your needs. Check out the following getting-started guides for a smooth start:
If you're new to the idea of embedded machine learning, or machine learning in general, you may enjoy our quick articles: and
For professionals who want additional compute time, more private projects, and more flexibility in usage, we also offer a professional tier version of our platform.
Edge Impulse Datasets
These datasets highlight specific use cases, helping you understand the types of data commonly encountered in projects like object detection, audio classification, and visual anomaly detection.
Think your model is awesome, and want to share it with the world? Go to Dashboard and click Make this project public. This will make your whole project - including all data, machine learning models, and visualizations - available, and can be viewed and cloned by anyone with the URL.
Welcome to Edge Impulse! When we started Edge Impulse, we initially focused on developing a suite of engineering tools designed to empower embedded engineers to harness the power of machine learning on edge devices. As we grew, we also started to develop advanced tools for ML practitioners to ease the collaboration between teams in organizations.
In this getting started guide, we'll walk you through the essential steps to dive into Edge Impulse and leverage it for your embedded projects.
Embedded systems are becoming increasingly intelligent, and Edge Impulse is here to streamline the integration of machine learning into your hardware projects. Here's why embedded engineers are turning to Edge Impulse:
Extend hardware capabilities: Edge Impulse can extend hardware capabilities by enabling the integration of machine learning models, allowing edge devices to process complex tasks, recognize patterns, and make intelligent decisions that are complex to develop using rule-based algorithms.
Open-source export formats: Exported models and libraries contain both digital signal processing code and machine learning models, giving you full explainability of the code.
Powerful integrations: Edge Impulse provides complete and documented integrations with various hardware platforms, allowing you to focus on the application logic rather than the intricacies of machine learning.
Support for diverse sensors: Whether you're working with accelerometers, microphones, cameras, or custom sensors, Edge Impulse accommodates a wide range of data sources for your projects.
Predict on-device performances: Models trained in Edge Impulse run directly on your edge devices, ensuring real-time decision-making with minimal latency. We provide tools to ensure the DSP and models developed with Edge Impulse can fit your device constraints.
Device-aware optimization: You have full control over model optimization, enabling you to tailor your machine-learning models to the specific requirements and constraints of your embedded systems. Our can help you select the best model by training many different variants of models only from an existing dataset and your device constraints!
Ready to embark on your journey with Edge Impulse? Follow these essential steps to get started:
Start by creating your . Registration is straightforward, granting you immediate access to the comprehensive suite of tools and resources.
Now that you have a roadmap, it's time to explore Edge Impulse and discover the exciting possibilities of embedded machine learning. Let's get started!
First, start by creating your .
You can import data using , , or our . These allow you to easily upload and manage your existing data samples and datasets to Edge Impulse Studio.
If you are working with image datasets, the Studio uploader and the CLI uploader currently handle these types of : Edge Impulse object detection, COCO JSON, Open Images CSV, Pascal VOC XML, Plain CSV, and YOLO TXT.
See the documentation.
To visualize how your labeled data items are clustered, use the feature available for most dataset types, where we apply dimensionality reduction techniques (t-SNE or PCA) on your embeddings.
To extract features from your data items, either choose an available (MFE, MFCC, spectral analysis using FFT or Wavelets, etc.) or from your expertise. These can be written in any language.
Similarly, to train your machine learning model, you can choose from different (Classification, Anomaly Detection, Regression, Image or Audio Transfer Learning, Object Detection). In most of these blocks, we expose the Keras API in an . You can also bring your own architecture/training pipeline as a .
You can do this directly from the or using .
And then directly generate a customizable library or any other supported
You can easily in a .eim
format, a Linux executable that contains your signal processing and ML code, compiled with optimizations for your processor or GPU. This executable can then be called with our . We have inferencing libraries and examples for Python, Node.js, C++, and Go.
If you want to get familiar with the full end-to-end flow using Edge Impulse Studio, please have a look at our :
,
,
,
,
,
To understand the full potential of Edge Impulse, see our that describes an end-to-end ML workflow for building a wearable health product using Edge Impulse. It handles data coming from multiple sources, data alignment, and a multi-stage pipeline before the data is imported into an Edge Impulse project.
While the Edge Impulse Studio is a great interface for guiding you through the process of collecting data and training a model, the Python SDK allows you to programmatically Bring Your Own Model (BYOM), developed and trained on any platform:
(access Keras API in the studio)
(access state-of-the-art pre-trained models)
For startups and enterprises looking to scale edge ML algorithm development from prototype to production, we offer an . This includes all of the tools needed to go from data collection to model deployment, such as a robust dataset builder to future-proof your data, integrations with all major cloud vendors, dedicated technical support, custom DSP and ML capabilities, and full access to the Edge Impulse APIs to automate your algorithm development.
Sign up for a FREE today!
Try our today!
We have some great tutorials, but you have full freedom in the models that you design in Edge Impulse. You can plug in new signal processing blocks and completely new neural networks. See and .
Need inspiration? Check out our that contains publicly available datasets collected, generated or curated by Edge Impulse or its partners.
You can access any feature in the Edge Impulse Studio through the . We also have the if you want to send data directly, and we have an open to control devices from the Studio.
Edge Impulse offers a thriving community of engineers, developers, researchers, and machine learning experts. Connect with like-minded professionals, share your knowledge, and collaborate to enhance your embedded machine-learning projects. Head to the to ask questions or share your awesome ideas!
We reference all the public projects here: . If you need some inspiration, just clone a project and fine-tune it to your needs!
Upon logging in, initiate your first project. Select a name that resonates with your project's objectives. If you already which hardware target or system architecture you will be using, you can set it up directly in the section. This will help you to make sure your model fits your device constraints.
We offer various methods to collect data from your sensors or to import datasets (see for all methods). For the , we provide binaries or simple steps to attach your device to Edge Impulse Studio and collect data from the Studio. However, as an embedded engineer, you might want to collect data from sensors that are not necessarily available on these devices. To do so, you can use the and print out your sensor values over serial (up to 8kHz) or use our , a portable header-only library (designed to reliably store sampled data from sensors at a high frequency in very little memory).
Edge Impulse offers an intuitive model training process through and . You don't need to write Python code to train your model; the platform guides you through feature extraction, model creation, and training. Customize and fine-tune your blocks for optimal performance on your hardware. Each block will provide on-device performance information showing you the estimated RAM, flash, and latency.
This is where the fun start, you can easily as ready-to-flash binaries for all the officially supported hardware targets. This method will let you test your model on real hardware very quickly.
In addition, we also provide a wide variety of export methods to easily integrate your model with your application logic. See to run your model on any device that supports C++ or our guides for , , , , , Ethos-U library, Meta TF model, Simplicity Studio Component, Tensai Flow library, TensorRT library, TIDL-RT library, etc...
The C++ inferencing library is a portable library for digital signal processing and machine learning inferencing, and it contains native implementations for both processing and learning blocks in Edge Impulse. It is written in C++11 with all dependencies bundled and can be built on both desktop systems and microcontrollers. See .
Building Edge AI solutions is an iterative process. Feel free to try our to automate your machine-learning pipelines, collaborate with your colleagues, and create custom blocks.
If you want to get familiar with the full end-to-end flow, please have a look at our :
,
,
,
,
,
In the section, you will discover useful techniques to leverage our inferencing libraries or how you can use the inference results in your application logic:
Edge Impulse offers a of embedded engineers, developers, and experts. Connect with like-minded professionals, share your knowledge, and collaborate to enhance your embedded machine-learning projects.
This section provides detailed end-to-end tutorials to help you get started with Edge Impulse:
Detect objects using MobileNet SSD (bounding boxes)
Object detection using FOMO (centroids)
Edge Impulse Datasets
Need inspiration? Check out our Edge Impulse datasets collection that contains publicly available datasets collected, generated or curated by Edge Impulse or its partners.
These datasets highlight specific use cases, helping you understand the types of data commonly encountered in projects like object detection, audio classification, and visual anomaly detection.
Object detection tasks take an image and output information about the class and number of objects, position, (and, eventually, size) in the image.
Edge Impulse provides, by default, two different model architectures to perform object detection, MobileNetV2 SSD FPN-Lite uses bounding boxes (objects location and size) and FOMO uses centroids (objects location only).
Want to compare the two models?
Detect objects using MobileNet SSD (bounding boxes) Can run on systems starting from Linux CPUs up to powerful GPUs
Object detection using FOMO (centroids) Can run on high-end MCUs, Linux CPUs, and GPUs
In this tutorial, you'll use machine learning to build a system that can recognize objects in your house through a camera - a task known as image classification - connected to a microcontroller. Adding sight to your embedded devices can make them see the difference between poachers and elephants, do quality control on factory lines, or let your RC cars drive themselves. In this tutorial you'll learn how to collect images for a well-balanced dataset, how to apply transfer learning to train a neural network, and deploy the system to an embedded device.
At the end of this tutorial, you'll have a firm understanding of how to classify images using Edge Impulse.
There is also a video version of this tutorial:
You can view the finished project, including all data, signal processing and machine learning blocks here: Tutorial: adding sight to your sensors.
For this tutorial, you'll need a supported device.
If you don't have any of these devices, you can also upload an existing dataset through the Uploader. After this tutorial you can then deploy your trained machine learning model as a C++ library and run it on your device.
In this tutorial we'll build a model that can distinguish between two objects in your house - we've used a plant and a lamp, but feel free to pick two other objects. To make your machine learning model see it's important that you capture a lot of example images of these objects. When training the model these example images are used to let the model distinguish between them. Because there are (hopefully) a lot more objects in your house than just lamps or plants, you also need to capture images that are neither a lamp or a plant to make the model work well.
Capture the following amount of data - make sure you capture a wide variety of angles and zoom levels:
50 images of a lamp.
50 images of a plant.
50 images of neither a plant nor a lamp - make sure to capture a wide variation of random objects in the same room as your lamp or plant.
You can collect data from the following devices:
Collecting image data from the Studio - for all other officially supported boards with camera sensors.
Or you can capture your images using another camera, and then upload them by going to Data acquisition and clicking the 'Upload' icon.
Afterwards you should have a well-balanced dataset listed under Data acquisition in your Edge Impulse project. You can switch between your training and testing data with the two buttons above the 'Data collected' widget.
With the training set in place you can design an impulse. An impulse takes the raw data, adjusts the image size, uses a preprocessing block to manipulate the image, and then uses a learning block to classify new data. Preprocessing blocks always return the same values for the same input (e.g. convert a color image into a grayscale one), while learning blocks learn from past experiences.
For this tutorial we'll use the 'Images' preprocessing block. This block takes in the color image, optionally makes the image grayscale, and then turns the data into a features array. If you want to do more interesting preprocessing steps - like finding faces in a photo before feeding the image into the network -, see the Building custom processing blocks tutorial. Then we'll use a 'Transfer Learning' learning block, which takes all the images in and learns to distinguish between the three ('plant', 'lamp', 'unknown') classes.
In the studio go to Create impulse, set the image width and image height to 96
, and add the 'Images' and 'Transfer Learning (Images)' blocks. Then click Save impulse.
To configure your processing block, click Images in the menu on the left. This will show you the raw data on top of the screen (you can select other files via the drop down menu), and the results of the processing step on the right. You can use the options to switch between 'RGB' and 'Grayscale' mode, but for now leave the color depth on 'RGB' and click Save parameters.
This will send you to the 'Feature generation' screen. In here you'll:
Resize all the data.
Apply the processing block on all this data.
Create a 3D visualization of your complete dataset.
Click Generate features to start the process.
Afterwards the 'Feature explorer' will load. This is a plot of all the data in your dataset. Because images have a lot of dimensions (here: 96x96x3=27,648 features) we run a process called 'dimensionality reduction' on the dataset before visualizing this. Here the 27,648 features are compressed down to just 3, and then clustered based on similarity. Even though we have little data you can already see some clusters forming (lamp images are all on the right), and can click on the dots to see which image belongs to which dot.
With all data processed it's time to start training a neural network. Neural networks are a set of algorithms, modeled loosely after the human brain, that are designed to recognize patterns. The network that we're training here will take the image data as an input, and try to map this to one of the three classes.
It's very hard to build a good working computer vision model from scratch, as you need a wide variety of input data to make the model generalize well, and training such models can take days on a GPU. To make this easier and faster we are using transfer learning. This lets you piggyback on a well-trained model, only retraining the upper layers of a neural network, leading to much more reliable models that train in a fraction of the time and work with substantially smaller datasets.
To configure the transfer learning model, click Transfer learning in the menu on the left. Here you can select the base model (the one selected by default will work, but you can change this based on your size requirements), optionally enable data augmentation (images are randomly manipulated to make the model perform better in the real world), and the rate at which the network learns.
Set:
Number of training cycles to 20
.
Learning rate to 0.0005
.
Data augmentation: enabled.
Minimum confidence rating: 0.7.
Important: If you're using a development board with less memory, like the Arduino Nano 33 BLE Sense click Choose a different model and select MobileNetV1 96x96 0.25. This is a smaller transfer learning model.
And click Start training. After the model is done you'll see accuracy numbers, a confusion matrix and some predicted on-device performance on the bottom. You have now trained your model!
With the model trained let's try it out on some test data. When collecting the data we split the data up between a training and a testing dataset. The model was trained only on the training data, and thus we can use the data in the testing dataset to validate how well the model will work in the real world. This will help us ensure the model has not learned to overfit the training data, which is a common occurrence.
To validate your model, go to Model testing, select the checkbox next to 'Sample name' and click Classify selected. Here we hit 89% accuracy, which is great for a model with so little data.
To see a classification in detail, click the three dots next to an item, and select Show classification. This brings you to the Live classification screen with much more details on the file (if you collected data with your mobile phone you can also capture new testing data directly from here). This screen can help you determine why items were misclassified.
With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the preprocessing steps, neural network weights, and classification code - in a single C++ library that you can include in your embedded software.
To run your impulse on either the OpenMV camera or your phone, follow these steps:
OpenMV Cam H7 Plus: Running your impulse on your OpenMV camera
Mobile phone: just click Switch to classification mode at the bottom of your phone screen.
For other boards: click on Deployment in the menu. Then under 'Build firmware' select your development board, and click Build. This will export the impulse, and build a binary that will run on your development board in a single step. After building is completed you'll get prompted to download a binary. Save this on your computer.
When you click the Build button, you'll see a pop-up with text and video instructions on how to deploy the binary to your particular device. Follow these instructions. Once you are done, we are ready to test your impulse out.
We can connect to the board's newly flashed firmware over serial. Open a terminal and run:
To also see a preview of the camera, run:
To run continuous (without a pause every 2 seconds), but without the preview, run:
Congratulations! You've added sight to your sensors. Now that you've trained your model you can integrate your impulse in the firmware of your own embedded device, see Running your impulse locally. There are examples for Mbed OS, Arduino, STM32CubeIDE, and any other target that supports a C++ compiler. Note that the model we trained in this tutorial is relatively big, but you can choose a smaller transfer learning model.
Or if you're interested in more, see our tutorials on Continuous motion recognition or Sound recognition. If you have a great idea for a different project, that's fine too. Edge Impulse lets you capture data from any sensor, build custom processing blocks to extract features, and you have full flexibility in your Machine Learning pipeline with the learning blocks.
We can't wait to see what you'll build! 🚀
In this tutorial, you'll use machine learning to build a system that can recognize audible events, particularly your voice through audio classification. The system you create will work similarly to "Hey Siri" or "OK, Google" and is able to recognize keywords or other audible events, even in the presence of other background noise or background chatter.
You'll learn how to collect audio data from microphones, use signal processing to extract the most important information, and train a deep neural network that can tell you whether your keyword was heard in a given clip of audio. Finally, you'll deploy the system to an embedded device and evaluate how well it works.
At the end of this tutorial, you'll have a firm understanding of how to classify audio using Edge Impulse.
There is also a video version of this tutorial:
You can view the finished project, including all data, signal processing and machine learning blocks here: Tutorial: responding to your voice.
Detect non-voice audio?
We have a tutorial for that too! See Sound recognition.
For this tutorial, you'll need a supported device.
If your device is connected under Devices in the studio you can proceed:
Device compatibility
Edge Impulse can ingest data from any device - including embedded devices that you already have in production. See the documentation for the Ingestion service for more information.
In this tutorial we want to build a system that recognizes keywords, so your first job is to think of a great one. It can be your name, an action, or even a growl - it's your party. Do keep in mind that some keywords are harder to distinguish from others, and especially keywords with only one syllable (like 'One') might lead to false-positives (e.g. when you say 'Gone'). This is the reason that Apple, Google and Amazon all use at least three-syllable keywords ('Hey Siri', 'OK, Google', 'Alexa'). A good one would be "Hello world".
To collect your first data, go to Data acquisition, set your keyword as the label, set your sample length to 10s., your sensor to 'microphone' and your frequency to 16KHz. Then click Start sampling and start saying your keyword over and over again (with some pause in between).
Note: Data collection from a development board might be slow, you can use your Mobile phone as a sensor to make this much faster.
Afterwards you have a file like this, clearly showing your keywords, separated by some noise.
This data is not suitable for Machine Learning yet though. You will need to cut out the parts where you say your keyword. This is important because you only want the actual keyword to be labeled as such, and not accidentally label noise, or incomplete sentences (e.g. only "Hello"). Fortunately the Edge Impulse Studio can do this for you. Click ⋮
next to your sample, and select Split sample.
If you have a short keyword, enable Shift samples to randomly shift the sample around in the window, and then click Split. You now have individual 1s. long samples in your dataset. Perfect!
Now that you know how to collect data we can consider other data we need to collect. In addition to your keyword we'll also need audio that is not your keyword. Like background noise, the TV playing ('noise' class), and humans saying other words ('unknown' class). This is required because a machine learning model has no idea about right and wrong (unless those are your keywords), but only learns from the data you feed into it. The more varied your data is, the better your model will work.
For each of these three classes ('your keyword', 'noise', and 'unknown') you want to capture an even amount of data (balanced datasets work better) - and for a decent keyword spotting model you'll want at least 10 minutes in each class (but, the more the better).
Thus, collect 10 minutes of samples for your keyword - do this in the same manner as above. The fastest way is probably through your mobile phone, collecting 1 minute clips, then automatically splitting this data. Make sure to capture wide variations of the keyword: leverage your family and your colleagues to help you collect the data, make sure you cover high and low pitches, and slow and fast speakers.
For the noise and unknown datasets you can either collect this yourself, or make your life a bit easier by using dataset of both 'noise' (all kinds of background noise) and 'unknown' (random words) data that we built for you here: Keyword Spotting Dataset.
To import this data, go to Data acquisition, click the Upload icon, and select a number of 'noise' or 'unknown' samples (there's 25 minutes of each class, but you can select less files if you want), and clicking Begin upload. The data is automatically labeled and added to your project.
If you've collected all your training data through the 'Record new data' widget you'll have all your keywords in the 'Training' dataset. This is not great, because you want to keep 20% of your data separate to validate the machine learning model. To mitigate this you can go to Dashboard and select Perform train/test split. This will automatically split your data between a training class (80%) and a testing class (20%). Afterwards you should see something like this:
With the data set in place you can design an impulse. An impulse takes the raw data, slices it up in smaller windows, uses signal processing blocks to extract features, and then uses a learning block to classify new data. Signal processing blocks always return the same values for the same input and are used to make raw data easier to process, while learning blocks learn from past experiences.
For this tutorial we'll use the "MFCC" signal processing block. MFCC stands for Mel Frequency Cepstral Coefficients. This sounds scary, but it's basically just a way of turning raw audio—which contains a large amount of redundant information—into simplified form. Edge Impulse has many other processing blocks for audio, including "MFE" and the "Spectrogram" blocks for non-voice audio, but the "MFCC" block is great for dealing with human speech.
We'll then pass this simplified audio data into a Neural Network block, which will learn to distinguish between the three classes of audio.
In the Studio, go to the Create impulse tab, add a Time series data, an Audio (MFCC) and a Classification (Keras) block. Leave the window size to 1 second (as that's the length of our audio samples in the dataset) and click Save Impulse.
Now that we've assembled the building blocks of our Impulse, we can configure each individual part. Click on the MFCC tab in the left hand navigation menu. You'll see a page that looks like this:
This page allows you to configure the MFCC block, and lets you preview how the data will be transformed. The right of the page shows a visualization of the MFCC's output for a piece of audio, which is known as a spectrogram. An MFCC spectrogram is a specially tuned spectrogram which highlights frequencies which are common in human speech (Edge Impulse also has normal spectrograms if that's more your thing).
In the spectrogram the vertical axis represents the frequencies (the number of frequency bands is controlled by 'Number of coefficients' parameter, try it out!), and the horizontal axis represents time (controlled by 'frame stride' and 'frame length'). The patterns visible in a spectrogram contain information about what type of sound it represents. For example, the spectrogram in this image shows "Hello world":
And the spectrogram in this image shows "On":
These differences are not necessarily easy for a person to describe, but fortunately they are enough for a neural network to learn to identify.
It's interesting to explore your data and look at the types of spectrograms it results in. You can use the dropdown box near the top right of the page to choose between different audio samples to visualize, or play with the parameters to see how the spectrogram changes.
In addition, you can see the performance of the MFCC block on your microcontroller below the spectrogram. This is the complete time that it takes on a low-power microcontroller (Cortex-M4F @ 80MHz) to analyze 1 second of data.
You might think based on this number that we can only classify 2 or 3 windows per second, but we continuously build up the spectrogram (as it has a time component), which takes less time, and we can thus continuously listen for events 5-6x a second, even on an 40MHz processor. This is already implemented on all fully supported development boards, and easy to implement on your own device.
The spectrograms generated by the MFCC block will be passed into a neural network architecture that is particularly good at learning to recognize patterns in this type of tabular data. Before training our neural network, we'll need to generate MFCC blocks for all of our windows of audio. To do this, click the Generate features button at the top of the page, then click the green Generate features button. This will take a minute or so to complete.
Afterwards you're presented with one of the most useful features in Edge Impulse: the feature explorer. This is a 3D representation showing your complete dataset, with each data-item color-coded to its respective label. You can zoom in to every item, find anomalies (an item that's in a wrong cluster), and click on items to listen to the sample. This is a great way to check whether your dataset contains wrong items, and to validate whether your dataset is suitable for ML (it should separate nicely).
With all data processed it's time to start training a neural network. Neural networks are algorithms, modeled loosely after the human brain, that can learn to recognize patterns that appear in their training data. The network that we're training here will take the MFCC as an input, and try to map this to one of three classes—your keyword, noise or unknown.
Click on NN Classifier in the left hand menu. You'll see the following page:
A neural network is composed of layers of virtual "neurons", which you can see represented on the left hand side of the NN Classifier page. An input—in our case, an MFCC spectrogram—is fed into the first layer of neurons, which filters and transforms it based on each neuron's unique internal state. The first layer's output is then fed into the second layer, and so on, gradually transforming the original input into something radically different. In this case, the spectrogram input is transformed over four intermediate layers into just two numbers: the probability that the input represents your keyword, and the probability that the input represents 'noise' or 'unknown'.
During training, the internal state of the neurons is gradually tweaked and refined so that the network transforms its input in just the right ways to produce the correct output. This is done by feeding in a sample of training data, checking how far the network's output is from the correct answer, and adjusting the neurons' internal state to make it more likely that a correct answer is produced next time. When done thousands of times, this results in a trained network.
A particular arrangement of layers is referred to as an architecture, and different architectures are useful for different tasks. The default neural network architecture provided by Edge Impulse will work well for our current project, but you can also define your own architectures. You can even import custom neural network code from tools used by data scientists, such as TensorFlow and Keras (click the three dots at the top of the page).
Before you begin training, you should change some values in the configuration. Change the Minimum confidence rating to 0.6. This means that when the neural network makes a prediction (for example, that there is 0.8 probability that some audio contains "hello world") Edge Impulse will disregard it unless it is above the threshold of 0.6.
Next, enable 'Data augmentation'. When enabled your data is randomly mutated during training. For example, by adding noise, masking time or frequency bands, or warping your time axis. This is a very quick way to make your dataset work better in real life (with unpredictable sounds coming in), and prevents your neural network from overfitting (as the data samples are changed every training cycle).
With everything in place, click Start training. You'll see a lot of text flying past in the Training output panel, which you can ignore for now. Training will take a few minutes. When it's complete, you'll see the Last training performance panel appear at the bottom of the page:
Congratulations, you've trained a neural network with Edge Impulse! But what do all these numbers mean?
At the start of training, 20% of the training data is set aside for validation. This means that instead of being used to train the model, it is used to evaluate how the model is performing. The Last training performance panel displays the results of this validation, providing some vital information about your model and how well it is working. Bear in mind that your exact numbers may differ from the ones in this tutorial.
On the left hand side of the panel, Accuracy refers to the percentage of windows of audio that were correctly classified. The higher number the better, although an accuracy approaching 100% is unlikely, and is often a sign that your model has overfit the training data. You will find out whether this is true in the next stage, during model testing. For many applications, an accuracy above 85% can be considered very good.
The Confusion matrix is a table showing the balance of correctly versus incorrectly classified windows. To understand it, compare the values in each row. For example, in the above screenshot, 96 of the helloworld audio windows were classified as helloworld, while 10 of them were incorrectly classified as unknown or noise. This appears to be a great result.
The On-device performance region shows statistics about how the model is likely to run on-device. Inferencing time is an estimate of how long the model will take to analyze one second of data on a typical microcontroller (an Arm Cortex-M4F running at 80MHz). Peak RAM usage gives an idea of how much RAM will be required to run the model on-device.
The performance numbers in the previous step show that our model is working well on its training data, but it's extremely important that we test the model on new, unseen data before deploying it in the real world. This will help us ensure the model has not learned to overfit the training data, which is a common occurrence.
Fortunately we've put aside 20% of our data already in the 'Test set' (see Data acquisition). This is data that the model has never seen before, and we can use this to validate whether our model actually works on unseen data. To run your model against the test set, head to Model testing, select all items and click Classify selected.
To drill down into a misclassified sample, click the three dots (⋮
) next to a sample and select Show classification. You're then transported to the classification view, which lets you inspect the sample, and compare the sample to your training data. This way you can inspect whether this was actually a classification failure, or whether your data was incorrectly labeled. From here you can either update the label (when the label was wrong), or move the item to the training set to refine your model.
Misclassifications and uncertain results
It's inevitable that even a well-trained machine learning model will sometimes misclassify its inputs. When you integrate a model into your application, you should take into account that it will not always give you the correct answer.
For example, if you are classifying audio, you might want to classify several windows of data and average the results. This will give you better overall accuracy than assuming that every individual result is correct.
With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the MFCC algorithm, neural network weights, and classification code - in a single C++ library that you can include in your embedded software.
Mobile phone
Your mobile phone can build and download the compiled impulse directly from the mobile client. See 'Deploying back to device' on the Using your mobile phone page.
To export your model, click on Deployment in the menu. Then under 'Build firmware' select your development board, and click Build. This will export the impulse, and build a binary that will run on your development board in a single step. After building is completed you'll get prompted to download a binary. Save this on your computer.
When you click the Build button, you'll see a pop-up with text and video instructions on how to deploy the binary to your particular device. Follow these instructions. Once you are done, we are ready to test your impulse out.
We can connect to the board's newly flashed firmware over serial. Open a terminal and run:
Serial daemon
If the device is not connected over WiFi, but instead connected via the Edge Impulse serial daemon, you'll need stop the daemon. Only one application can connect to the development board at a time.
This will capture audio from the microphone, run the MFCC code, and then classify the spectrogram:
Great work! You've captured data, trained a model, and deployed it to an embedded device. You can now control LEDs, activate actuators, or send a message to the cloud whenever you say a keyword!
Is your model working properly in the Studio, but does not recognize your keyword when running in continuous mode on your device? Then this is probably due to dataset imbalance (a lot more unknown / noise data compared to your keyword) in combination with our moving average code to reduce false positives.
When running in continuous mode we run a moving average over the predictions to prevent false positives. E.g. if we do 3 classifications per second you’ll see your keyword potentially classified three times (once at the start of the audio file, once in the middle, once at the end). However, if your dataset is unbalanced (there’s a lot more noise / unknown than in your dataset) the ML model typically manages to only find your keyword in the 'center' window, and thus we filter it out as a false positive.
You can fix this by either:
Adding more data :-)
Or, by disabling the moving average filter by going into ei_run_classifier.h (in the edge-impulse-sdk directory) and removing:
Note that this might increase the number of false positives the model detects.
Congratulations! you've used Edge Impulse to train a neural network model capable of recognizing audible events. There are endless applications for this type of model, from monitoring industrial machinery to recognizing voice commands. Now that you've trained your model you can integrate your impulse in the firmware of your own embedded device, see Running your impulse locally. There are examples for Mbed OS, Arduino, STM32CubeIDE, Zephyr, and any other target that supports a C++ compiler.
Or if you're interested in more, see our tutorials on Continuous motion recognition or Image classification. If you have a great idea for a different project, that's fine too. Edge Impulse lets you capture data from any sensor, build custom processing blocks to extract features, and you have full flexibility in your Machine Learning pipeline with the learning blocks.
We can't wait to see what you'll build! 🚀
In this tutorial, you'll use machine learning to build a system that can recognize when a particular sound is happening—a task known as audio classification. The system you create will be able to recognize the sound of water running from a faucet, even in the presence of other background noise.
You'll learn how to collect audio data from microphones, use signal processing to extract the most important information, and train a deep neural network that can tell you whether the sound of running water can be heard in a given clip of audio. Finally, you'll deploy the system to an embedded device and evaluate how well it works.
At the end of this tutorial, you'll have a firm understanding of how to classify audio using Edge Impulse.
There is also a video version of this tutorial:
You can view the finished project, including all data, signal processing and machine learning blocks here: Tutorial: recognize sounds from audio.
Detecting human speech?
Do you want a device that listens to your voice? We have a specific tutorial for that! See Keyword spotting.
For this tutorial, you'll need a supported device.
If you don't see your supported development board listed here, be sure to check the Hardware specific tutorials page for the appropriate tutorial.
If your device is connected under Devices in the studio you can proceed:
Device compatibility
Edge Impulse can ingest data from any device - including embedded devices that you already have in production. See the documentation for the Ingestion service for more information.
To build this project, you'll need to collect some audio data that will be used to train the machine learning model. Since the goal is to detect the sound of a running faucet, you'll need to collect some examples of that. You'll also need some examples of typical background noise that doesn't contain the sound of a faucet, so the model can learn to discriminate between the two. These two types of examples represent the two classes we'll be training our model to detect: background noise, or running faucet.
You can use your device to collect some data. In the studio, go to the Data acquisition tab. This is the place where all your raw data is stored, and - if your device is connected to the remote management API - where you can start sampling new data.
Let's start by recording an example of background noise that doesn't contain the sound of a running faucet. Under Record new data, select your device, set the label to noise
, the sample length to 1000
, and the sensor to Built-in microphone
. This indicates that you want to record 1 second of audio, and label the recorded data as noise
. You can later edit these labels if needed.
After you click Start sampling, the device will capture a second of audio and transmit it to Edge Impulse. The LED will light while recording is in progress, then light again during transmission.
When the data has been uploaded, you will see a new line appear under 'Collected data'. You will also see the waveform of the audio in the 'RAW DATA' box. You can use the controls underneath to listen to the audio that was captured.
Since you now know how to capture audio with Edge Impulse, it's time to start building a dataset. For a simple audio classification model like this one, we should aim to capture around 10 minutes of data. We have two classes, and it's ideal if our data is balanced equally between each of them. This means we should aim to capture the following data:
5 minutes of background noise, with the label "noise"
5 minutes of running faucet noise, with the label "faucet"
Real world data
In the real world, there are usually additional sounds present alongside the sounds we care about. For example, a running faucet is often accompanied by the sound of dishes being washed, teeth being brushed, or a conversation in the kitchen. Background noise might also include the sounds of television, kids playing, or cars driving past outside.
It's important that your training data contains these types of real world sounds. If your model is not exposed to them during training, it will not learn to take them into account, and it will not perform well during real-world usage.
For this tutorial, you should try to capture the following:
Background noise
2 minutes of background noise without much additional activity
1 minute of background noise with a TV or music playing
1 minute of background noise featuring occasional talking or conversation
1 minutes of background noise with the sounds of housework
Running faucet noise
1 minute of a faucet running
1 minute of a different faucet running
1 minute of a faucet running with a TV or music playing
1 minute of a faucet running with occasional talking or conversation
1 minute of a faucet running with the sounds of housework
It's okay if you can't get all of these, as long as you still obtain 5 minutes of data for each class. However, your model will perform better in the real world if it was trained on a representative dataset.
Dataset diversity
There's no guarantee your model will perform well in the presence of sounds that were not included in its training set, so it's important to make your dataset as diverse and representative of real-world conditions as possible.
Data capture and transmission
The amount of audio that can be captured in one go varies depending on a device's memory. The ST B-L475E-IOT01A developer board has enough memory to capture 60 seconds of audio at a time, and the Arduino Nano 33 BLE Sense has enough memory for 16 seconds. To capture 60 seconds of audio, set the sample length to 60000
. Because the board transmits data quite slowly, it will take around 7 minutes before a 60 second sample appears in Edge Impulse.
Once you've captured around 10 minutes of data, it's time to start designing an Impulse.
Prebuilt dataset
Alternatively, you can load an example test set that has about ten minutes of data in these classes (but how much fun is that?). See the Running faucet dataset for more information.
With the training set in place you can design an impulse. An impulse takes the raw data, slices it up in smaller windows, uses signal processing blocks to extract features, and then uses a learning block to classify new data. Signal processing blocks always return the same values for the same input and are used to make raw data easier to process, while learning blocks learn from past experiences.
For this tutorial we'll use the "MFE" signal processing block. MFE stands for Mel Frequency Energy. This sounds scary, but it's basically just a way of turning raw audio—which contains a large amount of redundant information—into simplified form.
Spectrogram block
Edge Impulse supports three different blocks for audio classification: MFCC, MFE and spectrogram blocks. If your accuracy is not great using the MFE block you can switch to the spectrogram block, which is not tuned to frequencies for the human ear.
We'll then pass this simplified audio data into a Neural Network block, which will learn to distinguish between the two classes of audio (faucet and noise).
In the studio, go to the Create impulse tab. You'll see a Raw data block, like this one.
As mentioned above, Edge Impulse slices up the raw samples into windows that are fed into the machine learning model during training. The Window size field controls how long, in milliseconds, each window of data should be. A one second audio sample will be enough to determine whether a faucet is running or not, so you should make sure Window size is set to 1000 ms. You can either drag the slider or type a new value directly.
Each raw sample is sliced into multiple windows, and the Window increase field controls the offset of each subsequent window from the first. For example, a Window increase value of 1000 ms would result in each window starting 1 second after the start of the previous one.
By setting a Window increase that is smaller than the Window size, we can create windows that overlap. This is actually a great idea. Although they may contain similar data, each overlapping window is still a unique example of audio that represents the sample's label. By using overlapping windows, we can make the most of our training data. For example, with a Window size of 1000 ms and a Window increase of 100 ms, we can extract 10 unique windows from only 2 seconds of data.
Make sure the Window increase field is set to 300 ms. The Raw data block should match the screenshot above.
Next, click Add a processing block and choose the 'MFE' block. Once you're done with that, click Add a learning block and select 'Classification (Keras)'. Finally, click Save impulse. Your impulse should now look like this:
Now that we've assembled the building blocks of our Impulse, we can configure each individual part. Click on the MFE tab in the left hand navigation menu. You'll see a page that looks like this:
This page allows you to configure the MFE block, and lets you preview how the data will be transformed. The right of the page shows a visualization of the MFE's output for a piece of audio, which is known as a spectrogram.
The MFE block transforms a window of audio into a table of data where each row represents a range of frequencies and each column represents a span of time. The value contained within each cell reflects the amplitude of its associated range of frequencies during that span of time. The spectrogram shows each cell as a colored block, the intensity which varies depends on the amplitude.
The patterns visible in a spectrogram contain information about what type of sound it represents. For example, the spectrogram in this image shows a pattern typical of background noise:
You can tell that it is slightly different from the following spectrogram, which shows a pattern typical of a running faucet:
These differences are not necessarily easy for a person to describe, but fortunately they are enough for a neural network to learn to identify.
It's interesting to explore your data and look at the types of spectrograms it results in. You can use the dropdown box near the top right of the page to choose between different audio samples to visualize, and drag the white window on the audio waveform to select different windows of data:
There are a lot of different ways to configure the MFCC block, as shown in the Parameters box:
Handily, Edge Impulse provides sensible defaults that will work well for many use cases, so we can leave these values unchanged. You can play around with the noise floor to quickly see the effect it has on the spectrogram.
The spectrograms generated by the MFE block will be passed into a neural network architecture that is particularly good at learning to recognize patterns in this type of tabular data. Before training our neural network, we'll need to generate MFE blocks for all of our windows of audio. To do this, click the Generate features button at the top of the page, then click the green Generate features button. If you have a full 10 minutes of data, the process will take a while to complete:
Once this process is complete the feature explorer shows a visualization of your dataset. Here dimensionality reduction is used to map your features onto a 3D space, and you can use the feature explorer to see if the different classes separate well, or find mislabeled data (if it shows in a different cluster). You can find more information in visualizing complex datasets.
Next, we'll configure the neural network and begin training.
With all data processed it's time to start training a neural network. Neural networks are algorithms, modeled loosely after the human brain, that can learn to recognize patterns that appear in their training data. The network that we're training here will take the MFE as an input, and try to map this to one of two classes—noise, or faucet.
Click on NN Classifier in the left hand menu. You'll see the following page:
A neural network is composed of layers of virtual "neurons", which you can see represented on the left hand side of the NN Classifier page. An input—in our case, an MFE spectrogram—is fed into the first layer of neurons, which filters and transforms it based on each neuron's unique internal state. The first layer's output is then fed into the second layer, and so on, gradually transforming the original input into something radically different. In this case, the spectrogram input is transformed over four intermediate layers into just two numbers: the probability that the input represents noise, and the probability that the input represents a running faucet.
During training, the internal state of the neurons is gradually tweaked and refined so that the network transforms its input in just the right ways to produce the correct output. This is done by feeding in a sample of training data, checking how far the network's output is from the correct answer, and adjusting the neurons' internal state to make it more likely that a correct answer is produced next time. When done thousands of times, this results in a trained network.
A particular arrangement of layers is referred to as an architecture, and different architectures are useful for different tasks. The default neural network architecture provided by Edge Impulse will work well for our current project, but you can also define your own architectures. You can even import custom neural network code from tools used by data scientists, such as TensorFlow and Keras.
The default settings should work, and to begin training, click Start training. You'll see a lot of text flying past in the Training output panel, which you can ignore for now. Training will take a few minutes. When it's complete, you'll see the Model panel appear at the right side of the page:
Congratulations, you've trained a neural network with Edge Impulse! But what do all these numbers mean?
At the start of training, 20% of the training data is set aside for validation. This means that instead of being used to train the model, it is used to evaluate how the model is performing. The Last training performance panel displays the results of this validation, providing some vital information about your model and how well it is working. Bear in mind that your exact numbers may differ from the ones in this tutorial.
On the left hand side of the panel, Accuracy refers to the percentage of windows of audio that were correctly classified. The higher number the better, although an accuracy approaching 100% is unlikely, and is often a sign that your model has overfit the training data. You will find out whether this is true in the next stage, during model testing. For many applications, an accuracy above 80% can be considered very good.
The Confusion matrix is a table showing the balance of correctly versus incorrectly classified windows. To understand it, compare the values in each row. For example, in the above screenshot, all of the faucet audio windows were classified as faucet, but a few noise windows were misclassified. This appears to be a great result though.
The On-device performance region shows statistics about how the model is likely to run on-device. Inferencing time is an estimate of how long the model will take to analyze one second of data on a typical microcontroller (here: an Arm Cortex-M4F running at 80MHz). Peak memory usage gives an idea of how much RAM will be required to run the model on-device.
The performance numbers in the previous step show that our model is working well on its training data, but it's extremely important that we test the model on new, unseen data before deploying it in the real world. This will help us ensure the model has not learned to overfit the training data, which is a common occurrence.
Edge Impulse provides some helpful tools for testing our model, including a way to capture live data from your device and immediately attempt to classify it. To try it out, click on Live classification in the left hand menu. Your device should show up in the 'Classify new data' panel. Capture 5 seconds of background noise by clicking Start sampling:
The sample will be captured, uploaded, and classified. Once this has happened, you'll see a breakdown of the results:
Once the sample is uploaded, it is split into windows–in this case, a total of 41. These windows are then classified. As you can see, our model classified all 41 windows of the captured audio as noise. This is a great result! Our model has correctly identified that the audio was background noise, even though this is new data that was not part of its training set.
Of course, it's possible some of the windows may be classified incorrectly. Since our model was 99% accurate based on its validation data, you can expect that at least 1% of windows will be classified wrongly—and likely much more than this, since our validation data doesn't represent every possible type of background or faucet noise. If your model didn't perform perfectly, don't worry. We'll get to troubleshooting later.
Misclassifications and uncertain results
It's inevitable that even a well-trained machine learning model will sometimes misclassify its inputs. When you integrate a model into your application, you should take into account that it will not always give you the correct answer.
For example, if you are classifying audio, you might want to classify several windows of data and average the results. This will give you better overall accuracy than assuming that every individual result is correct.
Using the Live classification tab, you can easily try out your model and get an idea of how it performs. But to be really sure that it is working well, we need to do some more rigorous testing. That's where the Model testing tab comes in. If you open it up, you'll see the sample we just captured listed in the Test data panel:
In addition to its training data, every Edge Impulse project also has a test dataset. Samples captured in Live classification are automatically saved to the test dataset, and the Model testing tab lists all of the test data.
To use the sample we've just captured for testing, we should correctly set its expected outcome. Click the ⋮
icon and select Edit expected outcome, then enter noise
. Now, select the sample using the checkbox to the left of the table and click Classify selected:
You'll see that the model's accuracy has been rated based on the test data. Right now, this doesn't give us much more information that just classifying the same sample in the Live classification tab. But if you build up a big, comprehensive set of test samples, you can use the Model testing tab to measure how your model is performing on real data.
Ideally, you'll want to collect a test set that contains a minimum of 25% the amount of data of your training set. So, if you've collected 10 minutes of training data, you should collect at least 2.5 minutes of test data. You should make sure this test data represents a wide range of possible conditions, so that it evaluates how the model performs with many different types of inputs. For example, collecting test audio for several different faucets is a good idea.
You can use the Data acquisition tab to manage your test data. Open the tab, and then click Test data at the top. Then, use the Record new data panel to capture a few minutes of test data, including audio for both background noise and faucet. Make sure the samples are labelled correctly. Once you're done, head back to the Model testing tab, select all the samples, and click Classify selected:
The screenshot shows classification results from a large number of test samples (there are more on the page than would fit in the screenshot). The panel shows that our model is performing at 85% accuracy, which is 5% less than how it performed on validation data. It's normal for a model to perform less well on entirely fresh data, so this is a successful result. Our model is working well!
For each test sample, the panel shows a breakdown of its individual performance. For example, one of the samples was classified with only 62% accuracy. Samples that contain a lot of misclassifications are valuable, since they have examples of types of audio that our model does not currently fit. It's often worth adding these to your training data, which you can do by clicking the ⋮
icon and selecting Move to training set. If you do this, you should add some new test data to make up for the loss!
Testing your model helps confirm that it works in real life, and it's something you should do after every change. However, if you often make tweaks to your model to try to improve its performance on the test dataset, your model may gradually start to overfit to the test dataset, and it will lose its value as a metric. To avoid this, continually add fresh data to your test dataset.
Data hygiene
It's extremely important that data is never duplicated between your training and test datasets. Your model will naturally perform well on the data that it was trained on, so if there are duplicate samples then your test results will indicate better performance than your model will achieve in the real world.
If the network performed great, fantastic! But what if it performed poorly? There could be a variety of reasons, but the most common ones are:
The data does not look like other data the network has seen before. This is common when someone uses the device in a way that you didn't add to the test set. You can add the current file to the test set by adding the correct label in the 'Expected outcome' field, clicking ⋮
, then selecting Move to training set.
The model has not been trained enough. Increase number of epochs to 200
and see if performance increases (the classified file is stored, and you can load it through 'Classify existing validation sample').
The model is overfitting and thus performs poorly on new data. Try reducing the number of epochs, reducing the learning rate, or adding more data.
The neural network architecture is not a great fit for your data. Play with the number of layers and neurons and see if performance improves.
As you see, there is still a lot of trial and error when building neural networks. Edge Impulse is continually adding features that will make it easier to train an effective model.
With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the MFE algorithm, neural network weights, and classification code - in a single C++ library that you can include in your embedded software.
Mobile phone
Your mobile phone can build and download the compiled impulse directly from the mobile client. See 'Deploying back to device' on the Using your mobile phone page.
To export your model, click on Deployment in the menu. Then under 'Build firmware' select your development board, and click Build. This will export the impulse, and build a binary that will run on your development board in a single step. After building is completed you'll get prompted to download a binary. Save this on your computer.
When you click the Build button, you'll see a pop-up with text and video instructions on how to deploy the binary to your particular device. Follow these instructions. Once you are done, we are ready to test your impulse out.
We can connect to the board's newly flashed firmware over serial. Open a terminal and run:
Serial daemon
If the device is not connected over WiFi, but instead connected via the Edge Impulse serial daemon, you'll need stop the daemon. Only one application can connect to the development board at a time.
This will capture audio from the microphone, run the MFE code, and then classify the spectrogram:
Great work! You've captured data, trained a model, and deployed it to an embedded device. It's time to celebrate—by pouring yourself a nice glass of water, and checking whether the sound is correctly classified by you model.
Congratulations! you've used Edge Impulse to train a neural network model capable of recognizing a particular sound. There are endless applications for this type of model, from monitoring industrial machinery to recognizing voice commands. Now that you've trained your model you can integrate your impulse in the firmware of your own embedded device, see Running your impulse locally. There are examples for Mbed OS, Arduino, STM32CubeIDE, and any other target that supports a C++ compiler.
Or if you're interested in more, see our tutorials on Continuous motion recognition or Image classification. If you have a great idea for a different project, that's fine too. Edge Impulse lets you capture data from any sensor, build custom processing blocks to extract features, and you have full flexibility in your Machine Learning pipeline with the learning blocks.
We can't wait to see what you'll build! 🚀
In this tutorial, you'll use machine learning to build a system that can recognize and track multiple objects in your house through a camera - a task known as object detection. Adding sight to your embedded devices can make them see the difference between poachers and elephants, count objects, find your lego bricks, and detect dangerous situations. In this tutorial, you'll learn how to collect images for a well-balanced dataset, how to apply transfer learning to train a neural network and deploy the system to an edge device.
At the end of this tutorial, you'll have a firm understanding of how to do object detection using Edge Impulse.
There is also a video version of this tutorial:
You can view the finished project, including all data, signal processing and machine learning blocks here: Tutorial: object detection.
Running on a microcontroller?
We recently released a brand-new approach to perform object detection tasks on microcontrollers, FOMO, if you are using a constraint device that does not have as much compute, RAM, and flash as Linux platforms, please head to this end-to-end tutorial: Detect objects using FOMO
Alternatively, if you only need to recognize a single object, you can follow our tutorial on Image classification - which performs image classification, hence, limits you to a single object but can also fit on microcontrollers.
You can view the finished project, including all data, signal processing and machine learning blocks here: Tutorial: object detection.
For this tutorial, you'll need a supported device.
If you don't have any of these devices, you can also upload an existing dataset through the Uploader - including annotations. After this tutorial you can then deploy your trained machine learning model as a C++ library and run it on your device.
In this tutorial we'll build a model that can distinguish between two objects on your desk - we've used a lamp and a coffee cup, but feel free to pick two other objects. To make your machine learning model see it's important that you capture a lot of example images of these objects. When training the model these example images are used to let the model distinguish between them.
Capturing data
Capture the following amount of data - make sure you capture a wide variety of angles and zoom level. It's fine if both images are in the same frame. We'll be cropping the images later to be square so make sure the objects are in the frame.
30 images of a lamp.
30 images of a coffee cup.
You can collect data from the following devices:
Collecting image data from the Studio - for the Raspberry Pi 4 and the Jetson Nano.
Or you can capture your images using another camera, and then upload them by going to Data acquisition and clicking the 'Upload' icon.
With the data collected we need to label this data. Go to Data acquisition, verify that you see your data, then click on the 'Labeling queue' to start labeling.
No labeling queue? Go to Dashboard, and under 'Project info > Labeling method' select 'Bounding boxes (object detection)'.
Labeling data
The labeling queue shows you all the unlabeled data in your dataset. Labeling your objects is as easy as dragging a box around the object, and entering a label. To make your life a bit easier we try to automate this process by running an object tracking algorithm in the background. If you have the same object in multiple photos we thus can move the boxes for you and you just need to confirm the new box. After dragging the boxes, click Save labels and repeat this until your whole dataset is labeled.
AI-Assisted Labeling
Use AI-Assisted Labeling for your object detection project! For more information, check out our blog post.
Afterwards you should have a well-balanced dataset listed under Data acquisition in your Edge Impulse project.
Rebalancing your dataset
To validate whether a model works well you want to keep some data (typically 20%) aside, and don't use it to build your model, but only to validate the model. This is called the 'test set'. You can switch between your training and test sets with the two buttons above the 'Data collected' widget. If you've collected data on your development board there might be no data in the testing set yet. You can fix this by going to Dashboard > Perform train/test split.
With the training set in place you can design an impulse. An impulse takes the raw data, adjusts the image size, uses a preprocessing block to manipulate the image, and then uses a learning block to classify new data. Preprocessing blocks always return the same values for the same input (e.g. convert a color image into a grayscale one), while learning blocks learn from past experiences.
For this tutorial we'll use the 'Images' preprocessing block. This block takes in the color image, optionally makes the image grayscale, and then turns the data into a features array. If you want to do more interesting preprocessing steps - like finding faces in a photo before feeding the image into the network -, see the Building custom processing blocks tutorial. Then we'll use a 'Transfer Learning' learning block, which takes all the images in and learns to distinguish between the two ('coffee', 'lamp') classes.
In the studio go to Create impulse, set the image width and image height to 320
, the 'resize mode' to Fit shortest axis
and add the 'Images' and 'Object Detection (Images)' blocks. Then click Save impulse.
Configuring the processing block
To configure your processing block, click Images in the menu on the left. This will show you the raw data on top of the screen (you can select other files via the drop down menu), and the results of the processing step on the right. You can use the options to switch between 'RGB' and 'Grayscale' mode, but for now leave the color depth on 'RGB' and click Save parameters.
This will send you to the 'Feature generation' screen. In here you'll:
Resize all the data.
Apply the processing block on all this data.
Create a 3D visualization of your complete dataset.
Click Generate features to start the process.
Afterwards the 'Feature explorer' will load. This is a plot of all the data in your dataset. Because images have a lot of dimensions (here: 320x320x3=307,200 features) we run a process called 'dimensionality reduction' on the dataset before visualizing this. Here the 307,200 features are compressed down to just 3, and then clustered based on similarity. Even though we have little data you can already see the clusters forming (lamp images are all on the left, coffee all on the right), and can click on the dots to see which image belongs to which dot.
Configuring the transfer learning model
With all data processed it's time to start training a neural network. Neural networks are a set of algorithms, modeled loosely after the human brain, that are designed to recognize patterns. The network that we're training here will take the image data as an input, and try to map this to one of the three classes.
It's very hard to build a good working computer vision model from scratch, as you need a wide variety of input data to make the model generalize well, and training such models can take days on a GPU. To make this easier and faster we are using transfer learning. This lets you piggyback on a well-trained model, only retraining the upper layers of a neural network, leading to much more reliable models that train in a fraction of the time and work with substantially smaller datasets.
To configure the transfer learning model, click Object detection in the menu on the left. Here you can select the base model (the one selected by default will work, but you can change this based on your size requirements), and set the rate at which the network learns.
Leave all settings as-is, and click Start training. After the model is done you'll see accuracy numbers below the training output. You have now trained your model!
With the model trained let's try it out on some test data. When collecting the data we split the data up between a training and a testing dataset. The model was trained only on the training data, and thus we can use the data in the testing dataset to validate how well the model will work in the real world. This will help us ensure the model has not learned to overfit the training data, which is a common occurrence.
To validate your model, go to Model testing and select Classify all. Here we hit 92.31% precision, which is great for a model with so little data.
To see a classification in detail, click the three dots next to an item, and select Show classification. This brings you to the Live classification screen with much more details on the file (you can also capture new data directly from your development board from here). This screen can help you determine why items were misclassified.
Live Classification Result
This view is particularly useful for a direct comparison between the raw image and the model's interpretation. Each object detected in the image is highlighted with a bounding box. Alongside these boxes, you'll find labels and confidence scores, indicating what the model thinks each object is and how sure it is about its prediction. This mode is ideal for understanding the model's performance in terms of object localization and classification accuracy.
Overlay Mode for the Live Classification Result
In this view, bounding boxes are drawn around the detected objects, with labels and confidence scores displayed within the image context. This approach offers a clearer view of how the bounding boxes align with the objects in the image, making it easier to assess the precision of object localization. The overlay view is particularly useful for examining the model's ability to accurately detect and outline objects within a complex visual scene.
Summary Table
Name: This field displays the name of the sample file analyzed by the model. For instance, 'sample.jpg.22l74u4f' is the file name in this case.
CATEGORY: Lists the types of objects that the model has been trained to detect. In this example, two categories are shown: 'coffee' and 'lamp'.
COUNT: Indicates the number of times each category was detected in the sample file. In this case, both 'coffee' and 'lamp' have a count of 1, meaning each object was detected once in the sample.
INFO: This column provides additional information about the model's performance. It displays the 'Precision score', which, in this example, is 95.00%. The precision score represents the model's accuracy in making correct predictions over a range of Intersection over Union (IoU) values, known as the mean Average Precision (mAP).
With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the preprocessing steps, neural network weights, and classification code - in a single C++ library or model file that you can include in your embedded software.
Running the impulse on your Raspberry Pi 4 or Jetson Nano
From the terminal just run edge-impulse-linux-runner
. This will build and download your model, and then run it on your development board. If you're on the same network you can get a view of the camera, and the classification results directly from your dev board. You'll see a line like:
Open this URL in a browser to see your impulse running!
Running the impulse on your mobile phone
On your mobile phone just click Switch to classification mode at the bottom of your phone screen. Point it at an object and press 'Capture'.
Integrating the model in your own application
Congratulations! You've added object detection to your sensors. Now that you've trained your model you can integrate your impulse in the firmware of your own edge device, see the Edge Impulse for Linux documentation for the Node.js, Python, Go and C++ SDKs that let you do this in a few lines of code and make this model run on any device. Here's an example of sending a text message through Twilio when an object is seen.
Or if you're interested in more, see our tutorials on Continuous motion recognition or Sound recognition. If you have a great idea for a different project, that's fine too. Edge Impulse lets you capture data from any sensor, build custom processing blocks to extract features, and you have full flexibility in your Machine Learning pipeline with the learning blocks.
We can't wait to see what you'll build! 🚀
is a brand-new approach to run object detection models on constrained devices. FOMO is a ground-breaking algorithm that brings real-time object detection, tracking and counting to microcontrollers for the first time. FOMO is 30 times faster than MobileNet SSD and can run under 200K of RAM.
In this tutorial, we will explain how to count cars to estimate parking occupancy using FOMO.
View the finished project, including all data, signal processing and machine learning blocks here: .
Limitations of FOMO
FOMO does not output bounding boxes but will give you the object's location using centroids. Hence the size of the object is not available.
FOMO works better if the objects have a similar size.
Objects shouldn’t be too close to each other, although this can be optimized when increasing the image input resolution.
If you need the size of the objects for your project, head to the default . tutorial.
For this tutorial, you'll need a .
If you don't have any of these devices, you can also upload an existing dataset through the or use your to connect your device to Edge Impulse. After this tutorial, you can then deploy your trained machine learning model as a C++ library or as a WebAssembly package and run it on your device.
You can collect data from the following devices:
- for the Raspberry Pi 4 and the Jetson Nano.
Collecting image data from any of the that have a camera.
With the data collected, we need to label this data. Go to Data acquisition, verify that you see your data, then click on the 'Labeling queue' to start labeling.
Why use bounding box inputs?
To keep the interoperability with other models, your training image input will use bounding boxes although we will output centroids in the inference process. As such FOMO will use in the background translation between bounding boxes and segmentation maps in various parts of the end-to-end flow. This includes comparing sets between the bounding boxes and the segmentation maps to run profiling and scoring.
Using your own trained model - Useful when you already have a trained model with classes similar to your new task.
Using Object tracking - Useful when you have objects that are similar in size and common between images/frames.
For our case, since the 'car' object is part of the COCO dataset, we will use the YoloV5 pre-trained model to accelerate this process. To enable this feature, we will first click the Label suggestions dropdown,then select “Classify using YOLOv5.”
From the image above, the YOLOV5 model can already help us annotate more than 90% of the cars without us having to do it manually by our hands.
To validate whether a model works well you want to keep some data (typically 20%) aside, and don't use it to build your model, but only to validate the model. This is called the 'test set'. You can switch between your training and test sets with the two buttons above the 'Data collected' widget. If you've collected data on your development board there might be no data in the testing set yet. You can fix this by going to Dashboard > Perform train/test split.
To configure this, go to Create impulse, set the image width and image height to 96, the 'resize mode' to Fit shortest axis and add the 'Images' and 'Object Detection (Images)' blocks. Then click Save Impulse.
To configure your processing block, click Images in the menu on the left. This will show you the raw data on top of the screen (you can select other files via the drop-down menu), and the results of the processing step on the right. You can use the options to switch between RGB
and Grayscale
modes. Finally, click on Save parameters.
This will send you to the 'Feature generation' screen. In here you'll:
Resize all the data.
Apply the processing block on all this data.
Create a 3D visualization of your complete dataset.
Click Generate features to start the process.
Afterward, the Feature explorer will load. This is a plot of all the data in your dataset. Because images have a lot of dimensions (here: 96x96x1=9216 features for grayscale) we run a process called 'dimensionality reduction' on the dataset before visualizing this. Here the 9216 features are compressed down to 2, and then clustered based on similarity as shown in the feature explorer below.
With all data processed it's time to start training our FOMO model. The model will take an image as input and output objects detected using centroids. For our case, it will show centroids of cars detected on the images.
FOMO is fully compatible with any MobileNetV2 model, and depending on where the model needs to run you can pick a model with a higher or lower alpha. Transfer learning also works (although you need to train your base models specifically with FOMO in mind). Another advantage of FOMO is that it has very few parameters to learn from compared to normal SSD networks making the network even much smaller and faster to train. Together this gives FOMO the capabilities to scale from the smallest microcontrollers to full gateways or GPUs.
To configure FOMO, head over to the ‘Object detection’ section, and select 'Choose a different model' then select one of the FOMO models as shown in the image below.
Make sure to start with a learning rate of 0.001 then click start training. After the model is done you'll see accuracy numbers below the training output. You have now trained your FOMO object detection model!
As you may have noticed from the training results above, FOMO uses F1 Score as its base evaluating metric as compared to SSD MobileNetV2 which uses Mean Average Precision (mAP). Using Mean Average Precision (mAP) as the sole evaluation metric can sometimes give limited insights into the model’s performance. This is particularly true when dealing with datasets with imbalanced classes as it only measures how accurate the predictions are without putting into account how good or bad the model is for each class. The combination between F1 score and a confusion matrix gives us both the balance between precision and recall of our model as well as how the model performs for each class.
With the model trained let's try it out on some test data. When collecting the data we split the data up between a training and a testing dataset. The model was trained only on the training data, and thus we can use the data in the testing dataset to validate how well the model will work in the real world. This will help us ensure the model has not learned to overfit the training data, which is a common occurrence. To validate our model, we will go to Model Testing and select Classify all.
Given the little training data we had and the few cycles we trained on, we got an accuracy of 84.62% which can be improved further. To see the classification in detail, we will head to Live Classification* and select one image from our test sample. Click the three dots next to an item, and select Show classification. We can also capture new data directly from your development board from here.
Live Classification Result
From the test image above, our model was able to detect 16 cars out of the actual possible 18 which is a good performance. This can be seen in side by side by default, but you can also switch to overlay mode to see the model's predictions against the actual image content.
Overlay Mode for the Live Classification Result
A display option where the original image and the model's detections overlap, providing a clear juxtaposition of the model's predictions against the actual image content.
Summary Table
The summary table for a FOMO classification result provides a concise overview of the model's performance on a specific sample file, such as 'Parking_data_2283.png.2tk8c1on'. This table is organized as follows:
CATEGORY: Metric, Object category, or class label, e.g., car. COUNT: Shows detection accuracy, frequency, e.g., car detected 7 times.
INFO: Provides performance metrics definitions, including F1 Score, Precision, and Recall, which offer insights into the model's accuracy and efficacy in detection:
Table Metrics F1 Score: (77.78%): Balances precision and recall. Precision: (100.00%): Accuracy of correct predictions. Recall: (63.64%): Proportion of actual objects detected.
Viewing Options
Bottom-right controls adjust the visibility of ground truth labels and model predictions, enhancing the analysis of the model's performance:
Prediction Controls: Customize the display of model predictions, including:
Show All: Show all detections and confidence scores.
Show Correct Only: Focus on accurate model predictions.
Show incorrect only: Pinpoint undetected objects in the ground truth.
Ground Truth Controls: Toggle the visibility of original labels for direct comparison with model predictions.
Show All: Display all ground truth labels.
Hide All: Conceal all ground truth labels.
Show detected only: Highlight ground truth labels detected by the model.
Show undetected only: Identify ground truth labels missed by the model.
With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the preprocessing steps, neural network weights, and classification code - in a single C++ library or model file that you can include in your embedded software.
From the terminal just run edge-impulse-linux-runner
. This will build and download your model, and then run it on your development board. If you're on the same network you can get a view of the camera, and the classification results directly from your dev board. You'll see a line like:
Open this URL in a browser to see your impulse running!
Go to the Deployment tab, on Build firmware section and select the board-compatible firmware to download it.
Follow the instruction provided to flash the firmware to your board and head over to your terminal and run the edge-impulse-run-impulse --debug
command:
You'll also see a URL you can use to view the image stream and results in your browser:
We can't wait to see what you'll build! 🚀
In this tutorial, you'll learn how to set up the HR/HRV Features block within Edge Impulse Studio to process your physiological data and extract meaningful heart rate (HR) and heart rate variability (HRV) features. These features can be leveraged in various downstream machine learning tasks, such as activity classification, stress detection, and health monitoring. We will extend this section over time with more examples.
For this guide, we will be using a subset of the publicly available PPG-DaLiA dataset, which contains data collected from 15 subjects during daily activities. This dataset includes physiological signals such as photoplethysmography (PPG), accelerometer data, and heart rate ground truth, allowing us to explore how the HR/HRV Features block can be applied to real-world data.
You can download the DaLIA-PPG dataset or clone the project S1_E4 from the links below:
All users (Community, Professional and Enterprise Plan) can extract HR/HRV features using this processing block for testing purposes. However, the deployment option is only available for Enterprise users. Contact your Solution Engineer to enable it.
Heart Rate refers to the number of times your heart beats per minute (BPM). It's a straightforward metric that indicates your cardiac activity and is influenced by factors like physical activity, stress, and overall health.
Heart Rate Variability measures the variation in time intervals between successive heartbeats, known as interbeat intervals (IBIs). HRV provides deeper insights into the autonomic nervous system's regulation of the heart, reflecting your body's ability to adapt to stress, recover, and maintain homeostasis.
Key HRV Metrics:
RMSSD (Root Mean Square of Successive Differences): Reflects short-term HRV and parasympathetic activity.
SDNN (Standard Deviation of NN intervals): Indicates overall HRV.
pNN50: Percentage of intervals differing by more than 50 ms, another measure of parasympathetic activity.
Understanding both HR and HRV can provide a comprehensive view of an individual's cardiovascular and autonomic health.
Physiological Data: You can download a subset of PPG-DaLIA the DaLIA-PPG dataset S1_E4:
Ensure your CSV file is structured with the following columns:
Time: The timestamp for each data point.
X, Y, Z: Accelerometer data for motion artifact compensation (optional).
ECG/PPG: The PPG or ECG signal for heart rate estimation.
HR: Heart rate value, if available (should be uploaded as a multi-label sample for regression).
Label: The activity label (optional but useful for classification tasks).
HR Calculation: Avoid uploading data in short snippets. HR calculation relies on history and feedback to accumulate a continuous stream for better calculation and smoothing. Upload long, contiguous data for the most accurate results.
Community tier need to use single label and should keep the length to 2 seconds for the best accuracy.
You can download a subset of the PPG-DaLIA dataset S1_E4:
Log into your Edge Impulse Studio.
Navigate to Data Acquisition in the left-hand panel.
Click Upload Data and select your CSV file for each subject.
Ensure that the data is correctly parsed, and assign appropriate labels for heartrate applicable (e.g., 100, 90, etc.).
Go to Impulse Design > Create Impulse.
Add a HR and HRV Features block:
Input Axes: Select accX, accY, accZ, PPG (if using accelerometer data for motion artifact correction).
Set Window Size to 2000 ms.
Set Frequency (Hz) to 25 Hz (tolerance +/- 1 Hz) or 50 Hz (tolerance +/- 3 Hz).
Under Impulse Design, add the HR/HRV Features block.
Input Axes: Select your PPG signal for HR estimation.
HR Window Size: Set the window size for heart rate computation (e.g., 9 seconds, and no shorter than 2 seconds).
HRV Settings: Enable all HRV features to see RMSSD and other params.
Input Axes: Select your ECG signal for HR estimation.
HR Window Size: Set the window size for heart rate computation (e.g., 10 seconds and upwards of 90 seconds for best accuracy).
HRV Settings: Enable HRV features such as RMSSD or time-series statistics.
Input Axes: Select your PPG signal for HR estimation.
Accelerometer X, Y, Z: Include these axes to filter motion artifacts.
HR Window Size: Set the window size for heart rate computation (e.g., 10 seconds).
HRV Settings: Enable HRV features such as RMSSD or time-series statistics.
Click on Generate Features from the top menu.
Review the feature generation output to ensure that the raw signals are correctly processed.
Edge Impulse will generate the features for both heart rate and HRV over the specified window size.
Add a Learning Block such as a Regression if you wish to estimate HR
Ensure that HR/HRV is selected as the input feature.
Train the model using your labeled data to predict different heart rate.
After training, use Model Testing to evaluate the performance of your HR/HRV feature extraction and heart rate prediction model.
Upload test data to ensure that the heart rate is correctly estimated and any HRV features are extracted as expected.
For enterprise users, you can deploy the model to your target device and start collecting real-time heart rate and HRV data. When the model is trained and validated, you can deploy it to your target device or C++ binary for real-time heart rate and HRV monitoring.
Deployment will look as follows:
To include heart rate extraction alongside your classifier results, define the following macro in your deployment:
This macro will enable the HR/HRV Features block to calculate heart rate and HRV features in real-time.
By following this tutorial, you've successfully set up the HR/HRV Features block in Edge Impulse Studio, extracted meaningful cardiovascular features, and optionally trained a machine learning model. This foundation enables you to build robust, real-time heart rate and HRV monitoring solutions for applications like health monitoring, stress detection, and fitness tracking.
Neural networks are not limited to working with one type of data at a time. One of their biggest advantages is that they are incredibly flexible with the type of input data, so long as the format and ordering of that data stays the same from training to inference. As a result, we can use them to perform sensor fusion for a variety of tasks.
Sensor fusion is the process of combining data from different types of sensors or similar sensors mounted in different locations, which gives us more information to make decisions and classifications. For example, you could use temperature data with accelerometer data to get a better idea of a potential anomaly!
In this tutorial, you will learn how to use Edge Impulse to perform sensor fusion on the Arduino Nano 33 BLE Sense.
Example Project: You can find the dataset and impulse used throughout this tutorial in .
Multi-impulse vs multi-model vs sensor fusion
Running multi-impulse refers to running two separate projects (different data, different DSP blocks and different models) on the same target. It will require modifying some files in the EI-generated SDKs. See the
Running multi-model refers to running two different models (same data, same DSP block but different tflite models) on the same target. See how to run a motion classifier model and an anomaly detection model on the same device in .
Sensor fusion refers to the process of combining data from different types of sensors to give more information to the neural network. To extract meaningful information from this data, you can use the same DSP block (like in this tutorial), multiples DSP blocks, or use neural networks embeddings like this tutorial.
For this tutorial, you'll need a .
For this demo, we'll show you how to identify different environments by using a fusion of temperature, humidity, pressure, and light data. In particular, I'll have the Arduino board identify different rooms in my house as well as outside. Note that the we assume that the environment is static--if I turn out lights or the outside temperature changes, the model will not work. However, it demonstrates how we can combine different sensor data with machine learning to do classification!
As we will be collecting data from our Arduino board connected to a computer, it helps to have a laptop that you can move to different rooms.
Create a new project on the Edge Impulse studio.
Connect the Arduino Nano 33 BLE to your computer. Follow the to upload the Edge Impulse firmware to the board and connect it to your project.
Go to Data acquisition. Under Record new data, select your device and set the label to bedroom
. Change Sensor to Environmental + Interactional
, set the Sample length to 10000
ms and Frequency to 12.5Hz
.
Stand in one of your rooms with your Arduino board (and laptop). Click Start sampling and slowly move the board around while data is collected. After sampling is complete, you should see a new data plot with a different line for each sensor.
Variations
Try to stand in different parts of each room while collecting data.
Repeat this process to record about 3 minutes of data for the bedroom class. Try to stand in a different spot in the room while collecting data--we want a robust dataset that represents the features of each room. Head to another room and repeat data collection. Continue doing this until you have around 3 minutes of data for each of the following classes:
Bedroom
Hallway
Outside
You are welcome to try other rooms or locations. For this demo, I found that my bedroom, kitchen, and living room all exhibited similar environmental and lighting properties, so the model struggled to tell them apart.
Head to Dashboard and scroll down to Danger zone. Click Perform train/test split and follow the instructions in the pop-up window to split your dataset into training and testing groups. When you're done, you can head back to Data acquisition to see that your dataset has been split. You should see about 80% of your samples in Training data and about 20% in Test data.
Head to Create impulse. Change the Window increase to 500 ms
. Add a Flatten block. Notice that you can choose which environmental and interactional sensor data to include. Deselect proximity and gesture, as we won't need those to detect rooms. Add a Classification (Keras) learning block
Click Save impulse.
Head to Flatten. You can select different samples and move the window around to see what the DSP result will look like for each set of features to be sent to the learning block.
The Flatten block will compute the average, minimum, maximum, root-mean square, standard deviation, skewness, and kurtosis of each axis (e.g. temperature, humidity, brightness, etc.). With 7 axes and 7 features computed for each axis, that gives us 49 features for each window being sent to the learning block. You can see these computed features under Processed features.
Click Save parameters. On the next screen, select Calculate feature importance and click Generate features.
After a few moments, you should be able to explore the features of your dataset to see if your classes are easily separated into categories.
Interestingly enough, it looks like temperature and red light values were the most important features in determining the location of the Arduino board.
With our dataset collected and features processed, we can train our machine learning model. Click on NN Classifier. Change the Number of training cycles to 300
and click Start training. We will leave the neural network architecture as the default for this demo.
During training, parameters in the neural network's neurons are gradually updated so that the model will try to guess the class of each set of data as accurately as possible. When training is complete, you should see a Model panel appear on the right side of the page.
The Confusion matrix gives you an idea of how well the model performed at classifying the different sets of data. The top row gives the predicted label and the column on the left side gives the actual (ground-truth) label. Ideally, the model should predict the classes correctly, but that's not always the case. You want the diagonal cells from the top-left to the bottom-right to be as close to 100% as possible.
The On-device performance provides some statistics about how the model will likely run on a particular device. By default, an Arm Cortex-M4F running at 80 MHz is assumed to be your target device. The actual memory requirements and run time may vary on different platforms.
Rather than simply assume that our model will work when deployed, we can run inference on our test dataset as well as on live data.
First, head to Model testing, and click Classify all. After a few moments, you should see results from your test set.
You can click on the three dots next to an item and select Show classification. This will give you a classification result screen where you can see results information in more detail.
Additionally, we can test the impulse in a real-world environment to make sure the model has not overfit the training data. To do that, head to Live classification. Make sure your device is connected to the Studio and that the Sensor, Sample length, and Frequency match what we used to initially capture data.
Click Start sampling. A new sample will be captured from your board, uploaded, and classified. Once complete, you should see the classification results.
In the example above, we sampled 10 seconds of data from the Arduino. This data is split into 1-second windows (the window moves over 0.5 seconds each time), and the data in that window is sent to the DSP block. The DSP block computes the 49 features that are then sent to the trained machine learning model, which performs a forward pass to give us our inference results.
As you can see, the inference results from all of the windows claimed that the Arduino board was in the bedroom, which was true! This is great news for our model--it seems to work even on unseen data.
Now that we have an impulse with a trained model and we've tested its functionality, we can deploy the model back to our device. This means the impulse can run locally without an internet connection to perform inference!
Edge Impulse can package up the entire impulse (preprocessing block, neural network, and classification code) into a single library that you can include in your embedded software.
Click on Deployment in the menu. Select the library that you would like to create, and click Build at the bottom of the page.
Running your impulse locally
Well done! You've trained a neural network to determine the location of a development board based on a fusion of several sensors working in tandem. Note that this demo is fairly limited--as the daylight or temperature changes, the model will no longer be valid. However, it hopefully gives you some ideas about how you can mix and match sensors to achieve your machine learning goals.
We can't wait to see what you'll build! 🚀
This tutorial demonstrates how to implement and integrate on an Arduino Opta using Edge Impulse. Anomaly detection is a machine learning technique that identifies unusual patterns in data, making it ideal for monitoring industrial processes and equipment. By training a model to recognize normal behavior, you can detect deviations that may indicate faults or malfunctions.
In this tutorial, you will:
Collect temperature data from a sensor connected to the Opta PLC.
Train a machine learning model in Edge Impulse to detect anomalies in the data.
Deploy the model to the Opta PLC for real-time inference.
Optionally, integrate the model with Arduino Cloud for remote monitoring and visualization.
We ran a webinar on integrating Edge Impulse with Arduino Opta PLC and Blues Wireless for remote monitoring and anomaly detection. The webinar covered the following topics:
Arduino Opta PLC: Learn about the flexibility and ease of integration of the Arduino Opta micro PLC, designed for industrial environments.
Edge Impulse for Machine Learning: Understand how to train and implement ML models for anomaly detection on industrial data directly on your Opta PLC.
Blues for Wireless Connectivity: Explore how Blues' Wireless for PLC Expansion enables secure cellular connectivity, allowing your Opta PLC to communicate with cloud-based monitoring systems.
About the DIN Celsius Board
The DIN Celsius is an all-in-one temperature laboratory offering two independent heaters and a temperature sensor. It allows you to simulate heating scenarios and monitor temperature changes, making it ideal for testing our anomaly detection use case, we can introduce an anomaly by turning off one of the heaters to cause a deviation from the normal condition.
Connecting the DIN Celsius to the Opta PLC
Safety First: Before making any connections, ensure that all power sources are disconnected to prevent electric shock or short circuits.
Connections Overview:
Power Connections:
Connect the +24V and GND terminals of the DIN Celsius to the corresponding power supply pins.
Heater Control:
Connect Relay 3 (pin 2) on the Opta PLC to Input Heat 1 on the DIN Celsius.
Connect Relay 4 (pin 3) on the Opta PLC to Input Heat 2 on the DIN Celsius.
These connections will control the two independent heaters on the DIN Celsius.
Temperature Sensor Input:
Connect the Output Voltage from the DIN Celsius to the I8 input (analog pin A7) on the Opta PLC.
This connection allows the PLC to read the temperature sensor data.
Pin Definitions:
HEAT_LEFT (Relay 3) connected to pin 2
HEAT_RIGHT (Relay 4) connected to pin 3
TEMP_SENS connected to analog pin A7
BTN (User Button) on the Opta PLC
Arduino IDE Configuration
Install Necessary Libraries:
Ensure you have the latest Arduino IDE 2 installed.
Install any required libraries via the Library Manager, such as Edge Impulse SDK and Arduino_HTS221 if using temperature/humidity sensors.
Define Pin Constants:
Testing the Connections
Heater Control Test Code:
Upload the Code: Use the Arduino IDE to upload the sketch to the Opta PLC.
Verify Operation: The LEDs on the DIN Celsius should blink, indicating the heaters are being activated.
Temperature Sensor Reading Test:
Open Serial Monitor: Set the baud rate to 9600.
Observe Readings: You should see numerical values corresponding to the temperature sensor output.
Log In: Access your Edge Impulse account.
New Project: Create a project named, for example, "Opta PLC Anomaly Detection."
Upload a sketch to the Opta PLC that reads the temperature sensor and sends data to Edge Impulse.
Steps:
Upload the Data Forwarder Sketch: Use the Arduino IDE to upload the sketch to the Opta PLC.
Run Data Forwarder: In your terminal, execute the data forwarding command.
Select Serial Port: Choose the serial port corresponding to the Opta PLC.
Label the Data: As you collect data, assign labels to your data (e.g., "normal," "anomalous") based on the system's behavior.
Run Data Forwarder: In your terminal, execute:
Select Serial Port: Choose the serial port corresponding to the Opta PLC.
Label the Data: Assign labels to your data (e.g., "normal," "anomalous") as you collect it.
Add Blocks: 2. Add Blocks:
Processing Block: Select a Time Series or Spectral Analysis block based on your data characteristics.
Learning Block: Choose Anomaly Detection using the Gaussian Mixture Model (GMM).
Window Size: Set according to the data frequency (e.g., 1000 ms).
Window Increase: Set overlap (e.g., 500 ms).
Compute Features: Navigate to the Generate Features tab and run feature generation.
Training Parameters:
Epochs: Start with 100 and adjust as needed.
Learning Rate: Default of 0.005 is usually sufficient. Start Training: Monitor the accuracy and loss graphs.
Model Testing: Use a separate dataset to evaluate model performance.
Adjust if Necessary: Retrain or adjust parameters based on results.
Deployment Tab: In Edge Impulse, go to Deployment.
Select Arduino Library: Download the library tailored for your model.
Add Library to IDE: Import the downloaded library into your Arduino IDE.
Replace Your_Edge_Impulse_Inference_Library.h
with the actual header file name from your downloaded library.
Set ANOMALY_THRESHOLD
to an appropriate value based on your model's performance.
Compile and Upload: Use the Arduino IDE to program the Opta PLC with your new inference code.
Monitor Output: Open the Serial Monitor to observe inference results and system behavior.
This section demonstrates integrating Blues Wireless for remote connectivity and monitoring. By connecting your Opta PLC to the cloud, you can visualize data, receive alerts, and monitor system performance from anywhere.
Explore how Blues' Wireless for PLC Expansion enables seamless, secure cellular connectivity, allowing your Opta PLC to communicate with your cloud-based monitoring systems, regardless of location and without the hassle of local Wi-Fi.
In this section, we will integrate the anomaly detection model deployed on the Arduino Opta PLC with a ladder logic program. This integration will allow the machine learning inferences to interact with the PLC's native ladder logic control system, providing intelligent control responses to anomalies in temperature or motor power.
The Arduino PLC IDE allows you to combine an Arduino sketch with ladder logic using Shared Variables. These shared variables act as a bridge between the Edge Impulse inference running on the PLC and the PLC's control logic written in Ladder Diagram (LD) or other IEC-61131-3 programming languages (such as Function Block Diagram or Structured Text). This enables real-time decision-making based on the machine learning model's output.
Create Shared Variables
In the Arduino PLC IDE, navigate to the Global_vars
section to create shared variables that will store the results from the Edge Impulse inference. Define a shared variable for the anomaly score or classification output of the model.
Example:
Shared variable for storing inference result (e.g., float anomaly_score
).
Modify Inference Sketch
Update the Edge Impulse inference sketch to store the inference result in the shared variable. This will allow the ladder logic to access the result.
Example:
Ladder Logic Program
In the PLC IDE, create a new ladder logic program that will read the anomaly_score
shared variable. The logic can then trigger actions based on the value, such as activating relays, generating alarms, or shutting down equipment in response to detected anomalies.
Example Ladder Logic:
Create a rung that monitors the anomaly_score
. If the score exceeds a certain threshold, the logic can trigger an alarm (e.g., turn on an LED or activate a relay).
Add Inputs and Outputs:
Define Inputs (e.g., sensor values, inference results from the Arduino sketch) and Outputs (e.g., control signals like turning on a relay).
Click Add in the "Shared Inputs" and "Shared Outputs" sections to create global variables. These variables will allow communication between your inference sketch and ladder logic.
Set up the Inputs/Outputs:
Inputs: Define variables that will receive values from the Arduino sketch (e.g., anomaly_score).
Outputs: Define variables that will control actuators (e.g., relay_control).
Step 2: Accessing the Ladder Logic Editor
Create a New Ladder Logic Program:
Go to the Project tab (top-left section).
Right-click on Programs and select New Program.
Name the program (e.g., AnomalyDetection_LD) and select Ladder Diagram (LD) as the language.
Opening the Ladder Logic Editor:
Once the program is created, double-click it to open the Ladder Diagram editor. You will see a canvas where you can start adding blocks.
Step 3: Designing the Ladder Logic
Drag and Drop Components:
On the right panel under the Library Tree, you can see various block types, such as Comparison, Logic, and Arithmetic.
Comparison blocks will allow you to compare the input (e.g., anomaly_score) to a threshold (e.g., >=).
Creating Logic:
Input Condition: Drag a Comparison block (e.g., >=) to compare the anomaly_score to a threshold (e.g., 0.8).
Output Control: Connect the result of the comparison to an Output coil that controls the relay (e.g., relay_control).
Steps for Adding Logic:
Input: Select anomaly_score as the input to the comparison block.
Condition: Set the threshold (e.g., >= 0.8).
Output: Set the output to control a relay (e.g., activate relay_control when the condition is met).
Assigning the Ladder Logic to a Task
In the PLC IDE, assign the Arduino sketch (which runs the inference) to a task such as Fast (runs every 10ms) or Background (runs every 500ms) based on your system’s real-time requirements. Attach the ladder logic program to the same or another appropriate task to ensure it reacts to the updated shared variables in a timely manner.
Steps:
Go to the Project tab and locate the Tasks section.
Assign the ladder logic program (e.g., AnomalyDetection_LD) to an appropriate task (e.g., Fast Task for real-time control).
Monitor and Debug
Use the Watch window in the PLC IDE to monitor the shared variables and ensure the system responds correctly to anomalies detected by the machine learning model.
Upload the Arduino Sketch:
Ensure your Arduino sketch is uploaded to the Opta PLC to provide the inference results.
Run the Ladder Logic:
Start the ladder logic program from the Arduino PLC IDE.
Monitor the shared variables using the Watch window to see how the ladder logic reacts to the inference results.
Real-time Control: By integrating anomaly detection with ladder logic, you can implement real-time, intelligent control systems that take action based on data-driven decisions from the Edge Impulse model.
Easy Troubleshooting: Using ladder logic alongside machine learning allows for clear, visual representation of control logic, making it easier to debug and monitor the system's responses to anomalies.
Seamless PLC Integration: The Arduino PLC IDE provides a smooth environment for combining traditional control logic with modern machine learning, ensuring compatibility and ease of use.
Congratulations! You have successfully implemented anomaly detection on an Arduino Opta PLC using Edge Impulse. This tutorial demonstrates the seamless integration of machine learning models with industrial automation systems, enabling real-time monitoring and fault detection.
For more information on Edge Impulse and Arduino Opta PLC, visit the official websites:
Arduino Cloud Issues: We are aware of issues with Arduino Cloud .properties file format vs IDE and are working with Arduino. If you have issues try moving the .properties file to the same folder as the .ino file and re-uploading the sketch.
This page is part of and describes how you can use development boards with an integrated camera to import image data into Edge Impulse.
First, make sure your device is connected on the Devices page in the Edge Impulse Studio. Then, head to Data acquisition, and under 'Record new data', set a label and select 'Camera' as a sensor (most devices have multiple resolutions). This shows you a nice preview of the camera. Then click Start sampling.
A few moments later - depending on the speed of the development board and the resolution - you'll now have an image collected!
Do this until you have captured 30 images per class from a variety of angles. Also make sure to vary the things you capture for the unknown class.
This page is part of and describes how you can use your mobile phone to import image data into Edge Impulse.
To add your phone to your project, go to the Devices page, select Connect a new device and select Use your mobile phone. A QR code will pop up. Scan this code with your phone and your phone will pop up on the devices screen.
With your phone connected to your project, it's time to start capturing some images and build our dataset. We have a special UI for collecting images quickly, on your phone choose Collecting images?.
On your phone a permission prompt will show up, and then the viewfinder will be displayed. Set the label (in the top corner) to 'lamp', point your camera at your lamp and press Capture.
Afterwards the photo shows up in the studio on the Data acquisition page.
Do this until you have captured 30 images per class from a variety of angles. Also make sure to vary the things you capture for the unknown class.
If you want to upload files directly to an Edge Impulse project, we recommend using the . However, sometimes you cannot upload your samples directly, as you might need to convert the files to one of the accepted formats or modify the data prior to model training. Edge Impulse offers for some types of projects, but you might want to create your own custom augmentation scheme. Or perhaps you want to and script the upload process.
The Python SDK offers a set of functions to help you move data into and out of your project. This can be extremely helpful when generating or augmenting your dataset. The following cells demonstrate some of these upload and download functions.
You can find the API documentation for the functions found in this tutorial .
WARNING: This notebook will add and delete data in your Edge Impulse project, so be careful! We recommend creating a throwaway project when testing this notebook.
Note that you might need to refresh the page with your Edge Impulse project to see the samples appear.
You will need to obtain an API key from an Edge Impulse project. Log into and create a new project. Open the project, navigate to Dashboard and click on the Keys tab to view your API keys. Double-click on the API key to highlight it, right-click, and select Copy.
Note that you do not actually need to use the project in the Edge Impulse Studio. We just need the API Key.
Paste that API key string in the ei.API_KEY
value in the following cell:
The following file formats are allowed: .cbor, .json, .csv, .wav, .jpg, .png, .mp4, .avi.
If you head to the Data acquisition page on your project, you should see images in your dataset.
You can download samples from your Edge Impulse project if you know the sample IDs. You can get sample IDs by calling the ei.data.get_sample_ids()
function, which allows you to filter IDs based on filename, category, and label.
Take a look at the files in this directory. You should see the downloaded images. They should match the images in the dataset/ directory, which were the original images that we uploaded.
If you know the ID of the sample you would like to delete, you can call the delete_sample_by_id()
function. You can also delete all the samples in your project by calling delete_all_samples()
.
Take a look at the data in your project. The samples that we uploaded should be gone.
Important! The annotations file must be named exactly info.labels
If you head to the Data acquisition page on your project, you should see images in your dataset along with the bounding box information.
If you head to the Data acquisition page on your project, you should see your time series data.
The raw data must be encoded in an IO object. We convert the dictionary objects to a BytesIO
object, but you can also read in data from .json files.
If you head to the Data acquisition page on your project, you should see your time series data.
Important! NumPy arrays must be in the shape
(Number of samples, number of data points, number of sensors)
If you are working with image data in NumPy, we recommend saving those images as .png or .jpg files and using upload_directory()
.
If you head to the Data acquisition page on your project, you should see your time series data. Note that the sample names are randomly assigned, so we recommend recording the sample IDs when you upload.
Note that several other packages exist that work as drop-in replacements for pandas. You can use these replacements so long as you import that with the name pd
. For example, one of:
The first option is to upload one dataframe for each sample (non-time series)
You can also upload one dataframe for each sample (time series). As with previous examples, we'll assume that the sample rate is 10 ms.
You can upload non-time series data where each sample is a row in the dataframe. Note that you need to provide labels in the rows.
A "wide" dataframe is one where each column represents a value in the time series data, and the rows become individual samples. Note that you need to provide labels in the rows.
A DataFrame can also be divided into "groups" so you can upload multidimensional time series data.
Your project API key can be used to enable programmatic access to Edge Impulse. You can create and/or obtain a key from your project's Dashboard, under the Keys
tab. API keys are long strings, and start with ei_
:
Open a terminal and run the Edge Impulse daemon. The daemon is the service that connects your hardware with any Edge Impulse project:
Copy your project's ID from the project's Dashboard under the Project Info
section:
Replace the PROJECT_ID
below with the ID of your project you selected and enter your API key when prompted:
In this tutorial, you'll use machine learning to build a gesture recognition system that runs on a microcontroller. This is a hard task to solve using rule-based programming, as people don't perform gestures in the exact same way every time. But machine learning can handle these variations with ease. You'll learn how to collect high-frequency data from real sensors, use signal processing to clean up data, build a neural network classifier, and how to deploy your model back to a device. At the end of this tutorial, you'll have a firm understanding of applying machine learning in embedded devices using Edge Impulse.
There is also a video version of this tutorial:
You can view the finished project, including all data, signal processing and machine learning blocks here: .
If your device is connected (green dot) under Devices in the studio you can proceed:
Data ingestion
With your device connected, we can collect some data. In the studio go to the Data acquisition tab. This is the place where all your raw data is stored, and - if your device is connected to the remote management API - where you can start sampling new data.
Under Record new data, select your device, set the label to updown
, the sample length to 10000
, the sensor to Built-in accelerometer
and the frequency to 62.5Hz
. This indicates that you want to record data for 10 seconds, and label the recorded data as updown
. You can later edit these labels if needed.
After you click Start sampling move your device up and down in a continuous motion. In about twelve seconds the device should complete sampling and upload the file back to Edge Impulse. You see a new line appear under 'Collected data' in the studio. When you click it you now see the raw data graphed out. As the accelerometer on the development board has three axes you'll notice three different lines, one for each axis.
Continuous movement
It's important to do continuous movements as we'll later slice up the data in smaller windows.
Machine learning works best with lots of data, so a single sample won't cut it. Now is the time to start building your own dataset. For example, use the following four classes, and record around 3 minutes of data per class:
Idle - just sitting on your desk while you're working.
Snake - moving the device over your desk as a snake.
Wave - waving the device from left to right.
Updown - moving the device up and down.
Variations
Make sure to perform variations on the motions. E.g. do both slow and fast movements and vary the orientation of the board. You'll never know how your user will use the device. It's best to collect samples of ~10 seconds each.
Prebuilt dataset
With the training set in place, you can design an impulse. An impulse takes the raw data, slices it up in smaller windows, uses signal processing blocks to extract features, and then uses a learning block to classify new data. Signal processing blocks always return the same values for the same input and are used to make raw data easier to process, while learning blocks learn from past experiences.
For this tutorial we'll use the 'Spectral analysis' signal processing block. This block applies a filter, performs spectral analysis on the signal, and extracts frequency and spectral power data. Then we'll use a 'Neural Network' learning block, that takes these spectral features and learns to distinguish between the four (idle, snake, wave, updown) classes.
In the studio go to Create impulse, set the window size to 2000
(you can click on the 2000 ms.
text to enter an exact value), the window increase to 80
, and add the 'Spectral Analysis' and 'Classification (Keras)' blocks. Then click Save impulse.
To configure your signal processing block, click Spectral features in the menu on the left. This will show you the raw data on top of the screen (you can select other files via the drop down menu), and the results of the signal processing through graphs on the right. For the spectral features block you'll see the following graphs:
Filter response - If you have chosen a filter (with non zero order), this will show you the response across frequencies. That is, it will show you how much each frequency will be attenuated.
After filter - the signal after applying the filter. This will remove noise.
Spectral power - the frequencies at which the signal is repeating (e.g. making one wave movement per second will show a peak at 1 Hz).
A good signal processing block will yield similar results for similar data. If you move the sliding window (on the raw data graph) around, the graphs should remain similar. Also, when you switch to another file with the same label, you should see similar graphs, even if the orientation of the device was different.
Bonus exercise: filters
Try to reason about the filter parameters. What does the cut-off frequency control? And what do you see if you switch from a low-pass to a high-pass filter?
Set the filter to low pass with the following parameters:
Once you're happy with the result, click Save parameters. This will send you to the 'Feature generation' screen. In here you'll:
Split all raw data up in windows (based on the window size and the window increase).
Apply the spectral features block on all these windows.
Calculate feature importance. We will use this later to set up the anomaly detection.
Click Generate features to start the process.
Afterward the 'Feature explorer' will load. This is a plot of all the extracted features against all the generated windows. You can use this graph to compare your complete data set. A good rule of thumb is that if you can visually identify some clusters by classes, then the machine learning model will be able to do so as well.
With all data processed it's time to start training a neural network. Neural networks are a set of algorithms, modeled loosely after the human brain, that are designed to recognize patterns. The network that we're training here will take the signal processing data as an input, and try to map this to one of the four classes.
So how does a neural network know what to predict? A neural network consists of layers of neurons, all interconnected, and each connection has a weight. One such neuron in the input layer would be the height of the first peak of the X-axis (from the signal processing block); and one such neuron in the output layer would be wave
(one the classes). When defining the neural network all these connections are initialized randomly, and thus the neural network will make random predictions. During training, we then take all the raw data, ask the network to make a prediction, and then make tiny alterations to the weights depending on the outcome (this is why labeling raw data is important).
This way, after a lot of iterations, the neural network learns; and will eventually become much better at predicting new data. Let's try this out by clicking on NN Classifier in the menu.
Set 'Number of training cycles' to 1
. This will limit training to a single iteration. And then click Start training.
Now change 'Number of training cycles' to 2
and you'll see performance go up. Finally, change 'Number of training cycles' to 30
and let the training finish.
You've just trained your first neural networks!
100% accuracy
You might end up with 100% accuracy after training for 100 training cycles. This is not necessarily a good thing, as it might be a sign that the neural network is too tuned for the specific test set and might perform poorly on new data (overfitting). The best way to reduce this is by adding more data or reducing the learning rate.
From the statistics in the previous step we know that the model works against our training data, but how well would the network perform on new data? Click on Live classification in the menu to find out. Your device should (just like in step 2) show as online under 'Classify new data'. Set the 'Sample length' to 10000
(10 seconds), click Start sampling and start doing movements. Afterward, you'll get a full report on what the network thought you did.
If the network performed great, fantastic! But what if it performed poorly? There could be a variety of reasons, but the most common ones are:
There is not enough data. Neural networks need to learn patterns in data sets, and the more data the better.
The data does not look like other data the network has seen before. This is common when someone uses the device in a way that you didn't add to the test set. You can add the current file to the test set by clicking ⋮
, then selecting Move to training set. Make sure to update the label under 'Data acquisition' before training.
The model has not been trained enough. Up the number of epochs to 200
and see if performance increases (the classified file is stored, and you can load it through 'Classify existing validation sample').
The model is overfitting and thus performs poorly on new data. Try reducing the learning rate or add more data.
The neural network architecture is not a great fit for your data. Play with the number of layers and neurons and see if performance improves.
As you see there is still a lot of trial and error when building neural networks, but we hope the visualizations help a lot. You can also run the network against the complete validation set through 'Model validation'. Think of the model validation page as a set of unit tests for your model!
With a working model in place, we can look at places where our current impulse performs poorly.
Neural networks are great, but they have one big flaw. They're terrible at dealing with data they have never seen before (like a new gesture). Neural networks cannot judge this, as they are only aware of the training data. If you give it something unlike anything it has seen before it'll still classify as one of the four classes.
Let's look at how this works in practice. Go to 'Live classification' and record some new data, but now vividly shake your device. Take a look and see how the network will predict something regardless.
So, how can we do better? If you look at the feature explorer, you should be able to visually separate the classified data from the training data. We can use this to our advantage by training a new (second) network that creates clusters around data that we have seen before, and compares incoming data against these clusters. If the distance from a cluster is too large you can flag the sample as an anomaly, and not trust the neural network.
To add this block go to Create impulse, click Add learning block, and select 'Anomaly Detection (K-Means)'. Then click Save impulse.
To configure the clustering model click on Anomaly detection in the menu. Here we need to specify:
The number of clusters. Here use 32
.
Click Start training to generate the clusters. You can load existing validation samples into the anomaly explorer with the dropdown menu.
Axes
The anomaly explorer only plots two axes at the same time. Under 'average axis distance' you see how far away from each axis the validation sample is. Use the dropdown menu's to change axes.
If you now go back to 'Live classification' and load your last sample, it should now have tagged everything as anomaly. This is a great example where signal processing (to extract features), neural networks (for classification) and clustering algorithms (for anomaly detection) can work together.
With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the signal processing code, neural network weights, and classification code - up in a single C++ library that you can include in your embedded software.
Mobile phone
To export your model, click on Deployment in the menu. Then under 'Build firmware' select your development board, and click Build. This will export the impulse, and build a binary that will run on your development board in a single step. After building is completed you'll get prompted to download a binary. Save this on your computer.
When you click the Build button, you'll see a pop-up with text and video instructions on how to deploy the binary to your particular device. Follow these instructions. Once you are done, we are ready to test your impulse out.
We can connect to the board's newly flashed firmware over serial. Open a terminal and run:
Serial daemon
If the device is not connected over WiFi, but instead connected via the Edge Impulse serial daemon, you'll need stop the daemon. Only one application can connect to the development board at a time.
This will sample data from the sensor, run the signal processing code, and then classify the data:
Continuous movement
We trained a model to detect continuous movement in 2 second intervals. Thus, changing your movement while sampling will yield incorrect results. Make sure you've started your movement when 'Sampling...' gets printed. In between sampling, you have two seconds to switch movements.
To run the continuous sampling, run the following command:
Victory! You've now built your first on-device machine learning model.
We can't wait to see what you'll build! 🚀
This page is part of and describes how you can use the OpenMV Cam H7 Plus to build a dataset, and import the data into Edge Impulse.
To set up your OpenMV camera, and collect some data:
Install the .
Follow the to clean the sensor and focus the lens.
Connect a micro-USB cable to the camera, and open the OpenMV IDE. The camera should automatically update to the latest firmware.
Verify that the camera can capture live images, by clicking on the Connect button in the bottom left corner, then pressing Play to run the application.
A live feed from your camera will be displayed in the top right corner of the IDE.
Once your camera is up and running, it's time to start capturing some images and build our dataset.
First, set up a new dataset via Tools -> Dataset Editor, select New Dataset.
This opens the 'Dataset editor' panel on the left side, and the 'dataset capture script' in the main panel of the IDE. Here, create three classes: "plant", "lamp" and "unknown". It's important to add an unknown class that contains random images which are neither lamps nor plants.
As we'll build a model that takes in square images, change the 'Dataset capture script' to read:
Now you can capture data for the three classes.
Click the Play icon to run the 'dataset capture script' on your OpenMV camera.
Select one of the classes by clicking on the folder name in the 'Dataset editor'.
Take a snap by clicking the Capture data (camera icon) button.
Do this until you have captured 30 images per class from a variety of angles. Also make sure to vary the things you capture for the unknown class.
To import the dataset into Edge Impulse go to Tools > Dataset Editor > Export > Upload to Edge Impulse project.
Then, choose the project name, and the split between training and testing data (recommended to keep this to 80/20).
A duplicate check runs when you upload new data, so you can upload your dataset multiple times (for example, when you've added new files) without adding the same data twice.
Training and testing data split
The split between training and testing data is based on the hash of the file in order to have a deterministic process. As a consequence you may not have a perfect 80/20 split between training and testing, but this process ensures samples are always placed in the same category.
Our dataset now appears under the Data acquisition section of our project.
Alternatively, you can capture your images using another camera, and then upload them directly from the studio by going to Data acquisition and clicking the 'Upload' icon or using Edge Impulse CLI .
All our collected images will be staged for annotation at the "labeling queue". Labeling your objects is as easy as dragging a box around the object, and entering a label. However, when you have a lot of images, this manual annotation method can become tiresome and time consuming. To make this task even easier, Edge impulse provides methods that can help you save time and energy. The AI assisted labeling techniques include:
Using YoloV5 - Useful when your objects are part of the common objects in the .
One of the beauties of FOMO is its fully convolutional nature, which means that just the ratio is set. Thus, it gives you more flexibility in its usage compared to the classical . method. For this tutorial, we have been using 96x96 images but it will accept other resolutions as long as the images are square.
To run using an Arduino library, go to the studio Deployment tab on Create Library section and select Arduino Library to download your custom Arduino library. Go to your Arduino IDE, then click on Sketch >> Include Library >> Add .Zip ( Your downloaded Arduino library). Make sure to follow the instruction provided on . Open Examples >> Examples from custom library and select your library. Upload the ''Portenta_H7_camera'' sketch to your Portenta then open your serial monitor to view results.
Congratulations! You've added object detection using FOMO to your sensors. Now that you've trained your model you can integrate your impulse in the firmware of your own edge device, see or the documentation for the Node.js, Python, Go and C++ SDKs that let you do this in a few lines of code and make this model run on any device.
when an object is seen.
Or if you're interested in more, see our tutorials on or . If you have a great idea for a different project, that's fine too. Edge Impulse lets you capture data from any sensor, build to extract features, and you have full flexibility in your Machine Learning pipeline with the learning blocks.
Read on in the or speak with your enterprise support team for more guidance.
For Professional and Community Plan users, the deployment option is not available, contact our sales team if you want to upgrade to an :
Edge Impulse Documentation:
If you have any questions or need further assistance, feel free to reach out on the or consult the documentation.
An impulse is a combination of preprocessing (DSP) blocks followed by machine learning blocks. It will slice up our data into smaller windows, use signal processing to extract features, and then train a machine learning model. Because we are using environmental and light data, which are slow-moving averages, we will use the for preprocessing.
You can also look at the Feature importance section to get an idea of which features are the most important in determining class membership. You can read more about feature importance .
If you see a lot of confusion between classes, it means you need to gather more data, try different features, use a different model architecture, or train for a longer period of time (more epochs). See to learn about ways to increase model performance.
See to learn how to deploy your impulse to a variety of platforms.
If you're interested in more, see our tutorials on or . If you have a great idea for a different project, that's fine too. Edge Impulse lets you capture data from any sensor, build to extract features, and you have full flexibility in your Machine Learning pipeline with the learning blocks.
To showcase how easy it is to integrate Edge Impulse with the , we'll walk through a practical example using the Arduino DIN Celsius board that comes with the kit, but also a Motor to demonstrate this setup can be used interchangeably. This example demonstrates how to set up a temperature-controlled system, collect data, train a machine learning model, and deploy it for anomaly detection.
The webinar is now available on-demand .
(Wi-Fi version recommended)
Arduino IDE 2 ()
Edge Impulse account ()
Edge Impulse CLI (
Note: If you are new to Edge Impulse, please refer to our for detailed instructions.
If you are new to Edge Impulse please see our .
Read on
If you are new to Arduino inference code, see our Arduino inference code documentation for more information.
See the full Arduino OPTA for Arduino Cloud guide .
See the Blues Wireless .
This section will be updated following the Blues Wireless Webinar on the 30th of October
Later will will revisit this with C++ and Siemens PLCs, for now, you can explore the .
Please share your experiences and feedback with us on the .
Alternatively you can also capture your dataset directly through a different app, and then upload the data directly to Edge Impulse There are both options to do this visually (click the 'Upload' icon on the data acquisition screen), or via the CLI. You can find instructions here: . In this case it's highly recommended to you use square images, as the transfer learning model expects these; and you probably want to resize these images before uploading them to make sure training remains fast.
You can upload all files in a directory using the Python SDK. Note that you can set the category, label, and metadata for all files with a single call. If you want to use a different label for each file set label=None
in the function call and name your files with <label>.<name>.<ext>. For example, wave.01.csv will have the label wave when uploaded. See for more information.
For object detection, you can put bounding box information (following the ) in a file named info.labels in that same directory.
The Edge Impulse ingestion service accepts CSV files, which we can use to upload raw data. Note that if you configure a CSV template using the , then the expected format of the CSV file might change. If you do not configure a CSV template, then the ingestion service expects CSV data to be in a particular format. See .
Another way to upload data is to encode it in JSON format. See the for more information on acceptable key/value pairs. Note that at this time, the signature
value can be set to 0
.
is powerful Python library for working with large arrays and matrices. You can upload NumPy arrays directly into your Edge Impulse project. Note that the arrays are required to be in a particular format, and must be uploaded with required metadata (such as a list of labels and the sample rate).
is popular Python library for performing data manipulation and analysis. The Edge Impulse library supports a number of ways to upload dataframes. We will go over each format.
For this tutorial, you'll need a .
Alternatively, use the either or SDK to collect data from any other development board, or your .
Edge Impulse can ingest data from many sources and any device - including embedded devices that you already have in production. See the documentation for the for more information.
Alternatively, you can load an example test set that has about ten minutes of data in these classes (but how much fun is that?). See the for more information.
See the dedicated page for the pre-processing block.
See the dedicated page for the learning block.
The axes that we want to select during clustering. Click on the Select suggested axes button to harness the results of the output. Alternatively, the data separates well on the accX RMS, accY RMS and accZ RMS axes, you can also include these axes.
See the dedicated page for the learning block. We also provide the learning block that is compatible with this tutorial.
Your mobile phone can build and download the compiled impulse directly from the mobile client. See 'Deploying back to device' on the page.
Congratulations! You have used Edge Impulse to train a machine learning model capable of recognizing your gestures and understand how you can build models that classify sensor data or find anomalies. Now that you've trained your model you can integrate your impulse in the firmware of your own embedded device, see . There are examples for Mbed OS, Arduino, STM32CubeIDE, and any other target that supports a C++ compiler.
Or if you're interested in more, see our tutorials on or . If you have a great idea for a different project, that's fine too. Edge Impulse lets you capture data from any sensor, build to extract features, and you have full flexibility in your Machine Learning pipeline with the learning blocks.
You can now go back to the tutorial to build your machine learning model.
In this ML & data engineering section, you will discover useful techniques to train your models, generate synthetic datasets, or to perform advanced feature extraction:
Learn about synthetic data and how to integrate synthetic data models into your Edge Impulse project with the following guides:
The following tutorials detail how to work with synthetic datasets in Edge Impulse:
See also:
Label image data using GPT-4o
Neural networks can work with multiple types of data simultaneously, allowing for sophisticated sensor fusion techniques. Here we demonstrate how to use a single model to perform classification using two separate sets of two-dimensional (2D) input features.
For this tutorial, the two sets of input features are assumed to be spectrograms that are passed through two different convolutional branches of a neural network. The two branches of the network are then combined and passed through a final dense output layer for classification. In the general case, however, there could be more than two branches and the input features could represent different sensor data, channels, or outputs from feature extractors.
This example is for when you would like to pass sets of input features to independent branches of a neural network and then combine the results for a final classification.
If you plan on passing all input features through a fully connected network, then these steps are unnecessary; Edge Impulse automatically concatenates output features from processing blocks and learning blocks operate on the concatenated array.
By using this architecture, you can perform sophisticated sensor fusion tasks, potentially improving classification accuracy and robustness compared to single-input models.
This type of model is particularly useful in situations such as:
Combining data from different sensor types (e.g. accelerometer and gyroscope)
Handling output from multiple feature extractors (e.g. time-domain and frequency-domain features)
Analyzing data from sensors in different locations
The architecture in this tutorial consists of separating the model input into two sets of 2D input features (input layer and reshape layers), passing them through independent convolutional branches, and then combining the results for classification (concatenate layer and dense layer). See below for additional details.
The input layer accepts a one-dimensional (1D) input of length input_length
. The input_length
argument is available if you are using the learning blocks pre-built by Edge Impulse and modifying them through expert mode. By default, Edge Impulse flattens the output of all processing blocks before passing them to the learning block, so the shape is a 1D array.
The model input is first split into two halves and then reshaped back into 2D formats. In this example, we assumed spectrogram inputs with 50 columns (time frames) each. We calculated the number of rows based on the number of columns and channels, then used the row and column information for the reshape layer.
Each half of the input goes through its own convolutional branch that contains multiple layers:
Conv2D layers (8 and 16 filters)
MaxPooling2D layers for downsampling
Dropout layers for regularization
Flatten layer to prepare for concatenation
Finally, the outputs of the two independent branches are combined with a concatenation layer. This combined output is then passed through a final dense layer with a softmax activation to perform classification.
The following code snippet can be copied and pasted into a classification learning block using the expert mode feature within Studio.
To extend this model architecture to handle more inputs, you can:
Split the input into more sections (e.g. thirds or quarters instead of halves).
Create additional convolutional branches for each new input section.
Include the new branches in the concatenation step before the final dense layer.
The architecture defined in this tutorial allows the model to process two separate sets of 2D input features independently before combining them for a final classification. Each branch can learn model parameters specific to its input, which can be particularly useful when dealing with different types of sensor data or feature extractors.
Welcome to the tutorial series on OTA Model Updates with Edge Impulse Docker Deploy on Jetson Nano! In this series, we will explore how to update machine learning models over-the-air (OTA) using Edge Impulse and Docker on the Jetson Nano platform.
Before getting started, make sure you have the following prerequisites:
Jetson Nano Developer Kit
Docker installed on Jetson Nano
Edge Impulse account
Be familiar with Edge Impulse and Docker deploy
In this tutorial, we will explore how to enable GPU usage and use a camera with the Jetson Nano. We will then deploy a machine learning model using Edge Impulse and Docker on the Jetson Nano. Finally, we will update the model over-the-air (OTA) using Edge Impulse.
After installing the toolkit, restart the Docker service:
Now you can use the GPU for machine learning tasks in Docker containers.
To use a camera with Jetson Nano, you need to install the libgstreamer and libv4l libraries. Run the following commands to install the libraries:
After installing the libraries, you can use a camera with Jetson Nano.
To deploy a machine learning model with Edge Impulse and Docker, follow these steps:
Export your model from Edge Impulse as a Docker container. Copy the generated Docker command from the deployment section. Modify the Docker command to use the GPU and camera on Jetson Nano. Run the Docker command on Jetson Nano to deploy the model.
To update the model over-the-air (OTA) with Edge Impulse, follow these steps:
Train a new model in Edge Impulse. Export the new model as a Docker container. Copy the generated Docker command from the deployment section and build a new Docker image.
Run the Docker command on Jetson Nano to deploy the new model.
docker run --rm -it --privileged --runtime nvidia -v /dev/bus/usb/001/002:/dev/video0 -p 80:80 public.ecr.aws/z9b3d4t5/inference-container:73d6ea64bf931f338de5183438915dc390120d5d --api-key ei_07e1e4fad55f06b30839c062076a2ad4bbc174386330493011e75566405a5603 --run-http-server 1337
Test the new model on Jetson Nano.
Monitor the model performance and update as needed.
In this tutorial series, we explored how to update machine learning models over-the-air (OTA) using Edge Impulse and Docker on the Jetson Nano platform. We enabled GPU usage, used a camera with Jetson Nano, deployed a machine learning model, and updated the model over-the-air.
Now you can easily update your machine learning models on Jetson Nano devices using Edge Impulse and Docker.
This guide provides a quick walk-through on how to upload and update time-series data with multiple labels using the Edge Impulse API.
API Key - Obtain from your project settings on Edge Impulse.
Example Github repository - Clone the following repository, it contains the scripts and the example files:
Export your API key to use in the upload scripts:
Data File: JSON formatted time-series data. See the Data acquisition format speicification
structured_labels.labels
File: JSON with structured labels. See the specification format
Use the upload.sh
script to send your data and labels to Edge Impulse:
To update a sample, run update-sample.sh
with the required project and sample IDs:
We hope this tutorial has helped you to understand how to ingest multi-label data samples to your Edge Impulse project. If you have any questions, please reach out to us on our forum.
Python 3
Pip package manager
Jupyter Notebook: https://jupyter.org/install
pip packages (install with pip install
packagename
):
pydub https://pypi.org/project/pydub/
google-cloud-texttospeech https://cloud.google.com/python/docs/reference/texttospeech/latest
requests https://pypi.org/project/requests/
First off you will need to set up and Edge Impulse account and create your first project. You will also need a Google Cloud account with the Text to Speech API enabled: https://cloud.google.com/text-to-speech, the first million characters generated each month are free (WaveNet voices), this should be plenty for most cases as you'll only need to generate your dataset once. From google you will need to download a credentials JSON file and set it to the correct environment variable on your system to allow the python API to work: (https://developers.google.com/workspace/guides/create-credentials#service-account)
First off we need to set our desired keywords and labels:
Then we need to set up the parameters for our speech dataset, all possible combinations will be iterated through:
languages - Choose the text to speech voice languages to use (https://cloud.google.com/text-to-speech/docs/voices)
pitches - Which voice pitches to apply
genders - Which SSML genders to apply
speakingRates - Which speaking speeds to apply
Then provide some other key parameters:
out_length - How long each output sample should be
count - Maximum number of samples to output (if all combinations of languages, pitches etc are higher then this restricts output)
voice-dir - Where to store the clean samples before noise is added
noise-url - Which noise file to download and apply to your samples
output-folder - The final output location of the noised samples
num-copies - How many different noisy versions of each sample to create
max-noise-level - in Db,
Then we need to check all the output folders are ready
And download the background noise file
Then we can generate a list of all possible parameter combinations based on the input earlier. If you have set num_copies
to be smaller than the number of combinations then these options will be reduced:
Finally we iterate though all the options generated, call the Google TTS API to generate the desired sample, and apply noise to it, saving locally with metadata:
The files in ./out-noisy
can be uploaded easily using the Edge Impulse CLI tool:
Now you can use your keywords to create a robust keyword detection model in Edge Impulse Studio!
Make use of our pre-built keyword dataset to add noise and 'unknown' words to your model: Keyword Spotting Dataset
Try out both classification models and the transfer learning keyword spotting model to see which works best for your case
This notebook takes you through a basic example of using the physics simulation tool PyBullet to generate an accelerometer dataset representing dropping the Nordic Thingy:53 devkit from different heights. This dataset can be used to train a regression model to predict drop height.
This idea could be used for a wide range of simulatable environments- for example generating accelerometer datasets for pose estimation or fall detection. The same concept could be applied in an FMEA application for generating strain datasets for structural monitoring.
There is also a video version of this tutorial:
Python 3
Pip package manager
Jupyter Notebook: https://jupyter.org/install
The dependencies can be installed with:
We need to load in a Universal Robotics Description Format file describing an object with the dimensions and weight of a Nordic Thingy:53. In this case, measuring our device it is 64x60x23.5mm and its weight 60g. The shape is given by a .obj 3D model file.
To generate the required data we will be running PyBullet in headless "DIRECT" mode so we can iterate quickly over the parameter field. If you run the python file below you can see how pybullet simulates the object dropping onto a plane
First off we need to set up a pybullet physics simulation environment. We load in our object file and a plane for it to drop onto. The plane's dynamics can be adjusted to better represent the real world (in this case we're dropping onto carpet)
We also need to define the output folder for our simulated accelerometer files
And define the drop parameters
We also need to define the characteristics of the IMU on the real device we are trying to simulate. In this case the Nordic Thingy:53 has a Bosch BMI270 IMU (https://www.bosch-sensortec.com/products/motion-sensors/imus/bmi270/) which is set to a range of +-2g with a resolution of 0.06g. These parameters will be used to restrict the raw acceleration output:
Finally we are going to give the object and plane restitution properties to allow for some bounce. In this case I dropped the real Thingy:53 onto a hardwood table. You can use p.changeDynamics to introduce other factors such as damping and friction.
Here we iterate over a range of heights, randomly changing its start orientation for i number of simulations per height. The acceleration is calculated relative to the orientation of the Thingy:53 object to represent its onboard accelerometer.
Finally we save the metadata file to the output folder. This can be used to tell the edge-impulse-uploader CLI tool the floating point labels for each file.
These files can then be uploaded to a project with these commands (run in a separate terminal window):
(run edge-impulse-uploader --clean if you have used the CLI before to reset the target project)
Now you can use your dataset a drop height detection regression model in Edge Impulse Studio!
See if you can edit this project to simulate throwing the object up in the air to predict the maximum height, or add in your own custom object. You could also try to better model the real environment you're dropping the object in- adding air resistance, friction, damping and material properties for your surface.
Synthetic datasets are a collection of data artificially generated rather than being collected from real-world observations or measurements. They are created using algorithms, simulations, or mathematical models to mimic the characteristics and patterns of real data. Synthetic datasets are a valuable tool to generate data for experimentation, testing, and development when obtaining real data is challenging, costly, or undesirable.
You might want to generate synthetic datasets for several reasons:
Cost Efficiency: Creating synthetic data can be more cost-effective and efficient than collecting large volumes of real data, especially in resource-constrained environments.
Data Augmentation: Synthetic datasets allow users to augment their real-world data with variations, which can improve model robustness and performance.
Data Diversity: Synthetic datasets enable the inclusion of uncommon or rare scenarios, enriching model training with a wider range of potential inputs.
Privacy and Security: When dealing with sensitive data, synthetic datasets provide a way to train models without exposing real information, enhancing privacy and security.
You can generate synthetic data directly from Edge Impulse using the Synthetic Data tab in the Data acquisition view. This tab provides a user-friendly interface to generate synthetic data for your projects. You can create synthetic datasets using a variety of tools and models.
We have put together the following tutorials to help you get started with synthetic datasets generation:
DALL-E Image Generation Block: Generate image datasets using Dall·E using the DALL-E model.
Whisper Keyword Spotting Generation Block: Generate keyword-spotting datasets using the Whisper model. Ideal for keyword spotting and speech recognition applications.
Eleven Labs Sound Generation Block: Generate sound datasets using the Eleven Labs model. Ideal for generating realistic sound effects for various applications.
Note that you will need an API Key/Access Token from the different providers to run the model used to generate the synthetic data.
If you want to create your own synthetic data block, see Custom synthetic data blocks.
Generate image datasets using Dall·E (Jupyter Notebook and Transformation block source code available).
Generate keyword-spotting datasets (Jupyter Notebook source code available).
Generate physics simulation datasets (Jupyter Notebook source code available).
Generate audio data using the Eleven Labs Sound Effects models. This integration allows you to generate realistic sound effects for your projects, such as glass breaking, car engine revving, or other custom sounds. You can customize the sound prompts and generate high-quality audio samples for your datasets.
This integration allows you to expand your datasets with sounds that may be difficult or expensive to record naturally. This approach not only saves time and money but also enhances the accuracy and reliability of the models we deploy on edge devices.
In this tutorial, we focus on a practical application that can be used in a smart security system, or in a factory to detect incidents, such as detecting the sounds of glass breaking.
There is also a video version of this guide:
Only available with Edge Impulse Pro Plan and Enterprise Plan
Try our FREE Enterprise Trial today.
You will also need an Eleven Labs account and API Key.
Navigate to Data Acquisition: Once you're in your project, navigate to the Data Acquisition section, go to Synthetic data and select the ElevenLabs Synthetic Audio Generator data source.
First, get your Eleven Labs API Key. Navigate to the Eleven Labs web interface to get your key and optionally test your prompt.
Here we will be trying to collect a glass-breaking sound or impact.
Prompt: "glass breaking"
Simple prompts are just that: they are simple, one-sided prompts where we try to get the AI to generate a single sound effect. This could be, for example, “person walking on grass” or “glass breaking.” These types of prompts will generate a single type of sound effect with a few variations either in the same generation or in subsequent generations. All in all, they are fairly simple.
There are a few ways to improve these prompts, however, and that is by adding a little bit more detail. Even if they are simple prompts, they can be made to give better output by improving the prompt itself. For example, something that sometimes works is adding details like “high-quality, professionally recorded footsteps on grass, sound effects foley.” It can require some experimentation to find a good balance between being descriptive and keeping it brief enough to have AI understand the prompt. e.g. high quality audio of window glass breaking
Label: The label of the generated audio sample.
Prompt influence: Between 0 and 1, this setting ranges from giving the AI more creativity in how it interprets the prompt to telling the AI to be more strict in following the exact prompt that you’ve given. 1 being more creative.
Number of samples: Number of samples to generate
Minimum length (seconds): Minimum length of generated audio samples. Audio samples will be padded with silence to minimum length. It also determines how long your generations should be. Depending on what you set this as, you can get quite different results. For example, if I write “kick drum” and set the length to 11 seconds, I might get a full drum loop with a kick drum in it, but that might not be what I want. On the other hand, if I set the length to 1 second, I might just get a one-shot with a single instance of a kick drum.
Frequency (Hz): Audio frequency, ElevenLabs generates data at 44100Hz, so any other value will be resampled.
Upload to category: Data will be uploaded to this category in your project.
See ElevenLabs API documentation for more information
Once you've set up your prompt, and api key, run the pipeline to generate the sound samples. You can then view the output in the Data Acquisition section.
Enhance Data Quality: Generative AI can create high-quality sound samples that are difficult to record naturally.
Increase Dataset Diversity: Access a wide range of sounds to enrich your training dataset and improve model performance.
Save Time and Resources: Quickly generate the sound samples you need without the hassle of manual recording.
Improve Model Accuracy: High-quality, diverse sound samples can help fill gaps in your dataset and enhance model performance.
By leveraging generative AI for sound generation, you can enhance the quality and diversity of your training datasets, leading to more accurate and reliable edge AI models. This innovative approach saves time and resources while improving the performance of your models in real-world applications. Try out the Eleven Labs block in Edge Impulse today and start creating high-quality sound datasets for your projects.
This notebook explores how we can use generative AI to create datasets which don't exist yet. This can be a good starting point for your project if you have not collected or cannot collect the data required. It is important to note the limitations of generative AI still apply here, biases can be introduced through your prompts, results can include "hallucinations" and quality control is important.
This example uses the OpenAI API to call the Dall-E image generation tool, it explores both generation and variation but there are other tools such as editing which could also be useful for augmenting an existing dataset.
There is also a video version of this tutorial:
We have wrapped this example into a Transformation Block (Enterprise Feature) to make it even easier to generate images and upload them to your organization. See: https://github.com/edgeimpulse/example-transform-Dall-E-images
Python 3
Pip package manager
Jupyter Notebook: https://jupyter.org/install
pip packages (install with pip install
packagename
):
openai https://pypi.org/project/openai/
First off you will need to set up and Edge Impulse account and create your first project.
You will also need to create an API Key for OpenAI: https://platform.openai.com/docs/api-reference/authentication
The API takes in a prompt, number of images and a size
The API also has a variations call which takes in an existing images and creates variations of it. This could also be used to modify existing images.
Here we are iterate through a number of images and variations to generate a dataset based on the prompts/labels given.
These files can then be uploaded to a project with these commands (run in a separate terminal window):
(run edge-impulse-uploader --clean if you have used the CLI before to reset the target project)
Now you can use your images to create an image classification model on Edge Impulse.
Why not try some other OpenAI calls, 'edit' could be used to take an existing image and translate it into different environments or add different humans to increase the variety of your dataset. https://platform.openai.com/docs/guides/images/usage
This example comes from the Edge Impulse Linux Inferencing Python SDK that has been slightly modify to upload the raw data back to Edge Impulse based on the inference results.
To run the example:
Clone this repository:
Install the dependencies:
Grab your the API key of the project you want to upload the inferred results raw data:
Past the new key in the EI_API_KEY
variable in the audio-classify-export.py
file. Alternatively, load it from your environment variable:
Download your modelfile.eim:
Run the script:
Here are the arguments you can set:
modelfile.eim
, path the model.eim
yes,no
, labels to upload, separated by comas, no space
0.6
, low confidence threshold
0.8
, high confidence threshold
<audio_device_ID, optional>
In a keyword spotting model, it can give the following results:
Sensor fusion is about combining data from various sensors to gain a more comprehensive understanding of your environment. In this tutorial, we will demonstrate sensor fusion by bringing together high-dimensional audio or image data with time-series sensor data. This combination allows you to extract deeper insights from your sensor data.
This is an advanced tutorial where you will need to parse your dataset to create multi-sensor data samples, train several Edge Impulse project in order to extract the embeddings from the tflite
models, create custom DSP blocks and, finally, modify the C++ inferencing SDK.
If you are looking for a more beginner-level tutorial, please head to the Sensor Fusion tutorial.
Multi-impulse vs multi-model vs sensor fusion
Running multi-impulse refers to running two separate projects (different data, different DSP blocks and different models) on the same target. It will require modifying some files in the EI-generated SDKs.
Running multi-model refers to running two different models (same data, same DSP block but different tflite models) on the same target. See how to run a motion classifier model and an anomaly detection model on the same device in this tutorial.
Sensor fusion refers to the process of combining data from different types of sensors to give more information to the neural network. To extract meaningful information from this data, you can use the same DSP block, multiples DSP blocks, or use neural networks embeddings like we will see in this tutorial.
Also, see this video (starting min 13):
When you have data coming from multiple sources, such as a microphone capturing audio, a camera capturing images, and sensors collecting time-series data. Integrating these diverse data types can be tricky and conventional methods fall short.
With the standard workflow, if you have data streams from various sources, you might want to create separate DSP blocks for each data type. For instance, if you're dealing with audio data from microphones, image data from cameras, and time-series sensor data from accelerometers, you could create separate DSP blocks for each. For example:
A spectrogram-based DSP block for audio data
An image DSP block for image data
A spectral analysis block for time-series sensor data
This approach initially seems logical but comes with limitations:
When using separate DSP blocks, you're constrained in your choice of neural networks. The features extracted from each data type are fundamentally different. For example, a pixel in an image or an image's spectrogram and a data point from an accelerometer's time-series data have distinct characteristics. This incompatibility makes it challenging to use a convolutional neural network (CNN) that is typically effective for image data or spectrogram. As a result, fully connected networks may be your only option, which are not ideal for audio or image data.
To bypass the limitation stated above, you may consider using neural networks embeddings. In essence, embeddings are compact, meaningful representations of your data, learned by a neural network.
Embeddings are super powerful, we use them for various features of Edge Impulse, such as the Data Explorer, the Auto Labeler or in this advanced sensor fusion tutorial.
While training the neural network, the model try to find the mathematical formula that best maps the input to the output. This is done by tweaking each neuron (each neuron is a parameter in our formula). The interesting part is that each layer of the neural network will start acting like a feature extracting step but highly tuned for your specific data.
Finally, instead of having a classifier for last layer (usually a softmax
layer), we cut the neural network somewhere at the end and we obtained the embeddings.
Thus, we can consider the embeddings as learnt features and we will pass these "features" to the final Impulse:
Here's how we approach advanced sensor fusion with Edge Impulse.
In this workflow, we will show how to perform sensor fusion using both audio data and accelerometer data to classify different stages of a grinding coffee machine (grind
, idle
, pump
and extract
). First, we are going to use a spectrogram DSP block and a NN classifier using two dense network. This first impulse will then be used to generate the embeddings and will be made available in a custom DSP block. Finally, we are going to train a fully connected layer using features coming from both the generated embeddings and a spectral feature DSP block.
We have develop two Edge Impulse public projects, one publicly available dataset and a Github repository containing the source code to help you follow the steps:
Dataset: Coffee Machine Stages
Edge Impulse project 1 (used to generate the embeddings): Audio Sensor Fusion - Step 1
Edge Impulse project 2 (final impulse): Audio Sensor Fusion - Step 2
Github repository containing the source code: Sensor fusion using NN Embeddings
Please note that with a few changes, you will be able to change the sensor type (audio to images) or the first pre-processing method (spectrogram to MFE/MFCC).
The first step is to have input data samples that contain both sensors. In Edge Impulse studio, you can easily visualize time-series data, like audio and accelerometer data.
Note: it is not trivial to group together images and time-series. Our core-engineering team is working on improving this workflow. In the meantime, as a workaround, you can encode your image as time-series with one axis per channel (red, green, blue) plus the sensor:
Train separate projects for high dimensional data (audio or image data). Each project contains both a DSP block and a trained neural network.
See Audio Sensor Fusion - Step 1
Clone this repository:
Download the generated Impulse to extract the embeddings, which encapsulate distilled knowledge about their respective data types.
Download Model: From the project dashboard, download the TensorFlow SavedModel (saved_model
). Extract the save_model directory and place it under the /input
repository.
Download Test Data: From the same dashboard, download the test or train data NPY file (input.npy
). Place this numpy array file under the /input
repository. This will allow us to generate a quantized version of the tflite embeddings. Ideally choose the test data if you have some data available.
Generate the embeddings:
This will cut off the last layer of the neural network and convert it to LiteRT (previously Tensorflow Lite) (TFLite) format. You can follow the process outlined in the saved_model_to_embeddings.py
script for this conversion for a better understanding.
To make sensor fusion work seamlessly, Edge Impulse enables you to create custom DSP blocks. These blocks combine the necessary spectrogram/image-processing and neural network components for each data type.
Custom DSP Block Configuration: In the DSP block, perform two key operations as specified in the dsp.py
script:
Run the DSP step with fixed parameters.
Run the neural network.
Replace this following lines in dsp-blocks/features-from-audio-embeddings/dsp.py
to match your DSP configuration:
If you want to use another DSP block than the spectrogram one, all the source code of the available DSP code can be found in this public repository: processing-blocks
Return Neural Network Embeddings: The DSP block should be configured to return the neural network embeddings, as opposed to the final classification result.
Implement get_tflite_implementation: Ensure that the get_tflite_implementation
function returns the TFLite model. Note that the on-device implementation will not be correct initially when generating the C++ library, as only the neural network part is compiled. We will fix this in the final exported C++ Library.
Now publish your new custom DSP block.
Fill the necessary information and push your block:
During development, it might be easier to host the block locally so you can make changes, see Custom processing blocks
Multiple DSP Blocks: Create a new impulse with three DSP blocks and a classifier block. The routing should be as follows:
Audio data routed through the custom block.
Sensor data routed through spectral analysis.
See Audio Sensor Fusion - Step 2
Training the Model: Train the model within the new impulse, using a fully-connected network.
Export as a C++ Library:
In the Edge Impulse platform, export your project as a C++ library.
Choose the model type that suits your target device (quantized
vs. float32
).
Make sure to select EON compiler option
Copy the exported C++ library to the example-cpp
folder for easy access.
Add a Forward Declaration:
In the model-parameters/model_variables.h
file of the exported C++ library, add a forward declaration for the custom DSP block you created.
For example:
And change &extract_tflite_eon_features
into &custom_sensor_fusion_features
in the ei_dsp_blocks
object.
Implement the Custom DSP Block:
In the main.cpp
file of the C++ library, implement the custom_sensor_fusion_features block. This block should:
Call into the Edge Impulse SDK to generate features.
Execute the rest of the DSP block, including neural network inference.
For example, see the main.cpp file in the Github repository
Copy a test sample's raw features into the features[]
array in source/main.cpp
Enter make -j
in this directory to compile the project. If you encounter any OOM memory error try make -j4
(replace 4 with the number of cores available)
Enter ./build/app
to run the application
Compare the output predictions to the predictions of the test sample in the Edge Impulse Studio.
Note that if you are using the quantized version of the model, you may encounter a slight difference between the Studio Live Classification page and the above results, the float32 model however should give you the same results.
Congratulations on successfully completing this advanced tutorial. You have been through the complex process of integrating high-dimensional audio or image data with time-series sensor data, employing advanced techniques like custom DSP blocks, neural network embeddings, and modifications to the C++ inferencing SDK. Also, note that you can simplify this workflow using custom deployment blocks to generate the custom DSP block with the embeddings.
If you are interested in using it for an enterprise project, please sign up for our FREE Enterprise Trial and our solution engineers can work with you on the integration.
Extracting meaningful features from your data is crucial to building small and reliable machine learning models, and in Edge Impulse this is done through processing blocks. We ship a number of processing blocks for common sensor data (such as vibration and audio), but they might not be suitable for all applications. Perhaps you have a very specific sensor, want to apply custom filters, or are implementing the latest research in digital signal processing. In this tutorial you'll learn how to support these use cases by adding custom processing blocks to the studio.
There is also a complete video covering how to implement your custom DSP block:
Make sure you follow the Continuous motion recognition tutorial, and have a trained impulse.
Development flow
This tutorial shows you the development flow of building custom processing blocks, and requires you to run the processing block on your own machine or server. Enterprise customers can share processing blocks within their organization, and run these on our infrastructure. See custom processing blocks for more details.
Processing blocks take data and configuration parameters in, and return features and visualizations like graphs or images. To communicate to custom processing blocks, Edge Impulse studio will make HTTP calls to the block, and then use the response both in the UI, while generating features, or when training a machine learning model. Thus, to load a custom processing block we'll need to run a small server that responds to these HTTP calls. You can write this in any language, but we have created an example in Python. To load this example, open a terminal and run:
This creates a copy of the example project locally. Then, you can run the example either through Docker or locally via:
Docker
Locally
Then go to http://localhost:4446 and you should be shown some information about the block.
As this block is running locally the studio cannot reach the block. To resolve this we can use ngrok which can make a local port accessible from a public URL. After you've finished development you can move the processing block to a server with a publicly accessible address (or run it on our infrastructure through your enterprise account). To set up a tunnel:
Sign up for ngrok.
Install the ngrok binary for your platform.
Get a URL to access the processing block from the outside world via:
This yields a public URL for your block under Forwarding
. Note down the address that includes https://
.
Now that the custom processing block was created, and you've made it accessible to the outside world, you can add this block to Edge Impulse. In a project, go to Create Impulse, click Add a processing block, choose Add custom block (in the bottom left corner of the modal), and paste in the public URL of the block:
After you click Add block the block will show like any other processing block.
Add a learning bloc, then click Save impulse to store the impulse.
Processing blocks have configuration options which are rendered on the block parameter page. These could be filter configurations, scaling options, or control which visualizations are loaded. These options are defined in the parameters.json
file. Let's add an option to smooth raw data. Open example-custom-processing-block-python/parameters.json
and add a new section under parameters
:
Then, open example-custom-processing-block-python/dsp.py
and replace its contents with:
Restart the Python script, and then click Custom block in the studio (in the navigation bar). You now have a new option 'Smooth'. Every time an option changes we'll re-run the block, but as we have not written any code to respond to these changes nothing will happen.
For the full documentation on customizing parameters, and a list of all configuration options; see parameters.json.
To show the user what is happening we can also draw visuals in the processing block. Right now we support graphs (linear and logarithmic) and arbitrary images. By showing a graph of the smoothed sample we can quickly identify what effect the smooth option has on the raw signal. Open dsp.py
and replace the content with the following script. It contains a very basic smoothing algorithm and draws a graph:
Restart the script, and click the Smooth toggle to observe the difference. Congratulations! You have just created your first custom processing block.
If you extract set features from the signal, like the mean, that you that return, you can also label these features. These labels will be used in the feature explorer. To do so, add a labels
array that contains strings that map back to the features you return (labels
and features
should have the same length).
In the previous step we drew a linear graph, but you can also draw logarithmic graphs or even full images. This is done through the type
parameter:
This draws a graph with a logarithmic scale:
To show an image you should return the base64 encoded image and its MIME type. Here's how you draw a small PNG image:
If you output high-dimensional data (like a spectrogram or an image) you can enable dimensionality reduction for the feature explorer. This will run UMAP over the data to compress the features into three dimensions. To do so, set:
On the info
object in parameters.json
.
For all options that you can return in a graph, see the Run DSP return types in the API documentation.
Your custom block behaves exactly the same as any of the built-in blocks. You can process all your data, train neural networks or anomaly blocks, and validate that your model works.
However, we cannot automatically generate optimized native code for the block, like we do for built-in processing blocks, but we try to help you write this code as much as possible.
Export as a C++ Library:
In the Edge Impulse platform, export your project as a C++ library.
Choose the model type that suits your target device (quantized
vs. float32
).
Forward Declaration:
You don't need to add this part, it is automatically generated!
In the model-parameters/model_variables.h
file of the exported C++ library, you can see a forward declaration for the custom DSP block you created.
For example:
The name of that function comes from the cppType
field in your custom DSP parameter.json
. It takes your {cppType}
and generates the following extract_{cppType}_features
function.
Implement the Custom DSP Block:
In the main.cpp
file of the C++ library, implement the extract_my_preprocessing_features
block. This block should:
Call into the Edge Impulse SDK to generate features.
Execute the rest of the DSP block, including neural network inference.
For examples, have a look at our official DSP blocks implementations in our Inferencing C++ SDK
Also, please have a look at the video on the top of this page (around minute 25) where Jan explains how to implement your custom DSP block with your C++ library.
Compile and Run the App
Copy a test sample's raw features into the features[]
array in source/main.cpp
Enter make -j
in this directory to compile the project. If you encounter any OOM memory error try make -j4
(replace 4 with the number of cores available)
Enter ./build/app
to run the application
Compare the output predictions to the predictions of the test sample in the Edge Impulse Studio.
Blog post: Utilize Custom Processing Blocks in Your Image ML Pipelines
With good feature extraction you can make your machine learning models smaller and more reliable, which are both very important when you want to deploy your model on embedded devices. With custom processing blocks you can now develop new feature extraction pipelines straight from Edge Impulse. Whether you're following the latest research, want to implement proprietary algorithms, or are just exploring data.
For inspiration we have published all our own blocks here: edgeimpulse/processing-blocks. If you've made an interesting block that you think is valuable for the community, please let us know on the forums or by opening a pull request. We'd be happy to help write efficient native code for the block, and then publish it as a standard block!
This is the format for the parameters.json
file:
\
At Edge Impulse, we recognize that the lifecycle of your impulse is dynamic. As data grows, unanticipated factors, or drift occurs retraining and redeployment becomes essential. Many of our partners have already starting to address this with integrations to our platform, or documenting details for implementation on aspects like OTA updates to your impulse, and Lifecycle Management. We have put together this section to help you understanding and explore how to create your own implementation of a Lifecycle Management system.
MLOps is a set of practices that combines Machine Learning, DevOps, and Data Engineering. The goal of MLOps is to streamline and automate the machine learning lifecycle, including integration, testing, releasing, deployment, and infrastructure management.\
Continuous Learning is a key concept in the domain of Machine Learning Operations (MLOps), which is a set of practices that combines Machine Learning, DevOps, and Data Engineering. Here is an example of the process:
In this section we will explore how firmware updates and other scenarios are currently addressed, with traditional OTA. It should help you to get started planning your own updated impulse across a range of platforms. Starting with platform-specific examples like Arduino Cloud, Nordic nRF Connect SDK, Zephyr, and Golioth, Particle Workbench and Blues Wireless.
Finally we will explore building an end-to-end example on the Espressif IDF. By covering a cross section of platforms we hope to provide a good overview of the process and how it can be applied to your own project.
With more generic examples like Arduino, Zephyr and C++ which can be applicable to all other vendors.
Edge AI solutions are typically not just about deploying once; it’s about building a Lifecycle Management ecosystem. You can configure your device to send labeled data back to Edge Impulse for ongoing model refinement, and leverage our version control to track your model performance over time.
This bidirectional data flow can be established with a straightforward call to our ingestion API you can explore how to collect data from your board in the following tutorial:
By integrating these elements, you establish an Lifecycle Management cycle, where the impulse is not static but evolves, learns, and adapts. This adaptation is can be as simple as adding new data to the existing model, or as complex as retraining the model with new data and deploying a new model to the device. Based on metrics you can define, you can trigger this process automatically, or manually. In the esp-idf example, we will explore how to trigger this process manually, and conditionally based on metrics.
When you are classifying audio - for example to detect keywords - you want to make sure that every piece of information is both captured and analyzed, to avoid missing events. This means that your device need to capture audio samples and analyze them at the same time. In this tutorial you'll learn how to continuously capture audio data, and then use the continuous inferencing mode in the Edge Impulse SDK to classify the data.
This tutorial assumes that you've completed the tutorial, and have your impulse running on your device.
Continuous inference mode
Continuous inferencing is automatically enabled for any impulses that use audio. Build and flash a ready-to-go binary for your development board from the Deployment tab in the studio, then - from a command prompt or terminal window - run edge-impulse-run-impulse --continuous
.
An Arduino sketch that demonstrates continuous audio sampling is part of the Arduino library deployment option. After importing the library into the Arduino IDE, look under 'Examples' for 'nano_ble33_sense_audio_continuous'.
In the normal (non-continuous) inference mode when classifying data you sample data until you have a full window of data (e.g. 1 second for a keyword spotting model, see the Create impulse tab in the studio), you then classify this window (using the run_classifier
function), and a prediction is returned. Then you empty the buffer, sample new data, and run the inferencing again. Naturally this has some caveats when deploying your model in the real world: 1) you have a delay between windows, as classifying the window takes some time and you're not sampling then, making it possible to miss events. 2) there's no overlap between windows, thus if an event is at the very end of the window, not the full event might be captured - leading to a wrong classification.
To mitigate this we have added several new features to the Edge Impulse SDK.
Using continuous inferencing, smaller sampling buffers (slices) are used and passed to the inferencing process. In the inferencing process, the buffers are time sequentially placed in a FIFO (First In First Out) buffer that matches the model size. After each iteration, the oldest slice is removed at the end of the buffer and a new slice is inserted at the beginning. On each slice now, the inference is run multiple times (depending on the number of slices used for a model). For example, a 1-second keyword model with 4 slices (each 250 ms), will infer each slice 4 times. So if now the keyword is on 2 edges of the slice buffers, they're glued back together in the FIFO buffer and the keyword will be classified correctly.
Another advantage of this technique is that it filters out false positives. Take for instance a yes-no keyword spotting model. The word 'yesterday' should not be classified as a yes (or no). But if the 'yes-' is sampled in the first buffer and '-terday' in the next, there is a big chance that the inference step will classify the first buffer as a yes.
By running inference multiple times over the slices, continuous inferencing will filter out this false positive. When the 'yes' buffer enters the FIFO it will surely classify as a 'yes'. But as the rest of the word enters, the classified value for 'yes' will drop quickly. We just have to make sure that we don't react on peak values. Therefore a moving average filter averages the classified output and so flattens the peaks. To have a valid 'yes', we now need multiple high-rated classifications.
In the standard way of running the impulse, the steps of collecting data and running the inference are run sequentially. First, the audio is sampled, filling a block the size of the model. This block is sent to the inferencing part, where first the features are extracted and then the inference is run. Finally, the classified output is used in your application (by default the output will be printed over the serial connection).
In the continuous sampling method, audio is sampled in parallel with the inferencing and output steps. So while inference is running, audio sampling continues on a background process.
The embedded target needs to support running of multiple processes in parallel. This can either be achieved by an operating system; 1 low priority thread will run inferencing and 1 high priority thread will collect sample data. Or the processor should support processor offloading. This is usually done by the audio peripheral or DMA (Direct Memory Access). Here audio samples are collected in a buffer without involvement of the processor.
How do we know when new sample data is available? For this we use a double buffering mechanism. Hereby 2 sample buffers are used:
1 buffer for the audio sampling process, filling the buffer with new sample data
1 buffer for the inference process, get sample data out the buffer, extract the features and run inference
At start, the sampling process starts filling a buffer with audio samples. Meanwhile, the inference process waits until the buffer is full. When that happens, the sampling process passes the buffer to the inference process and starts sampling on the second buffer. Each iteration, the buffers will be switched so that there is always an empty buffer for sampling and a full buffer of samples for inferencing.
There are 2 constraints in this story: timing and memory. When switching the buffers there must be a 100% guarantee that the inference process is finished when the sampling process passes a full buffer. If not, the sampling process overruns the buffer and sampled data will get lost. When that happens on the ST B-L475E-IOT01A or the Arduino Nano 33 BLE Sense target, running the impulse is aborted and the following error is returned:
The EI_CLASSIFIER_SLICES_PER_MODEL_WINDOW
macro is used to set the number of slices to fill the complete model window. The more slices per model, the smaller the slice size, thereby the more inference cycles on the sampled data. Leading to more accurate results. The sampling process uses this macro for the buffer size. Where following rule applies: the bigger the buffer, the longer the sampling cycle. So on targets with lower processing capabilities, we can increase this macro to meet the timing constraint.
Increasing the slice size, increases the volatile memory uses times 2 (since we use double buffering). On a target with limited volatile memory this could be a problem. In this case you want the slice size to be small.
On both the ST B-L475E-IOT01A and Arduino Nano 33 BLE Sense targets the audio sampling process calls the audio_buffer_inference_callback()
function when there is data. Here the number of samples (inference.n_samples
) are stored in one of the buffers. When the buffer is full, the buffers are switched by toggling inference.buf_select
. The inference process is signaled by setting the flag inference.buf_ready
.
The inferencing process then sets the callback function on the signal_t
structure to reference the selected buffer:
In the section, you will discover useful techniques to leverage our inferencing libraries or how you can use the inference results in your application logic:
Once you successfully trained or imported a model, you can use Edge Impulse to download a C++ library that bundles both your signal processing and your machine learning model. Until recently, we could only run one impulse on MCUs.
Feature under development
Please note that this method is still under integration in the studio and has . This tutorial is for advanced users only. Thus, we will provide limited support on the forum until the integration is completed. If you have subscribed to an Enterprise Plan, you can contact our customer success or solution engineering team.
In this tutorial, we will see how to run multiple impulses using the downloaded C++ libraries of two different projects.
We have put together a that will automate all the processes and provide a C++ library that can be compiled and run as a standalone.
In this page, we will explain the high level concepts of how to merge two impulses. Feel free to look at the code to gain a deeper understanding.
Multi-impulse vs multi-model vs sensor fusion
Running multi-impulse refers to running two separate projects (different data, different DSP blocks and different models) on the same target. It will require modifying some files in the EI-generated SDKs.
Running multi-model refers to running two different models (same data, same DSP block but different tflite models) on the same target. See how to run a motion classifier model and an anomaly detection model on the same device in .
Sensor fusion refers to the process of combining data from different types of sensors to give more information to the neural network. See how to use sensor fusion in .
Also see this video (starting min 13):
Make sure you have at least two impulses fully trained.
You can use one of the following examples:
By default, the quantized version is used when downloading the C++ libraries. To use float32, add the option --float32
as an argument.
Similarly by default the EON compiled model is used, if you want to use full tflite then add the option --full-tflite
and be sure to include a recent version of tensorflow lite compiled for your device architecture in the root of your project in a folder named tensorflow-lite
If you need a mix of quantized and float32, you can look at the dzip.download_model
function call in generate.py and change the code accordingly.
By default, the block will download cached version of builds. You can force new builds using the --force-build
option.
Retrieve API Keys of your projects and run the generate.py command as follows:
python generate.py --out-directory output --api-keys ei_0b0e...,ei_acde... --quantization-map <0/1>,<0/1>
Build the container: docker build -t multi-impulse .
Then run: docker run --rm -it -v $PWD:/home multi-impulse --api-keys ei_0b0e...,ei_acde...
Initialize the custom block - select Deployment block and Library when prompted: edge-impulse-blocks init
Push the block: edge-impulse-blocks push
Then go your Organization and Edit the deployment block with:
CLI arguments: --api-keys ei_0b0e...,ei_acde...
Privaliged mode: Enabled
If you have a look at the generate.py
script, it streamline the process of generating a C++ library from multiple impulses through several steps:
Library Download and Extraction:
If the script detects that the necessary projects are not already present locally, it initiates the download of C++ libraries required for edge deployment. These libraries are fetched using API keys provided by the user.
Libraries are downloaded and extracted into a temporary directory. If the user specifies a custom temporary directory, it's used; otherwise, a temporary directory is created.
Customization of Files:
For each project's library, the script performs several modifications:
At the file name level:
It adds a project-specific suffix to certain patterns in compiled files within the tflite-model
directory. This customization ensures that each project's files are unique.
Renamed files are then copied to a target directory, mainly the first project's directory.
At the function name level:
It edits model_variables.h
functions by adding the project-specific suffix to various patterns. This step ensures that model parameters are correctly associated with each project.
Merging the projects
model_variables.h
is merged into the first project's directory to consolidate model information.
The script saves the intersection of lines between trained_model_ops_define.h
files for different projects, ensuring consistency.
Copying Templates:
Generating Custom Code:
The script retrieves impulse IDs from model_variables.h
for each project. Impulses are a key part of edge machine learning models.
Custom code is generated for each project, including functions to get signal data, define raw features, and run the classifier.
This custom code is inserted into the main.cpp
file of each project at specific locations.
Archiving for Deployment:
Finally, the script archives the target directory, creating a zip file ready for deployment. This zip file contains all the customized files and code necessary for deploying machine learning models on edge devices.
When changing between projects and running generate.py
locally:
You may need to include the --force-build
option to ensure correctness of the combined library.
Now to test the library generated:
Download and unzip your Edge Impulse C++ multi-impulse library into a directory
Copy a test sample's raw features into the features[]
array in source/main.cpp
Enter make -j
in this directory to compile the project. If you encounter any OOM memory error try make -j4
(replace 4 with the number of cores available)
Enter ./build/app
to run the application
Compare the output predictions to the predictions of the test sample in the Edge Impulse Studio
Want to add your own business logic?
You can change the template you want to use in step 4 to use another compilation method, implement your custom sampling strategy and how to handle the inference results in step 5 (apply post-processing, send results somewhere else, trigger actions, etc.).
General limitations:
The custom ML accelerator deployments are unlikely to work (TDA4VM, DRPAI, MemoryX, Brainchip).
The custom tflite kernels (ESP NN, Silabs MVP, Arc MLI) should work - but may require some additional work. I.e: for ESP32 you may need to statically allocate arena for the image model.
In general, running multiple impulses on an MCU can be challenging due to limited processing power, memory, and other hardware constraints. Make sure to thoroughly evaluate the capabilities and limitations of your specific MCU and consider the resource requirements of the impulses before attempting to run them concurrently.
Use case specific limitations:
The model_metadata.h
comes from the first API Key of your project. This means some #define
statement might be missing or conflicting.
Object detection: If you want to run at least one Object Detection project. Make sure to use this project API KEY first! This will set the #define EI_CLASSIFIER_OBJECT_DETECTION 1
and eventually the #define EI_HAS_FOMO 1
. Note that you can overwrite them manually but it requires an extra step.
Anomaly detection: If your anomaly detection model API Key is not in the first position, the model-parameter/anomaly_metadata.h
file will not be included.
Visual anomaly detection AND time-series anomaly detection (K-Means or GMM): It is currently not possible to combine two different anomaly detection models. The #define EI_CLASSIFIER_HAS_ANOMALY
statement expect ONLY one of the following argument:
If you see an error like the following, you probably used twice the same API Key:
Make sure you use distinct projects.
When we first wrote this tutorial, we explained how to merge two impulses manually; This process is now deprecated due to recent changes in our C++ SDK, some files and functions may have been renamed.
The Edge Impulse object detection model (FOMO) is effective at classifying objects and very lightweight (can run on MCUs). It does not however have any object persistence between frames. One common use of computer vision is for object counting- in order to achieve this you will need to add in some extra logic when deploying.
This notebook takes you through how to count objects using the linux deployment block (and provides some pointers for how to achieve similar logic other firmware deployment options).
Relevant links:
Raw python files for the linux deployment example:
An end-to-end demo for on-device deployment of object counting:
To run your model locally, you need to deploy to a linux target in your project. First you need to enable all linux targets. Head to the deployment screen and click "Linux Boards" then in the following pop-up select "show all Linux deployment options on this page":
Then download the linux/mac target which is relevant to your machine:
Finally, follow the instructions shown as a pop-up to make your .eim file executable (for example for MacOS):
Open a terminal window and navigate to the folder where you downloaded this model.
Mark the model as executable: chmod +x path-to-model.eim
Remove the quarantine flag: xattr -d com.apple.quarantine ./path-to-model.eim
Ensure you have these libraries installed before starting:
(see next heading for running on a webcam)
This program will run object detection on an input video file and count the objects going upwards which pass a threshold (TOP_Y). The sensitivity can be tuned with the number of columns (NUM_COLS) and the DETECT_FACTOR which is the factor of width/height of the object used to determine object permanence between frames.
Ensure you have added the relevant paths to your model file and video file:
modelfile = '/path/to/modelfile.eim'
videofile = '/path/to/video.mp4'
This program will run object detection on a webcam port and count the objects going upwards which pass a threshold (TOP_Y). The sensitivity can be tuned with the number of columns (NUM_COLS) and the DETECT_FACTOR which is the factor of width/height of the object used to determine object permanence between frames.
Ensure you have added the relevant paths to your model file and video file:
modelfile = '/path/to/modelfile.eim'
[OPTIONAL] camera_port = '/camera_port'
While running object counting on linux hardware is fairly simple, it would be more useful to be able to deploy this to one of the firmware targets. This method varies per target but broadly speaking it is simple to add the object counting logic into existing firmware.
Here are the main steps:
This can be found on our github pages e.g. https://github.com/edgeimpulse/firmware-arduino-nicla-vision
You'll need to replace the "edge-impulse-sdk", "model-parameters" and "tflite-model" folders within the cloned firmware with the ones you've just downloaded for your model.
This will be in a .h or similar file somewhere in the firmware. Likely in the ei_image_nn.h file. It can be found by searching for these lines:
The following lines must be added into the logic in these files (For code itself see below, diff for clarity). Firstly these variables must be instantiated:
Then this logic must be inserted into the bounding box printing logic here:
Full code example for nicla vision (src/inference/ei_run_camera_impulse.cpp):
Follow the instructions in the README.md file for the firmware repo you have been working in.
Use the command below to see on-device inference (follow the local link to see bounding boxes and count output in the browser)
In this tutorial, we will see how Grad-CAM can help you visualize neural network decisions. The output of this tutorial will be a Grad-CAM overlay of your test dataset.
Example of a “car vs not car” model, highlighting the learned “features” of a car:
This tutorial will use a custom deployment block that you can add into your Organization (Enterprise only). This will make the Grad-CAM overlay output available in all your organization's projects.
Note: This currently works for image classification and visual regression models where at least one 2D convolution layer is present in your architecture.
Jupyter notebook:
Grad-CAM (Gradient-weighted Class Activation Mapping) is a technique that helps interpret the predictions of a convolutional neural network (CNN) by visualizing the regions of the input image that most influenced the model's decision. This is achieved by:
Computing gradients of the target output with respect to the feature maps of the last convolutional layer.
Weighting the feature maps by the importance of these gradients.
Generating a heatmap that highlights the areas of interest.
This script extends Grad-CAM for:
Classification Models: Highlights areas contributing to the predicted class.
Visual Regression Models: Highlights areas contributing to the numerical regression output.
Dataset export:
Exports the test dataset from an Edge Impulse project.
Automatically retries until the dataset is ready for download.
Model download:
Retrieves the trained model (.h5
format) for your Edge Impulse project.
Ensures compatibility with models containing 2D convolutional layers.
Grad-CAM visualization:
Applies the Grad-CAM technique to visualize the regions of the input image that contributed most to the model's predictions.
Works for:
Classification models: Highlights regions associated with the predicted class.
Regression models: Highlights regions contributing to the predicted regression value.
Output generation:
Saves Grad-CAM heatmaps overlaid on the original images.
Separates correctly predicted and incorrectly predicted samples into separate directories for easy analysis.
Clone this repository:
Initialize the block:
Push the deployment block to your organization:
Now you should see your custom deployment block in your organization:
Make sure to enable the Privileged mode, this custom block needs to make API requests to retrieve the dataset and the .h5
model
To use your custom block within your project, head to the Deployment page and select the Grad-CAM visualization option, the output will be a .zip
file containing your test dataset with the Grad-Cam overlay.
Build:
Run:
You can adjust three parameters to fine-tune the visualization: alpha
, pooling-gradients
, and heatmap-normalization
.
--alpha
)The alpha
parameter controls the transparency of the Grad-CAM overlay when it is superimposed on the original image.
Default Value: 0.4
Range:
A value between 0
and 1
.
0
: Fully transparent (only the original image is visible).
1
: Fully opaque (only the Grad-CAM overlay is visible).
How to Choose:
Recommended Default (0.4
): Provides a balance between showing the original image and highlighting the Grad-CAM heatmap.
Higher Alpha (> 0.5
): Use this if you want the Grad-CAM heatmap to dominate the visualization.
Lower Alpha (< 0.4
): Use this if you want the original image to be more prominent.
--pooling-gradients
)The pooling-gradients
parameter determines how gradients (importance signals) are combined across the spatial dimensions of the last convolutional layer to generate the heatmap.
Options:
mean
(default):
Averages the gradients across the spatial dimensions.
Smooth out the heatmap, providing a general overview of important regions.
sum_abs
:
Takes the sum of the absolute values of the gradients.
Highlights areas with strong activations, which can sometimes create sharper heatmaps.
How to Choose:
Classification Models:
Use mean
for a smoother and more generalized heatmap.
Use sum_abs
if you want to emphasize the most critical regions (e.g., sharp object boundaries).
Regression Models:
sum_abs
is often more useful for regression tasks, as it highlights features contributing to extreme values.
Experiment: Try both options to see which one provides more meaningful visualizations for your model and dataset.
--heatmap-normalization
)The heatmap-normalization
parameter determines how the heatmap values are scaled for visualization.
Options:
percentile
(default):
Ensures the heatmap values are scaled between 0
and 1
based on their maximum value.
Best for emphasizing high-activation areas while normalizing the output globally.
simple
:
Normalizes the heatmap by dividing by the global maximum.
A simpler approach that may work well for datasets where most activations are relevant.
How to Choose:
Default (percentile
):
Works well in most cases, especially if you expect only certain areas of the image to be significant.
Use simple
:
Suitable for models where the activations are uniformly spread, and all areas are equally important.
This script assumes the presence of at least one 2D convolutional layer in the model architecture.
It is designed for image classification and visual regression tasks.
For regression models, the script uses a threshold to determine correctness; adjust this threshold (threshold = 0.1
) as needed for your use case.
These tutorials will help you to get started.
We hope this section has helped you to understand the process of Lifecycle Management and how to implement it in your own project. If you have any questions, please reach out to us on our .
We've implemented continuous audio sampling already on the and the targets (the firmware for both targets is open source), but here's a guideline to implementing this on your own targets.
Then is called which will take the slice of data, run the DSP pipeline over the data, stitch data together, and then classify the data.
The source code and the generator script can be found .
See documentation for more details about custom deployment blocks.
The script copies template files from a templates
directory to the target directory. The template available includes files with code structures and placeholders for customization. It's adapted from the example available on Github.
If you see the following segmentation fault, make sure to
We also have an option to run it using a . This option will work for non-Enterprise projects.
Source code:
Blog post:
If you want more information on the Grad-CAM technique, we invite you to read this paper .
To test the functionality without setting up locally, use this . It comes pre-configured to run in a browser with no local setup required.
And you can by editing the custom deployment block:
This page is part of the Lifecycle Management with Edge Impulse tutorial series. If you haven't read the introduction yet, we recommend you to do so here.
In this tutorial, we'll guide you through deploying updated impulses over-the-air (OTA) to Arduino using Edge Impulse. We'll build on Arduino firmware update workflow, incorporating Edge Impulse's API to check for updates and download the latest build.
Let's get started!
Edge Impulse Account: If you haven't got one, sign up here.
Trained Impulse: If you're new, follow our data acquisition and impulse design guides.
Here’s the complete C code for implementing OTA updates with Edge Impulse on ESP-EYE (ESP32).
A trained impulse in Edge Impulse Studio
Installation of required software as detailed in the tutorial
Begin by setting up your device for OTA updates following Espressif's OTA firmware update workflow. Use the built binary from the C++ example and modify it to incorporate OTA functionality.
Clone the example repository and adjust it according to your project and connectivity settings.
Modify the ESP OTA example server to check for updates to your project
Modify the Edge Impulse C++ example for ESP32 to check for updates to your project and download the latest build.
We will need to add a few libraries to the project to facilitate the OTA update process. These are taken from the ESP32 OTA example and are already included in the example project.
NVS is utilized to persistently store data like configuration settings, WiFi credentials, or firmware update times, ensuring retention across reboots.
This library facilitates HTTP requests to the server for checking and retrieving new firmware updates.
These headers aid in executing OTA operations, including writing new firmware to the flash and switching boot partitions.
FreeRTOS ensures OTA updates are conducted in a separate task, preventing blockage of other tasks and maintaining system operations during the update.
Compare the model's timestamp or hash with the stored version. If it's different or newer, call the download_model() function.
Monitor the device to ensure the new impulse performs as expected and repeat the update process as needed.
This tutorial provides a comprehensive guide for implementing OTA updates on Espressif ESP-EYE (ESP32) with Edge Impulse. Follow each step meticulously, ensuring all prerequisites and preparation steps are completed before proceeding to the deployment phase. Happy coding!
Note: Adjust the code snippets and steps to suit your specific requirements and always ensure to test thoroughly before deploying updates to live environments.
This page is part of the Lifecycle Management with Edge Impulse tutorial series. If you haven't read the introduction yet, we recommend you do so here.
In this tutorial, we'll guide you through deploying updated impulses over-the-air (OTA) to Arduino using the Arduino IoT Cloud and Edge Impulse.
Let's get started!
Edge Impulse Account: Sign up here.
Trained Impulse: If you're new, follow our data acquisition and impulse design guides.
Remote Update: Update your Arduino board firmware remotely.
Integration with Edge Impulse: Seamlessly incorporate machine learning models.
Here’s an example Arduino sketch for implementing OTA updates with Edge Impulse:
To expand the Arduino sketch to work with the Edge Impulse API, you'll need to add functionality to fetch the latest model updates from Edge Impulse and apply them to your device. Below is an extended version of the loop()
, connectToWiFi()
, and onOTAEvent()
functions, integrating Edge Impulse API interactions:
loop()
FunctionThe loop()
function regularly checks for updates from the Edge Impulse API.
connectToWiFi()
FunctionThe connectToWiFi()
function includes a maximum number of attempts to connect to WiFi.
onOTAEvent()
FunctionThe onOTAEvent()
function is triggered when a new update is available, downloading and applying the update.
This sketch sets up the Arduino to connect to the Arduino IoT Cloud and listens for OTA update notifications. When an OTA update is available, the onOTAEvent
function is called, where you can implement the logic to download and update the firmware.
Setup Arduino IoT Cloud: Configure your device and variables in the Arduino IoT Cloud.
Connect your device: Ensure your Arduino board is connected to the internet.
Integrate with Edge Impulse: Export your trained impulse from Edge Impulse as an Arduino library.
Update Sketch: Incorporate the Edge Impulse library into your Arduino sketch.
Listen for OTA Updates: Use the Arduino IoT Cloud to push OTA updates to your device.
Test and Monitor: After deploying the update, monitor your device's performance and ensure the new impulse is functioning correctly.
ArduinoIoTCloud: Manages cloud connectivity and OTA updates.
WiFiNINA: Handles WiFi connections for compatible boards.
This tutorial provides a basic guide for implementing OTA updates on Arduino boards using the Arduino IoT Cloud and integrating with Edge Impulse. Expand and develop this example to suit your project's needs. Remember, testing is crucial before deploying updates in a live environment. Happy coding!
This tutorial is part of the Lifecycle Management with Edge Impulse series. If you haven't read the introduction yet, we recommend doing so here.
We'll guide you through deploying updated machine learning models over-the-air (OTA) to the Nordic Thingy:53 using Edge Impulse. This process leverages the Nordic Thingy:53 app, allowing users to deploy firmware updates and facilitating on-device testing for Lifecycle Management.
User-initiated firmware deployment via the Nordic Thingy:53 app.
Remote data collection and on-device testing for machine learning models.
Seamless integration with Edge Impulse for Lifecycle Management.
Edge Impulse Account: Sign up if you don't have one here.
Trained Impulse: If you're new, follow one of our end-to-end tutorials
Nordic Thingy:53: Have the device ready and charged.
Nordic Thingy:53 App: Installed on your smartphone or tablet.
Begin by connecting your Nordic Thingy:53 to the Edge Impulse platform and setting it up for data collection and model deployment.
Connect your Nordic Thingy:53 to the Edge Impulse using the Nordic Thingy:53 app. This will be your interface for managing the device and deploying updates.
Use the Nordic Thingy:53 to collect relevant data for your machine learning application.
Upload this data to Edge Impulse and train your model.
Once your model is trained and ready, use the Nordic Thingy:53 app to deploy it to the device.
The app allows you to initiate the OTA update, which downloads and installs the latest firmware containing the new model.
Conduct remote testing through the app to evaluate the model's performance in real-world scenarios.
This step is crucial for validating the effectiveness of your machine learning model.
Continuously collect new data with the Nordic Thingy:53.
Re-train your model on Edge Impulse with this new data.
Deploy these updates to the Thingy:53 via the app, maintaining the cycle of Lifecycle Management.
This tutorial provides a straightforward approach to implementing OTA updates and Lifecycle Management on the Nordic Thingy:53 using Edge Impulse. The user-friendly Nordic Thingy:53 app facilitates easy deployment of firmware updates, making it ideal for rapid prototyping and iterative machine learning model development.
This guide helps users leverage the capabilities of the Nordic Thingy:53 for advanced IoT applications, ensuring devices are always updated with the latest intelligence and improvements.
This page is part of the Lifecycle Management with Edge Impulse tutorial series. If you haven't read the introduction yet, we recommend you to do so here.
OTA DFU with Notehub Blues Wireless have created a unique way to update firmware on your Notecard device is to perform an over-the-air DFU with Notehub.io. For instructions on how to update your Notecard firmware with Notehub.io, please visit the Manage Notecard Firmware guide.
NOFU: Notehub Outbound Firmware Update allows you to update the firmware on the Swan from Notehub.
Blues Wireless has created their own Edge Impulse ingestion API integration which allows you to collect data from the notecard.
Edge Impulse Account: If you haven't got one, sign up here.
Trained Impulse: If you're new, follow one of our end-to-end tutorials
Blues Wireless Account: If you haven't got one, sign up here.
Blues Wireless Notecard: If you haven't got one, order here.
Watch the full webinar here for insights into MLOps and its integration with Edge Impulse, Blues, and Zephyr.
Webinar Overview:
Implementing MLOps Workflow:
Data collection using the Notecard.
Sending data to Edge Impulse for model training.
Deploying the trained model using Zephyr.
Monitoring and continuous improvement.
Author: TJ VanToll link: https://dev.blues.io/blog/robust-ml-ops-workflow/
In TinyML projects, the effectiveness of a machine learning model greatly depends on the quality of its training data. Gathering comprehensive training data is particularly challenging in TinyML due to the limited power and connectivity of tiny devices. At Blues, in collaboration with Edge Impulse and Zephyr, we have developed a workflow that facilitates MLOps (Machine Learning Operations) for tiny devices.
This tutorial outlines a method to implement a robust MLOps process, focusing on how to collect ML data from remote devices and update ML models on these devices in the field. The result is a workflow that seamlessly integrates data collection and model updating.
see the Establishing a Robust MLOps Workflow for TinyML Blues Guide for the in depth guide.
Let's assume we're managing devices monitoring industrial equipment with accelerometers to detect abnormalities. For a successful MLOps workflow, these devices need to send accelerometer data to the cloud for training an updated model. Here, we use the Blues Notecard for this purpose.
The Blues Notecard
The Notecard is a system on module designed for easy connectivity in embedded projects. Its features include:
500MB of cellular connectivity
Global cellular support over LTE-M, NB-IoT, or Cat-1
Secure device-to-cloud communication
Low-power hardware and firmware
Easy embedding options
To set up the hardware, you'll need a Blues Starter Kit, which includes a development board (Notecarrier) and a Swan microcontroller. Additionally, an LIS3DH accelerometer connected to the Notecarrier via a Qwiic cable is required.
The Firmware
The project’s firmware gathers accelerometer readings and sends them to the cloud via the Notecard. We use Zephyr for firmware implementation, as it offers both low-level access and development conveniences. Zephyr firmware is compatible with STM32-based boards like the Swan, and the Notecard has a Zephyr SDK.
The firmware performs the following functions:
Establishes a connection between the Notecard and Notehub
Gathers data from the accelerometer
Sends the data to Notehub
For example, the firmware takes accelerometer readings at intervals set by the Edge Impulse SDK and sends this data as a binary stream to Notehub.
Notecard's new firmware (v5.3.1) introduces 'card.binary' APIs, enabling fast data transfer for large data packets. The data is then sent to Notehub using the 'web.post' method.
The data from Notehub is forwarded to Edge Impulse through an HTTP server. This server, which can be created using any language or framework, listens for POST requests and parses the floating-point accelerometer values. The data is then sent to Edge Impulse's ingestion API, adding new entries to the model’s training set.
see the section of the Blues Guide for more information.
The final aspect of the MLOps process is updating the models on the devices in the field. For this, we use Notecard Outboard Firmware Update. This feature allows OTA firmware updates without writing any code, offering flexibility and safety against bricking devices.
see the Updating ML Models section of the Blues Guide for more information.
Notecard Outboard Firmware Update
Requires specific wiring (provided in the Blues Starter Kit)
Activated via a 'card.dfu' request in the firmware
Involves building a new firmware image file with the updated model
The new firmware is uploaded to Notehub and applied to devices
see the What is notecard outboud update? section of the Blues Guide for more information.
The combination of data collection and model updating forms a robust MLOps process. This approach ensures continuous improvement of ML models based on real data and updates models remotely. The provided GitHub samples offer a reference implementation, adaptable to various project needs.
For more information, refer to the [Notecard Outboard Firmware Update guide](https://dev.blues.io/guides-and-tutorials/notecard-guides
The combination of data collection and model updating forms a robust MLOps process. This approach ensures continuous improvement of ML models based on real data and updates models remotely. The provided GitHub samples offer a reference implementation, adaptable to various project needs.
Build Your Edge ML Application: Follow the step-by-step Blues Swan tutorial.
This tutorial acts as a reference to the webinar and tutorial from Blues Wireless: Optimized MLOps with Edge Impulse, Blues, and Zephy view the full webinar.
This page is part of the Lifecycle Management with Edge Impulse tutorial series. If you haven't read the introduction yet, we recommend you to do so here.
In this tutorial, we'll guide you through deploying updated impulses over-the-air (OTA) using the Particle Workbench. We'll build on Particle's firmware update workflow, incorporating Edge Impulse's API to check for updates and download the latest build.
We will modify the Edge Impulse Photon 2 example to incorporate the OTA functionality and update the device with the latest build. Based on the particle workbench OTA example.
Edge Impulse Account: If you haven't got one, sign up here.
Trained Impulse: If you're new, follow one of our end-to-end tutorials
Knowledge of the Particle Workbench
Installation of required software as detailed in the Photon 2 example
Begin by setting up your device for OTA updates following the particle documentation. https://docs.particle.io/getting-started/cloud/ota-updates/
Let's get started!
Development Devices: Specific devices can be marked for internal testing to validate firmware updates before a broader rollout.
Firmware Upload: Developers upload the compiled binary to the console, ensuring the firmware version is included.
Firmware Locking: Specific devices can be locked to run certain firmware versions for testing purposes.
Release Targets: Firmware can be released to all devices or specific groups to control the rollout process.
Intelligent Firmware Releases: Ensures timely delivery of updates without disrupting device operation, taking into account the device’s status and activity.
Particle OTA is a fully-integrated over-the-air software update system that is built into the Particle IoT PaaS and Device OS. It allows customers to safely and reliably push software updates to single devices or entire fleets of devices directly from Particle’s device management console and developer tools, with no implementation work necessary.
Particle OTA allows you to update your entire IoT device (both the Particle device and any other components) by delivering three kinds of updates:
Application OTA allows users to update the firmware application they are running on the Particle device in order to introduce new features, fix bugs, and generally improve the software application over time.
Device OS OTA allows users to update Device OS to the latest version alongside an application update so that Device OS can be kept up to date with improvements and bug fixes while properly testing against the user-generated software application.
Asset OTA allows users to include bundled assets in an OTA software update that can be delivered to other processors and components in the electronics system so that the Particle device can be responsible not just for updating itself but also the system that surrounds the device.
Export your impulse to the Particle Workbench library:
Add the library through the Particle Workbench via:
Extract the .zip library.
Particle: Import Project and select project.properties. Examples can then be found in: yourprojectname/examples/
We created an example repository which contains a small application for your particle device.
You will need to treat this as a new project and follow the instructions in the Particle Workbench to set up your environment for local development.
If your project includes a library that has not been registered in the Particle libraries system, you should create a new folder named /lib/<libraryname>/src
under /<project dir>
and add the .h
, .cpp
& library.properties
files for your library there. Read the Firmware Libraries guide for more details on how to develop libraries. Note that all contents of the /lib
folder and subfolders will also be sent to the Cloud for compilation.
When you're ready to compile your project, make sure you have the correct Particle device target selected and run particle compile <platform>
in the CLI or click the Compile button in the Desktop IDE. The following files in your project folder will be sent to the compile service:
Everything in the /src
folder, including your .ino
application file
The project.properties
file for your project
Any libraries stored under lib/<libraryname>/src
You should now have a compiled binary file in your project folder named firmware.bin
.
Next lets look at the OTA functionality and how we can incorporate it into our project.
You will need to have a device deployed with an existing firmware containing an impulse before you can roll out a new version. Then you can use the OTA workflow to update the firmware on your device to a new version. While blocking all other devices from updating to the new version.
Lock Devices: Lock specific devices to the new firmware version and monitor their behavior. Unlock Devices: Unlock the devices once testing is complete and you are satisfied with the firmware's performance.
Release Targets: Choose to release the firmware to specific groups or the entire fleet of devices. Intelligent Firmware Release: Opt for this to have the Particle Device Cloud intelligently manage the update rollout. Important Tips: Avoid Disruptions: Utilize Device OS APIs to control OTA availability, ensuring devices aren’t disrupted during critical operations. Managing OTA Updates: Use the console or REST API to force enable OTA updates if needed. Monitoring: Keep an eye on the devices' behaviors post-update to quickly identify and address any potential issues. With Particle's OTA capabilities, developers can ensure that their IoT devices are always running the latest, most secure, and most efficient version of their firmware, enhancing the reliability and functionality of their IoT ecosystems.
This guide supplements the tutorial on OTA Model Updates with Edge Impulse on Particle Workbench, focusing on configuring a Particle webhook for sending data to the Edge Impulse ingestion API.
Access Particle Console:
Visit Particle Console.
Log in with your Particle account credentials.
Navigate to Integrations:
Click on the "Integrations" tab in the left-hand menu.
Select "Webhooks" from the available options.
Create a New Webhook:
Click "New Integration".
Choose "Webhook".
Webhook Configuration:
Name: Assign a descriptive name to your webhook.
Event Name: Specify the event name that triggers the webhook (e.g., "edge/ingest").
URL: Set this to the Edge Impulse ingestion API URL, typically something like https://ingestion.edgeimpulse.com/api/training/data
.
Request Type: Choose "POST".
Request Format: Select "Custom".
Custom Request Body:
Input the JSON structure required by Edge Impulse. This will vary based on your project's data schema.
HTTP Headers:
Add necessary headers:
x-api-key
: Your Edge Impulse API key.
Content-Type
: "application/json".
x-file-name
: Use a dynamic data field like {{PARTICLE_EVENT_NAME}}
.
Advanced Settings:
Response Topic: Create a custom topic for webhook responses, e.g., {{PARTICLE_DEVICE_ID}}/hook-response/{{PARTICLE_EVENT_NAME}}
.
Enforce SSL: Choose "Yes" for secure transmission.
Save the Webhook:
After entering all details, click "Save".
Test the Webhook:
Use example device firmware to trigger the webhook.
Observe the responses in the Particle Console.
Debugging:
If errors occur, review the logs for detailed information.
Ensure payload format aligns with Edge Impulse requirements.
Verify the accuracy of your API key and other details.
Copy and paste the following into the Custom Template section of the webhook:
Below we will do a basic example of sending data to Edge Impulse for Lifecycle Management. We could further extend this example to include more advanced checking, utilizing more of Particle OTA functionality, and add checks for model performance and versioning. The code gathers data from analog sensors, processes it, and sends it to Edge Impulse for Lifecycle Management. The webhook is triggered with the event name edge/ingest/sample.
This code continuously samples voltage and current, calculates RMS current, and then sends a JSON array of the sampled data to the Edge Impulse ingestion API. The publishData function uses Particle's Particle.publish method to send the data to the specified event. This triggers the webhook configured to send data to Edge Impulse. See the Particle energy monitor code from our Imagine 2023 demo for a full example for more information.
This tutorial provides a basic guide for implementing OTA updates on your Particle Workbench connected device (Photon 2). It can be extended to include more advanced checking, utilizing more of espressif OTA functionality, and extending the python server to check for model performance and versioning. Happy coding!
This page is part of the Lifecycle management with Edge Impulse tutorial series. If you haven't read the introduction yet, we recommend you to do so here. Balena can serve as the platform for deploying OTA updates to edge devices, including new models trained with Edge Impulse.
An active Edge Impulse account with a project and trained impulse. See the Edge Impulse - Object Detection Tutorials for more information on how to create a project and train an impulse.
Follow the Edge Impulse Docker documentation.
Balena Example Repo - A sample balena project to deploy your impulse as on balena. It exposes an API which the "cam" service uses to get repeated inferences from the images captured by the webcam. See the README
Balena offers a comprehensive platform for building, deploying and managing large fleets of IoT devices. It simplifies fleet management, enhances security, and streamlines the deployment of software and hostOS updates. This tutorial will guide you through using balena to deploy Edge Impulse model updates across your device fleet efficiently. This can be particularly useful for managing multiple devices in the field, ensuring they are always running the latest model. Devices like the NVIDIA Jetson Nano, NVIDIA Jetson Orin, Raspberry Pi, and other single-board computers are supported by balena and can be used to deploy Edge Impulse models.
For this example, we will deploy an Edge Impulse model as a Docker container on a Raspberry Pi using balenaOS. The model will run an HTTP inference server, allowing you to send data to the device for processing and receive predictions in real-time.
An active Edge Impulse account with a trained model.
Follow the Edge Impulse Docker documentation.
Balena is a platform that provides tools for building, deploying, and managing large fleets of IoT devices. It simplifies the process of managing fleets of devices, offering a robust framework for deploying software and hostOS updates, monitoring device health, and ensuring security. Balena could serve as the fleet management and device management platform for deploying OTA updates, including new models trained with Edge Impulse.
Go to your Edge Impulse project, navigate to the Deployment section and select Docker container as the deployment option. Follow the instructions to generate the Docker container command. It will look something like this:
Note the Container address and the api-key, as we will need to use them in a subsequent step below.
Log in to your balenaCloud dashboard and create a new fleet.
BalenaOS - Add a new Device Select the appropriate device type that matches your hardware. Follow the instructions to add a device to your application and download the balenaOS image for it. Flash the downloaded OS image to your device's SD card using balenaEtcher or a similar tool. Power on your device with the SD card or USB key inserted; it should connect to your balena application automatically.
Clone the balena boilerplate Edge Impulse project from Github or start with a Dockerfile.template in a new directory on your local machine. Modify the Dockerfile.template to use your container address and api-key as mentioned earlier. Since balena uses Docker containers, you will simply use the container generated by the deployment screen of your Edge Impulse model.
Use the balena CLI to build, and scan for your local device and push your application to balenaCloud:
Wait for the application to build and deploy. You can monitor the progress in the balenaCloud dashboard.
Once deployed, your device will start the Docker container and run the HTTP inference server.
You can access it using the device's IP address on your local network or through the Public Device URL feature provided by balenaCloud if enabled for your device. Remember that we decided to point this server to the port 80, but feel free to use another port.
With your Edge Impulse inference server running on balena, now you can monitor and manage the other services running on your device fleet using balenaCloud's dashboard and tools. This includes monitoring device health, deploying updates, and rolling back changes if needed.
Furthermore, to deploy the latest ML model deployed from Edge Impulse after retraining, you will need to restart the edge-impulse service running in the connected devices. You can do it from the balenaCloud Fleet Devices page selecting them.
If you have any questions or any issues feel free to contact the balena team using the balena forums.
By following these steps, you should have a functional Edge Impulse inference server running on your balena devices, ready to process data and make predictions. This setup can be integrated into a robust OTA model update process, enabling Lifecycle management and improvement of your Edge AI enabled devices.
Additional Resources
Webinar - Edge Impulse and Balena webinar exploring this topic in more detail.
In this tutorial we will demonstrate how to securely collect and process sensor data from a managed device with the Golioth on AI example project. The managed device is connected via cellular and streams labeled but unstructured data to a mutually accessible Object Storage (S3 Bucket), by collecting labeled sensor data in this method we can also demonstrate a common use for this form of data acquisition, performing automated Data processing or Data Quality steps to the collected data via custom transformation blocks.
We will then apply data processing steps on the collected data using the custom CBOR transformation block to convert this data to a format we can use to later train a model. This same process can be adapted to perform Data Quality techniques that would typically be performed by an ML engineer.
To follow along with this example, ensure that you have the following:
Golioth account with an organization and a connected compatible device (e.g., nrf9160).
Edge Impulse enterprise account.
Trained Edge Impulse project ready for deployment.
AWS account with access to S3.
Basic knowledge of AWS S3, Docker, and Zephyr.
Example from the Golioth on AI tutorial repository here
This tutorial was created for the Golioth on AI launch see the associated video on youtube Read it here.
Before proceeding with the integration, ensure that your Golioth device is set up with the appropriate firmware. For detailed instructions on initializing your Zephyr workspace, building, and flashing the firmware, please refer to the Golioth on AI repository README.
Clone the Golioth on AI repository and initialize the Zephyr workspace with the provided manifest file.
Log in to Golioth and create a new project. This project will handle the routing of sensor data and classification results.
Log in to Edge Impulse Studio and create a new project. This project will receive and process the raw accelerometer data uploaded from the S3 bucket.
Once your device is set up, follow the instructions in the repository README to create the necessary Golioth pipelines. This includes setting up the Classification Results Pipeline and the Accelerometer Data Pipeline. Latest detailed steps can be found in the repository Golioth on AI repository README.
Golioth Pipelines allows you to route data between different services and devices efficiently. You will need to configure two pipelines for this demo:
This pipeline routes classification results (e.g., gesture predictions) to Golioth's LightDB Stream, which stores data in a timeseries format.
You will configure a path for classification results (e.g., /class) and ensure that the data is converted to JSON format.
This pipeline handles raw accelerometer data by forwarding it to an S3 object storage bucket. Ensure the pipeline is set up to transfer binary data.
Important: Configure your AWS credentials by creating secrets in Golioth for AWS_ACCESS_KEY and AWS_SECRET_KEY, and specify the target bucket name and region.
Follow the instructions in the continuous motion recognition tutorial to generate a gesture recognition model.
Download the generated library from Edge Impulse Studio and extract the contents.
Build the firmware:
Set Golioth Credentials::
Add the same configuration to the Golioth secret store
Navigate to your Golioth Secret Store: https://console.golioth.io/org//project//secrets
Add your AWS credentials (Access Key ID and Secret Access Key) and the S3 bucket details to the Golioth secret store.
Press the button on the Nordic Thingy91 to start sampling data from the device's accelerometer.
Raw accelerometer data will be automatically routed to your S3 bucket via the Golioth pipeline. You can later import this data into Edge Impulse Studio for further model training.
Classification results will be stored in Golioth's LightDB Stream. You can access this data for further analysis or visualization.
In Edge Impulse Studio, use the Data acquisition page to import your raw accelerometer data directly from the S3 bucket
Once imported, label your data appropriately to prepare it for model training.
Below is an example of classification data you can expect to see in the Golioth console:
Raw accelerometer data uploaded to S3 can be imported as an array of X-Y-Z float values, which will appear in the Edge Impulse Studio as time-series data.
From here we can perform a number of data processing steps on the collected data:
Use the custom CBOR transformation block to convert raw accelerometer data to a format suitable for training a model in Edge Impulse.
Apply custom transformation blocks to perform data quality checks or preprocessing steps on the collected data.
Import the transformed data into Edge Impulse Studio and train a new model using the collected accelerometer data.
Deploy the trained model back to the Golioth device for real-time gesture recognition.
In this tutorial, we demonstrated how to securely collect and process sensor data from a managed device using Golioth and Edge Impulse. By leveraging Golioth's data routing capabilities and Edge Impulse's machine learning tools, you can easily build and deploy custom models for gesture recognition or other applications. This example showcases the end-to-end workflow from data acquisition to model training and deployment, highlighting the seamless integration between Golioth and Edge Impulse.
For more information on Golioth and Edge Impulse, visit the official documentation and tutorials:
In this section we will explore how firmware updates and other scenarios are currently addressed, with traditional OTA. It should help you to get started planning your own updated impulse across a range of platforms.
Starting with platform-specific examples like Arduino Cloud, Nordic nRF Connect SDK / Zephyr and Golioth, Particle Workbench and Blues Wireless. Finally we will explore building an end-to-end example on the Espressif IDF.
By covering a cross section of platforms we hope to provide a good overview of the process and how it can be applied to your own project. With more generic examples like Arduino, Zephyr and C++ which can be applicable to all other vendors.
These tutorials will help you to get started with the following platforms:
Edge Impulse Account: If you haven't got one, sign up here.
Trained Impulse: If you're new, follow one of our end-to-end tutorials
Edge Impulse recognises the need for OTA model updates, as the process is commonly referred to although we are going to be updating the impulse which includes more than just a model, a complete review of your infrastructure is required. Here is an example of the process:
The initiation of an update to your device can be as straightforward as a call to our API to verify the availability of a new deployment. This verification can be executed either by a server or a device with adequate capabilities. Changes can be dependent on a range of factors, including but not limited to the last modified date of the project, the performance of the model, or the release version of the project e.g. last modified date of the project endpoint:
https://docs.edgeimpulse.com/reference/getprojectlastmodificationdate
After we inquire about the last modification, and if an update is available, proceed to download the latest build through:
https://docs.edgeimpulse.com/reference/downloadbuild
We could add further checking for impulse model performance, project release version tracking or other metrics to ensure the update is valid. However in this series we will try to keep it simple and focus on the core process. Here is an example of a more complete process:
Identify components that influence change: Determine the components of your project that require updates. This could be based on performance metrics, data drift, or new data incorporation.
Retrain: Focus on retraining based on the identified components of your project.
Test and Validate: Before deploying the updated components, ensure thorough testing and validation to confirm their performance before sending the update.
Deploy Updated Components: Utilize available OTA mechanisms to deploy the updated components to your devices. Ensure seamless integration with the existing deployment that remains unchanged.
Monitor on device Performance: Post-deployment, continuously monitor the performance of the updated model to ensure it meets the desired objectives. See Lifecycle Management for more details.
The aim will be to make sure your device is always equipped with the most recent and efficient impulse, enhancing performance and accuracy.
We hope this section has helped you to understand the process of OTA model updates and how to implement it in your own project. If you have any questions, please reach out to us on our forum.
Allxon provides essential remote device management solutions to simplify and optimize edge AI device operations. As an AI/IoT ecosystem enabler, connecting hardware (IHV), software (ISV), and service providers (SI/MSP), Allxon serves as the catalyst for fast, seamless connectivity across all systems.
Allxon Over-the-Air (OTA) deployment works perfectly with Edge Impulse OTA model update on NVIDIA Jetson devices. This tutorial guides you through the steps to deploy a new impulse on multiple devices.
This guide demonstrates how to deploy and manage Edge Impulse models on NVIDIA Jetson devices using Allxon's Over-the-Air (OTA) deployment capabilities. Allxon provides essential remote device management solutions to streamline and optimize edge AI device operations.
Before you begin, ensure you have the following:
Updated impulse as a Docker container from Edge Impulse.
Get Allxon officially supported devices(https://www.allxon.com/).
Create an Allxon account.
Allxon's services are compatible with a variety of hardware models. Follow these steps to complete the required preparations.
Install Allxon Agent: Use the command prompt to install the Allxon Agent on your device.
Pair Your Device: Follow the instructions to add your device to Allxon Portal.
Once added, your devices will appear in the Allxon Portal for management and monitoring.
To perform an OTA deployment, ensure you have your updated Impulse deployed as a Docker container from Edge Impulse.
Generate OTA Artifact: Use the Allxon CLI to generate the OTA artifact.
Deploy OTA Artifact: Follow the Deploy OTA Artifact guide to complete the deployment.
Below are example scripts to help you set up the OTA deployment.
Two minor modifications have been made to the standard Docker command from Edge Impulse docker deploy:
The -it
option has been removed from the Docker command to avoid an error related to the lack of standard input during deployment. An &
has been added to the end of the Docker command to send the process to the background.
By following these steps, you can efficiently deploy and manage Edge Impulse models on NVIDIA Jetson devices using Docker through Allxon. This setup leverages Allxon's remote management capabilities to streamline the process of updating and maintaining your edge AI devices.
We hope this section has helped you understand the process of Lifecycle Management and how to implement it in your own project. If you have any questions, please reach out to us on our forum.
The is built on top of the , which is known as the edgeimpulse_api package. These are Python wrappers for all of the that you can use to interact with Edge Impulse projects programmatically (i.e. without needing to use the Studio graphical interface).
The API reference guide for using the Python API bindings can be found .
This example will walk you through the process of using the Edge Impulse API bindings to upload data, define an impulse, process features, train a model, and deploy the impulse as a C++ library.
After creating your project and copying the API key, feel free to leave the project open in a browser window so you can watch the changes as we make API calls. You might need to refresh the browser after each call to see the changes take affect.
Important! This project will add data and remove any current features and models in a project. We highly recommend creating a new project when running this notebook! Don't say we didn't warn you if you mess up an existing project.
Note that you do not actually need to use the project in the Edge Impulse Studio. We just need the API Key.
Paste that API key string in the EI_API_KEY
value in the following cell:
The Python API bindings use a series of submodules, each encapsulating one of the API subsections (e.g. Projects, DSP, Learn, etc.). To use these submodules, you need to instantiate a generic API module and use that to instantiate the individual API objects. We'll use these objects to make the API calls later.
To configure a client, you generally create a configuration object (often from a dict) and then pass that object as an argument to the client.
API calls (links to associated documentation):
We rely on the ingestion service using the string before the first period in the filename to determine the label. For example, "idle.1.cbor" will be automatically assigned the label "idle." If you wish to set a label manually, you must specify the x-label
parameter in the headers. Note that you can only define a label this way when uploading a group of data at a time. For example, setting "x-label": "idle"
in the headers would give all data uploaded with that call the label "idle."
API calls used with associated documentation:
Now that we uploaded our data, it's time to create an impulse. An "impulse" is a combination of processing (feature extraction) and learning blocks. The general flow of data is:
data -> input block -> processing block(s) -> learning block(s)
Only the processing and learning blocks make up the "impulse." However, we must still specify the input block, as it allows us to perform preprocessing, like windowing (for time series data) or cropping/scaling (for image data).
Your project will have one input block, but it can contain multiple processing and learning blocks. Specific outputs from the processing block can be specified as inputs to the learning blocks. However, for simplicity, we'll just show one processing block and one learning block.
Note: Historically, processing blocks were called "DSP blocks," as they focused on time series data. In Studio, the name has been changed to "Processing block," as the blocks work with different types of data, but you'll see it referred to as "DSP block" in the API.
It's important that you define the input block with the same parameters as your captured data, especially the sampling rate! Additionally, the processing block axes names must match up with their names in the dataset.
API calls (links to associated documentation):
Before generating features, we need to configure the processing block. We'll start by printing all the available parameters for the spectral-analysis
block, which we set when we created the impulse above.
API calls (links to associated documentation):
After we've defined the impulse, we then want to use our processing block(s) to extract features from our data. We'll skip feature importance and feature explorer to make this go faster.
Generating features kicks off a job in Studio. A "job" involves instantiating a Docker container and running a custom script in the container to perform some action. In our case, that involves reading in data, extracting features from that data, and saving those features as Numpy (.npy) files in our project.
Because jobs can take a while, the API call will return immediately. If the call was successful, the response will contain a job number. We can then monitor that job and wait for it to finish before continuing.
API calls (links to associated documentation):
Now that we have trained features, we can run the learning block to train the model on those features. Note that Edge Impulse has a number of learning blocks, each with different methods of configuration. We'll be using the "keras" block, which uses TensorFlow and Keras under the hood.
API calls (links to associated documentation):
Now that the model has been trained, we can go back to the job logs to find the accuracy metrics for both the float32 and int8 quantization levels. We'll need to parse the logs to find these. Because the logs are printed with the most recent events first, we'll work backwards through the log to find these metrics.
As with any good machine learning project, we should test the accuracy of the model using our holdout ("testing") set. We'll call the classify
API function to make that happen and then parse the job logs to get the results.
In most cases, using int8
quantization will result in a faster, smaller model, but you will slightly lose some accuracy.
API calls (links to associated documentation):
Now that you've trained the model, let's build it as a C++ library and download it. We'll start by printing out the available target devices. Note that this list changes depending on how you've configured your impulse. For example, if you use a Syntiant-specific learning block, then you'll see Syntiant boards listed. We'll use the "zip" target, which gives us a generic C++ library that we can use for nearly any hardware.
The engine
must be one of:
We'll use tflite
, as that's the most ubiquitous.
modelType
is the quantization level. Your options are:
In most cases, using int8
quantization will result in a faster, smaller model, but you will slightly lose some accuracy.
API calls (links to associated documentation):
While the Edge Impulse Studio is a great interface for guiding you through the process of collecting data and training a model, the Python SDK allows you to programmatically Bring Your Own Model (BYOM), developed and trained on any platform. See .
With the following tutorials, you will learn how to use the Edge Impulse Python SDK with a number of other machine-learning frameworks and platforms:
The EON Tuner is Edge Impulse's AutoML (automated machine learning) tool to help you find and select the best embedded machine learning model for your application within the constraints of your target device.
This notebook will show you how to configure and run the EON Tuner programmatically using the !
This section will set up your environment and API credentials so that you can start making calls to the from this notebook. Run this block only once per runtime session, or every time you:
Open the notebook on your browser or IDE to start working on it, or
restart the runtime, or
change the project you are working on
API documentation is available at https://docs.edgeimpulse.com/reference/edge-impulse-api
PROJECT_ID
You will need to enter the correct PROJECT_ID
for the project you want to work with, in the code in section 1.3 below. The project ID can be obtained from your Edge Impulse Project's Dashboard under the Project Info section.
The block below will prompt you for your project's API Key. You can obtain this key from your Project's Dashboard, by selecting the Keys tab from the top navigation bar.
Run the block below and enter your API key when prompted. Then continue to the next section.
You can use the code in section 2.2 below to programmatically update the configuration of the EON Tuner.
In basic mode (the default) you will be able to modify the datasetCategory
, targetLatency
and targetDevice
. For additional control, ask your User Success or Solutions Engineer to enable the EON Tuner advanced mode for you.
Run the cell below to start spinning up EON Tuner optimization jobs. If your project is already running an EON Tuner optimization, go instead to section 2.4 to track the job's progress.
Run the cell below to track the progress of your EON Tuner job. You can safely stop and restart the cell at any time since this will not affect the running EON Tuner jobs.
Use the cell below to retrieve the EON Tuner optimization results and save them to the trials
variable.
The gives programmatic access to all features in the studio, and many tasks that might normally have to be performed manually can thus be automated. In this tutorial you'll create a job that deploys your model (as a ZIP file), you monitor the job status, and then finally download the deployed model. The tutorial uses Python, but you can use any environment capable of scripting HTTP requests.
To run this script you'll need:
A recent version of Python 3.
The requests
module:
Your project ID (can be found on Dashboard in the Edge Impulse Studio).
An API Key (under Dashboard > Keys).
Then, create the following script build-model.py:
When you now run python3 download-model.py
you'll see something like:
The exposes programmatic access to most functionality in the studio and it is particularly useful when it comes to automating tasks. You can use the API to edit the labels of many samples at once, train models, or create new impulses. In addition, you can subscribe to events, such as when a new file is processed by the ingestion service. An Edge Impulse Python API bindings is also available as the edgeimpulse-api pip package.
See the following examples:
This page is part of the tutorial series. If you haven't read the introduction yet, we recommend you to do so .
In this tutorial, we'll guide you through deploying updated impulses over-the-air (OTA) using the Zephyr RTOS. We'll build on Zephyr's firmware update workflow, incorporating API calls to check for updates to your edge impulse project and download the latest build. As Zephyr does not have an OTA update service, and Golioth is used in their OTA guide. We'll use Golioth's OTA update service to deliver the updated firmware to the device. On devices running Zephyr RTOS, leveraging Golioth's cloud platform for the firmware update portion of the process.
Secure and encrypted firmware delivery
Delta updates to minimize download size
Integration with Zephyr RTOS
Edge Impulse Account: If you haven't got one, .
Trained Impulse: If you're new, follow one of our
Knowledge of Zephyr RTOS and nRF Connect SDK
A Golioth account and the Golioth SDK installed
Clone the Golioth Zephyr SDK examples repository:
Navigate to the Golioth firmware examples folder we are going to modify the device firmware update example to connect to the ingestion API to send data back from device.
Now copy the edge-impulse-sdk folder from the cloned repository into the dfu folder.
You will need to merge in the run inference function from your existing project. This will be the function that will be called to run inference on the device.
Now add the run inference function to the main loop of the dfu example. This will run inference every 2 seconds.
With the classification, ingestion code and Golioth implementation in place, proceed to build and run the application on the nRF9160 Development Board. Utilize the following commands:
These commands will build and flash the firmware onto your development board.
To verify that the Edge Impulse model is running and updating OTA with Golioth, observe the console output. It should display inference results at regular intervals. Also, you can monitor the device’s firmware version and its updates via the Golioth console.
Simulate an OTA update by changing the Edge Impulse model and repeating the build and flash process. Monitor the Golioth console for updates and check the device console to observe the changes in inference results.
Congratulations! You've successfully implemented OTA model updates with Edge Impulse on Zephyr RTOS and nRF Connect SDK, facilitated by Golioth. You can now continuously enhance and deploy machine learning models to your edge devices securely and efficiently.
This page is part of the tutorial series. If you haven't read the introduction yet, we recommend you to do so .
In this tutorial, we'll guide you through deploying updated impulses over-the-air (OTA) to the ESP32 using Edge Impulse. We'll build on Espressif's end-to-end OTA firmware update workflow, incorporating Edge Impulse's API to check for updates and download the latest build.
We will modify the Edge Impulse C++ example for ESP32, and combine it with the OTA example from the ESP IDF repository. This will allow us to check for updates to your project on the server side of the IDF example and download the latest build. We will then modify the C++ example to incorporate the OTA functionality and update the device with the latest build. Finally we will add calls back to our ingestion service to monitor the performance of the updated impulse, and gather data. Closing the loop on our active learning cycle.
Let's get started!
Edge Impulse Account: If you haven't got one, .
Trained Impulse: If you're new, follow one of our
Knowledge of the ESP IDF development framework and C++ for ESP32
Installation of required software as detailed in the tutorial
Begin by setting up your device for OTA updates following Espressif's OTA firmware update workflow. Use the built binary from the C++ example and modify it to incorporate OTA functionality.
Let's get started!
We created an example repository which contains a small application for Espressif ESP32, which takes the raw features as an argument, and prints out the final classification. Download the application as a .zip, or import this repository using Git:
We will need to add a few libraries to the project to facilitate the OTA update process. These will be taken from the ESP32 OTA example and are already included in the example project.
We are going to use the ESP IDF OTA example as a starting point. Running a python server to check for updates and download the latest build from edge impulse. This detemines if there is a new build available and downloads it to the device.You could add more advanced checking for model performance and versioning to this process. This example will only download the latest build if there is a new build available based on date to save lengthy code examples.
This example is available in the ESP IDF repository. You can find the example in:
Starting with the device side code, we will need to add a few libraries to the project to facilitate the OTA update process. These are taken from the ESP32 OTA example and are already included in the example project.
Modify the ESP OTA example python server to check for updates to your project, we will use the date to determine when a new update should be performed to keep things simple. You could add more advanced checking for model performance and versioning to this process. This example will only download the latest build if there is a new build available based on date to save lengthy code examples.
Building the firmware to deploy to our device in the field. We will use the IDF build system to build the firmware. This would be automated but again to save time and complexity we will do this manually.
Compare the model's timestamp or hash with the stored version. If it's different or newer, call the download_model() function.
Monitor the device to ensure the new impulse performs as expected and repeat the update process as needed.
Here we would make calls back to our ingestion service to monitor the performance of the updated impulse, and gather data. Closing the loop on our active learning cycle.
Now lets modify the C++ example to incorporate the device side of the OTA functionality and update our edge impulse project with the device side data. We will add the OTA functionality to the C++ example and update the device with the latest build.
Include the necessary headers and components in your main ESP32 C++ based edge impulse project. There are a number of components that are required for the OTA update process:
NVS is utilized to persistently store data like configuration settings, WiFi credentials, or firmware update times, ensuring retention across reboots.
This library facilitates HTTP requests to the server for checking and retrieving new firmware updates.
These headers aid in executing OTA operations, including writing new firmware to the flash and switching boot partitions.
FreeRTOS ensures OTA updates are conducted in a separate task, preventing blockage of other tasks and maintaining system operations during the update.
Lets start by adding the necessary headers to the main.cpp file in the edge impulse project.
Step 3: OTA Task Implement the OTA task that will handle the OTA update process. Connecting to the python server we created in the beginning of this tutorial. Which is checking for updates to your project
In the simple_ota_example_task function, you'll need to replace http://192.168.1.10/your-ota-firmware.bin
with the actual URL where your firmware binary is hosted.
Step 4: App Main Here’s how the app_main function should look like:
Step 5: Set the target and configuration Make sure you have the partition table set up correctly to support OTA. In the IDF Menu Config, An OTA data partition (type data, subtype ota) must be included in the Partition Table of any project which uses the OTA functions.
Step 6: Compile and Flash You should compile the firmware and flash it to your ESP32 using the IDF build system commands.
Note: Before you run this code, make sure to replace http://192.168.1.10.com/your-ota-firmware.bin
with the actual URL where your new firmware binary is hosted. Make sure that your ESP32 is connected to the internet to access this URL. Also, always test OTA functionality thoroughly before deploying it in a production environment.
This tutorial provides a basic guide for implementing OTA updates on Espressif ESP-EYE (ESP32) with Edge Impulse. It can be extended to include more advanced checking, utilizing more of espressif OTA functionality, and extending the python server to check for model performance and versioning. Happy coding!
Mark Devices for Testing: In the Particle console, mark specific devices for internal testing to validate firmware updates. Upload Firmware: Compile and upload the firmware binary, ensuring to include the product ID and version.
You will need to obtain an API key from an Edge Impulse project. Log into and create a new project. Open the project, navigate to Dashboard and click on the Keys tab to view your API keys. Double-click on the API key to highlight it, right-click, and select Copy.
Before uploading data, we should make sure the project is in the regular impulse flow mode, rather than . We'll also need the project ID for most of the other API calls in the future.
Notice that the general pattern for calling API functions is to instantiate a configuration/request object and pass it to the API method that's part of the submodule. You can find which parameters a specific API call expects by looking at .
We'll start by downloading the gesture dataset from . Note that the is separate from the regular Edge Impulse API: the URL and interface are different. As a result, we must construct the request manually and cannot rely on the Python API bindings.
You can use the and functions to configure the granular settings. We'll use the defaults for that block and just set the number of epochs and learning rate for training.
You should have a .zip file in the same directory as this notebook. Download or move it to somewhere else on your computer and unzip it. You can now follow to link and compile the library as part of an application.
TensorFlow is an open source library for training machine learning models. Keras is an open source Python library that makes creating neural networks in TensorFlow much easier. We use these two libraries together to very quickly train a model to identify handwritten digits. From there, we use the Edge Impulse Python SDK library to profile the model to see how inference will perform on a target edge device. Then, we use the SDK again to convert our trained model to a C++ library that can be deployed to an edge hardware platform, such as a microcontroller.
Follow the code below to see how to train a simple machine learning model and deploy it to a C++ library using Edge Impulse.
To learn more about using the Python SDK, please see: Edge Impulse Python SDK Overview.
You will need to obtain an API key from an Edge Impulse project. Log into edgeimpulse.com and create a new project. Open the project, navigate to Dashboard and click on the Keys tab to view your API keys. Double-click on the API key to highlight it, right-click, and select Copy.
Note that you do not actually need to use the project in the Edge Impulse Studio. We just need the API Key.
Paste that API key string in the ei.API_KEY
value in the following cell:
We want to create a classifier that can uniquely identify handwritten digits. To start, we will use TensorFlow and Keras to train a very simple convolutional neural network (CNN) on the classic MNIST dataset, which consists of handwritten digits from 0 to 9.
To start, we need to list the possible target devices we can use for profiling. We need to pick from this list.
You should see a list printed such as:
A common option is the cortex-m4f-80mhz
, as this is a relatively low-power microcontroller family. From there, we can use the Edge Impulse Python SDK to generate a profile for your model to ensure it fits on your target hardware and meets your timing requirements.
Once you are happy with the performance of the model, you can deploy it to a number of possible hardware targets. To see the available hardware targets, run the following:
You should see a list printed such as:
The most generic target is to download a .zip file that holds a C++ library containing the inference runtime and your trained model, so we choose 'zip'
from the above list. To do that, we first need to create a Classification object which contains our label strings (and other optional information about the model). These strings will be added to the C++ library metadata so you can access them in your edge application.
Note that instead of writing the raw bytes to a file, you can also specify an output_directory
argument in the .deploy()
function. Your deployment file(s) will be downloaded to that directory.
Important! The deployment targets list will change depending on the values provided for model
, model_output_type
, and model_input_type
in the next part. For example, you will not see openmv
listed once you upload a model (e.g. using .profile()
or .deploy()
) if model_input_type
is not set to ei.model.input_type.ImageInput()
. If you attempt to deploy to an unavailable target, you will receive the error Could not deploy: deploy_target: ...
. If model_input_type
is not provided, it will default to OtherInput. See this page for more information about input types.
Your model C++ library should be downloaded as the file my_model_cpp.zip in the same directory as this notebook. You are now ready to use your C++ model in your embedded and edge device application! To use the C++ model for local inference, see our documentation here.
Amazon SageMaker Studio is an integrated development environment (IDE) that provides a single web-based visual interface where you can access purpose-built tools to perform all machine learning (ML) development steps, from preparing data to building, training, and deploying your ML models, improving data science team productivity by up to 10x. You can quickly upload data, create new notebooks, train and tune models, move back and forth between steps to adjust experiments, collaborate seamlessly within your organization, and deploy models to production without leaving SageMaker Studio.
To learn more about using the Python SDK, please see: Edge Impulse Python SDK Overview.
This guide has been built from AWS reference project Introduction to SageMaker TensorFlow - Image Classification, please have a look at this AWS documentation page.
Below are the changes made to the original training script and configuration:
The Python 3 (Data Science 3.0)
kernel was used.
The dataset has been changed to classify images as car
vs unknown
. You can download the dataset from this Edge Impulse public project and store it in your S3 bucket.
The dataset has been imported in the Edge Impulse S3 bucket configured when creating the SageMaker Studio domain. Make sure to adapt to your path or use the AWS reference project.
The training instance used is ml.m5.large
.
Install dependencies
Below is the structure of our dataset in our S3 bucket
We have used the default bucket created when configuring SageMaker Studio domain:
You can continue with the default model, or can choose a different model from the list. Note that this tutorial has been tested with MobileNetv2 based models. A complete list of SageMaker pre-trained models can also be accessed at Sagemaker pre-trained Models.
Optional, ship this next cell if you don't want to retrain the model. And uncomment the last line of the cell after
You will need to obtain an API key from an Edge Impulse project. Log into edgeimpulse.com and create a new project. Open the project, navigate to Dashboard and click on the Keys tab to view your API keys. Double-click on the API key to highlight it, right-click, and select Copy.
Note that you do not actually need to use the project in the Edge Impulse Studio. We just need the API Key.
Paste that API key string in the ei.API_KEY
value in the following cell:
Voila! You now have a C++ library ready to be compiled and integrated in your embedded targets. Feel free to have a look at Edge Impulse deployment options on the documentation to understand how you can integrate that to your embedded systems.
You can also have a look at the deployment page of your project to test your model on a web browser or test
Your Edge Impulse Organization enables your team to collaborate on multiple datasets, automation, and models in a shared workspace. It provides tools to automate data preparation tasks with reusable pipelines, enabling data transformation, preparation, and analysis of sensor data at scale. Allowing anyone in your team to quickly access relevant data through familiar tools, add versions and add traceability to your machine learning models, and lets you quickly create and monitor your Edge Impulse projects for optimal on-device performance.
Only available with Edge Impulse Enterprise Plan
Try our FREE Enterprise Trial today.
To get started, follow these guides:
User management - to add collaborators with different access rights.
Data campaigns - to track and visualize all your metrics over time.
Data - to connect a storage bucket, to learn how to deal with such complex data infrastructure and to import your data samples into your projects.
Data pipelines - to chain several transformation blocks and to import data into your projects.
Data transformations - to run your transformation blocks and get an overview of the running jobs.
Upload portals - to allow external parties to securely contribute data to your datasets.
Custom blocks - to match any specific use cases using dedicated cloud jobs.
Health reference design
We have built a health reference design that describes an end-to-end ML workflow for building a wearable health product using Edge Impulse. It is a good tutorial to understand how we handle complex data infrastructure and discover the organization's advanced features.
Existing enterprise users or enterprise trial users can view their entitlement limits via the dashboard of their enterprise organization:
This view allows you to see your organization's current usage of total users, projects, compute time and storage limits. To increase your organization's limits, select the Request limit increase button to contact sales.
🤗 Hugging Face offers a suite of tools that assist with various AI applications. Most notably, they provide a hub for people to share their pre-trained models. In this tutorial, we will demonstrate how to download a simple ResNet model from the Hugging Face hub, profile it, and convert it to a C++ library for use in your edge application. This particular model was trained to identify species of bean plants using the bean dataset.
To learn more about using the Python SDK, please see: Edge Impulse Python SDK Overview
You will need to obtain an API key from an Edge Impulse project. Log into edgeimpulse.com and create a new project. Open the project, navigate to Dashboard and click on the Keys tab to view your API keys. Double-click on the API key to highlight it, right-click, and select Copy.
Note that you do not actually need to use the project in the Edge Impulse Studio. We just need the API Key.
Paste that API key string in the ei.API_KEY
value in the following cell:
To download a model from the Hugging Face hub, we need to first find a model. Head to huggingface.co/models. On the left side, click Image Classification to filter under the Tasks tab and under the Libraries tab, filter by ONNX (as the Edge Impulse Python SDK easily accepts ONNX models). You should see the resnet-tiny-beans model trained by user fxmarty.
Click on the resnet-tiny-beans entry (or follow this link) to read about the model and view the files. If you click on the Files* tab, you can see all of the files available in this particular model.
Set the name of the repo (username/repo-name) and the file we want to download.
To start, we need to list the possible target devices we can use for profiling. We need to pick from this list.
You should see a list printed such as:
A common option is the cortex-m4f-80mhz
, as this is a relatively low-power microcontroller family. From there, we can use the Edge Impulse Python SDK to generate a profile for your model to ensure it fits on your target hardware and meets your timing requirements.
Once you are happy with the performance of the model, you can deploy it to a number of possible hardware targets. To see the available hardware targets, run the following:
You should see a list printed such as:
The most generic target is to download a .zip file containing a C++ library containing the inference runtime and your trained model, so we choose 'zip'
from the above list. We also need to tell Edge Impulse how we are planning to use the model. In this case, we want to perform classification, so we set the output type to Classification.
Note that instead of writing the raw bytes to a file, you can also specify an output_directory
argument in the .deploy()
function. Your deployment file(s) will be downloaded to that directory.
Your model C++ library should be downloaded as the file my_model_cpp.zip in the same directory as this notebook. You are now ready to use your C++ model in your embedded and edge device application! To use the C++ model for local inference, see our documentation here.
Weights & Biases is an online framework for helping manage machine learning training, data versioning, and experiments. When running experiments for edge-focused ML projects, it can be helpful to see the required memory (RAM and ROM) along with estimated inference times of your model for your target hardware. By viewing these metrics, you can quickly gauge if your model will fit onto your target device!
Follow the code below to see how to train a simple machine learning model with different hyperparameters and log those values to the Weights & Biases dashboard.
To learn more about using the Python SDK, please see: Edge Impulse Python SDK Overview
You will need to obtain an API key from an Edge Impulse project. Log into edgeimpulse.com and create a new project. Open the project, navigate to Dashboard and click on the Keys tab to view your API keys. Double-click on the API key to highlight it, right-click, and select Copy.
Note that you do not actually need to use the project in the Edge Impulse Studio. We just need the API Key.
Paste that API key string in the ei.API_KEY
value in the following cell:
To use Weights and Biases, you will need to create an account on wandb.ai and call the wandb.login()
function. This will prompt you to log in to your account. Your credentials should be stored, which allows you to use the wandb
package in your Python library.
We want to create a classifier that can uniquely identify handwritten digits. To start, we will use TensorFlow and Keras to train a very simple convolutional neural network (CNN) on the classic MNIST dataset, which consists of handwritten digits from 0 to 9.
We want to vary the hyperparameters in our model and see how it affects the accuracy and predicted RAM, ROM, and inference time on our target platform. To do that, we construct a function that builds a simple model using Keras, trains the model, and computes the accuracy and loss from our holdout test set. We then use the Edge Impulse Python SDK to generate a profile of our model for our target hardware. We log the hyperparameter (number of nodes in the hidden layer), test loss, test accuracy, estimated RAM, estimated ROM, and estimated inference time (ms) to our Weights and Biases console.
Now, it's time to run the experiment and log the results in Weights and Biases. Simply call our function and provide a new hyperparameter value for the number of nodes.
Head to wandb.ai and log in (if you have not already done so). Under My projects on the left, click on the nodes-sweep project. You can visualize the results of your experiments with the various charts that Weights & Biases offers. For example, here is a parallel coordinates plot that allows you to quickly visualize the different hyperparameters and metrics (including our new edge profile metrics).
If you would like to deploy your model to your target hardware, the Python SDK can help you with that, too. See our documentation here.
Once you are happy with the performance of your model, you can then deploy it to your target hardware. We will assume that 32 nodes in our hidden layer provided the best trade-off of RAM, flash, inference time, and accuracy for our needs. To start, we will retrain the model:
Next, we should evaluate the model on our holdout test set.
From there, we can see the available hardware targets for deployment:
You should see a list printed such as:
The most generic target is the .zip file that holds a C++ library containing our trained model and inference runtime. To pass our labels to the C++ library, we create a Classification object, which contains our label strings.
Note that instead of writing the raw bytes to a file, you can also specify an output_directory
argument in the .deploy() function. Your deployment file(s) will be downloaded to that directory.
Your model C++ library should be downloaded as the file my_model_cpp.zip in the same directory as this notebook. You are now ready to use your C++ model in your embedded and edge device application! To use the C++ model for local inference, see our documentation here.
Within an organization you can work on one or more projects with multiple people. These can be colleagues, outside researchers, or even members of the community. They will only get access to the specific data in the project, and not to any of the raw data in your organizational datasets.
Only available with Edge Impulse Enterprise Plan
Try our FREE Enterprise Trial today.
To invite a user in an organization, click on the "Add user button, enter the email address and select the role:
It is important to note that there are two types of users in Edge Impulse: Project Users and Organization Users.
Organization Users, typically holding roles like Admin, are responsible for the overarching management and customization of organizational elements, including datasets, storage buckets, and white label attributes. These users also encompass the capabilities of Project Users.
Conversely, Project Users, often in roles such as Member or Guest, are limited to specific project involvement, focusing on collaboration and contributions at the project level, without access to broader organizational management functions. They are granted access only to certain project data to maintain the security of raw data in organizational datasets.
For a more granular look at the capabilities of each role, see the table below:
Admins have full rights on the organization, overseeing organizational and white label functionalities, including dataset management and storage bucket updates. They also have all the rights of a Project Member.
Full Rights on the Organization
Project User rights
Manage organization datasets
Update and add storage buckets
Verify bucket connectivity
Customize white label (where applicable) attributes like themes and information
API access for organization and white label management
Members have full access on the datasets, custom blocks but cannot join a project without being invited.
Broad Access, with Restrictions on Project Joining
Project User rights
Full access to datasets and custom blocks
Can collaborate on projects, but only by invitation
Can access metrics via API
Guests have restricted access, limited to selected datasets within the projects they are associated with.
Limited Access to Selected Datasets
Project User rights
Access to selected datasets within the project they are invited to
Cannot access raw data in organizational datasets
Cannot access metrics via API
To give someone access to a project only, go to your project's dashboard, and find the "Collaborators" widget. Click the '+' icon, and type the username or e-mail address of the other user.
The EON Tuner is Edge Impulse's automated machine learning (AutoML) tool to help you find the best combination of blocks and hyperparameters for your model and within your hardware constraints. This example will walk you through uploading data, running the EON Tuner, and interpreting the results.
WARNING: This notebook will add and delete data in your Edge Impulse project, so be careful! We recommend creating a throwaway project when testing this notebook.
To start, create a new project in Edge Impulse. Do not add any data to it.
You will need to obtain an API key from an Edge Impulse project. Log into edgeimpulse.com and create a new project. Open the project, navigate to Dashboard and click on the Keys tab to view your API keys. Double-click on the API key to highlight it, right-click, and select Copy.
Note that you do not actually need to use the project in the Edge Impulse Studio. We just need the API Key.
Paste that API key string in the ei.API_KEY
value in the following cell:
We start by downloading the continuous motion dataset and uploading it to our project.
To start, we need to list the possible target devices we can use for profiling. We need to pick from this list.
You should see a list printed such as:
From there, we start the tuner with start_tuner()
and wait for completion via check_tuner()
. In this example, we configure the tuner to target for the cortex-m4f-80mhz device. Since we want to classify the motion, we choose classification
for our classifcation_type
and our dataset as motion continuous. We constrain our model to a latency of 100ms for running the impulse.
NOTE: We set the max trials to 3 here. In a real life situation, you will omit this so the tuner decides the best number of trials.
Once the tuner is done, you can print out the results to determine the best combination of blocks and hyperparameters.
To visualize the results of the tuner trials, you can head to the project page on Edge Impulse Studio.
Alternatively, you can access the results programmatically: the configuration settings and output of the EON Tuner is stored in the variable state
. You can access the results of the various trials with state.trials
. Note that some trials can fail, so it's a good idea to test the status of each trial.
From there, you will want to sort the results based on some metric. In this example, we will sort based on int8 test set accuracy from highest to lowest.
Note: Edge Impulse supports only one learning block per project at this time (excluding anomaly detection blocks). As a result, we will use the first learning block (e.g.
learning_blocks[0]
) in the list to extract metrics.
Now that we have the sorted results, we can extract the values we care about. We will print out the following metrics along with the impulse configuration (processing/learning block configuration and hyperparameters) of the top-performing trial.
This will help you determine if the impulse can fit on your target hardware and run fast enough for your needs. The impulse configuration can be used to recreate the processing and learning blocks on Edge Impulse. Later, we will set the project impulse based on the trial ID to simply deploy (rather than re-train).
Note: we assume the first learning block has the metrics we care about.
You can optionally use a plotting package like matplotlib to graph the results from the top results to compare the metrics.
If you have pandas installed, you can make the previous section much easier by reporting metrics as a DataFrame.
We can replace the current impulse with the top performing trial from the EON Tuner. From there, we can deploy it, just like we would any impulse.
You should see a list printed such as:
The most generic target is to download a .zip file that holds a C++ library containing the inference runtime and your trained model, so we choose 'zip'
from the above list. To do that, we first need to create a Classification object which contains our label strings (and other optional information about the model). These strings will be added to the C++ library metadata so you can access them in your edge application.
Note that instead of writing the raw bytes to a file, you can also specify an output_directory
argument in the .deploy()
function. Your deployment file(s) will be downloaded to that directory.
Important! The deployment targets list will change depending on the values provided for model
, model_output_type
, and model_input_type
in the next part. For example, you will not see openmv
listed once you upload a model (e.g. using .profile()
or .deploy()
) if model_input_type
is not set to ei.model.input_type.ImageInput()
. If you attempt to deploy to an unavailable target, you will receive the error Could not deploy: deploy_target: ...
. If model_input_type
is not provided, it will default to OtherInput. See this page for more information about input types.
Your model C++ library should be downloaded as the file my_model_cpp.zip in the same directory as this notebook. You are now ready to use your C++ model in your embedded and edge device application! To use the C++ model for local inference, see our documentation here.
By default, the EON Tuner will make a guess at a search space based on the type of data you uploaded (e.g. using spectral-analysis blocks for feature extraction). As a result, you can run the tuner without needing to construct a search space. However, you may want to define your own search space.
The best way to define a search space is to open your project (after uploading data), head to the EON Tuner page, click Run EON Tuner, and select the Space tab.
The search space is defined in JSON format, so we can just copy that to create a dictionary. This is a good place to start for tuning blocks and hyperparameters.
Note: Functions to get available blocks and search space parameters coming soon
Transformation blocks are a flexible way to pre-process, i.e. transform, your organizational data through transformation jobs before using it within a project or to create another dataset. They can also be used standalone (not operating on organizational data) as a way to run cloud processing jobs for specific actions.
You can use transformation blocks to fetch external datasets, augment and create variants of your data samples, extract metadata from config files, create helper graphs, align and interpolate measurements across sensors, remove duplicate entries, and more.
Transformation blocks can be run individually through transformation jobs, stacked together to run in series in data pipelines, or run as a data source (standalone mode only) importing data into a project. Please refer to the respective documentation for details.
Transformation blocks operate in one of three modes: file, directory, or standalone.
Transformation blocks that have been configured with the file operating mode will only appear in the block dropdown for transformation jobs.
As the name implies, file transformation blocks operate on files. The directory specified in the input data path field for the transformation job is automatically scanned for files. Any files that are found are shown on the right hand side as the selected items.
Transformation blocks that have been configured with the directory operating mode will only appear in the block dropdown for transformation jobs.
As the name implies, directory transformation blocks operate on directories. The directory specified in the input data path field for the transformation job is automatically scanned for directories. Any directories that are found are shown on the right hand side as the selected items.
Transformation blocks that have been configured with the standalone operating mode can appear in the block dropdown for transformation jobs, for project data sources, or both depending on how the block has been set up.
As the name implies, standalone transformation blocks do not operate on any files or directories. No dataset selection fields will be shown for standalone blocks when creating a transformation job. There will be no files or directories shown on the right hand side as the selected items.
Edge Impulse has the ability to incorporate pre-built transformation blocks into the platform and make these available for all organizations to use. Pre-built transformation blocks may be added over time as recurring needs or interests emerge.
As thees blocks are added to the platform, they will be found within your organization by going to the Transformation left sidebar menu item under Custom blocks. The pre-built blocks will be listed under the Public blocks section at the bottom of the transformation blocks overview page.
Please refer to the custom transformation blocks documentation for details.
Building data pipelines is a very useful feature where you can stack several transformation blocks similar to the Data sources pipelines. They can be used in a standalone mode (just execute several transformation jobs in a pipeline), to feed a dataset or to feed a project.
Only available with Edge Impulse Professional and Enterprise Plans
Try our Professional Plan or FREE Enterprise Trial today.
The examples in the screenshots below shows how to create and use a pipeline to create the 'AMS Activity 2022' dataset.
To create a new pipeline, click on '+Add a new pipeline:
In your organization workspace, go to Custom blocks -> Transformation and select Run job on the job you want to add.
Select Copy as pipeline step and paste it to the configuration json file.
You can then paste the copied step directly to the respected field.
Below, you have an option to feed the data to either a organisation dataset or an Edge Impulse project
By default, your pipeline will run every day. To schedule your pipeline jobs, click on the ⋮
button and select Edit pipeline.
Once the pipeline has successfully finished, it can send an email to the Users to notify.
Once your pipeline is set, you can run it directly from the UI, from external sources or by scheduling the task.
To run your pipeline from Edge Impulse studio, click on the ⋮
button and select Run pipeline now.
To run your pipeline from Edge Impulse studio, click on the ⋮
button and select Run pipeline from code. This will display an overlay with curl
, Node.js
and Python
code samples.
You will need to create an API key to run the pipeline from code.
Another useful feature is to create a webhook to call a URL when the pipeline has ran. It will run a POST request containing the following information:
Edge Impulse makes it easy to access data that you have stored in the cloud by offering integrations with several storage providers and the flexibility to connect a storage solution to an organization or directly to a project.
There are two locations within Edge Impulse where you can connect to cloud storage, from within an organization or within a project. These options are described below. For details related to the specific storage provider integration options available, please see the Storage provider integrations section of this document.
Cloud storage can be connected to an organization. By connecting your data to an organization, you are offered the flexibility to pre-process your datasets through the use of transformation blocks and to feed your datasets into multiple projects.
To connect, access your organization, select Data in the left sidebar menu, select the Buckets tab at the top of the page, then click the + Add new bucket
button. Follow the instructions in the modal window that pops up.
Cloud storage can be connected directly to a project. To connect, access your project, select Data acquisition in the left sidebar menu, select the Data sources tab at the top of the page, then click the + Add new data source
. Follow the instructions in the modal window that pops up.
Note that some options in the modal will be greyed out if your project is not on the Enterprise plan.
Edge Impulse allows you to integrate with several cloud storage options. These include:
To connect to an Amazon S3 bucket, you will need to provide:
The bucket name
The bucket region
An access key
A secret key
A path prefix (optional)
If the credentials provided do not have access to the root of the bucket, the prefix is used to specify the path for which the credentials are valid.
Currently only long-term credentials from AWS IAM users are supported; temporary credentials provided to AWS SSO users are not supported.
For Amazon S3 buckets, you will also need to enable CORS headers for the bucket. You can do this in the S3 console by going to your bucket, going to the permissions tab, and then adding the policy defined below to the cross-origin resource sharing section.
CORS policy (console):
Alternatively, you can save the below CORS policy as a cors.json
file (note there are some differences in the structure compared to the JSON above) and add it to your bucket using the AWS S3 CLI.
CORS policy (CLI):
To connect to a Google Cloud Storage bucket, you will need to provide:
The bucket name
The bucket region
An access key
A secret key
A path prefix (optional)
If the credentials provided do not have access to the root of the bucket, the prefix is used to specify the path for which the credentials are valid.
For Google Cloud Storage buckets, you will also need to enable CORS headers for the bucket. You cannot manage CORS policies using the Google Cloud console; you must use the gcloud CLI instead.
CORS policy:
Save the above CORS policy as a cors.json
file and add it to your bucket with the gcloud CLI using the following command:
gsutil is not the recommended CLI tool for Cloud Storage. You may have used this tool before, however, Google now recommends using gcloud storage commands in the Google Cloud CLI instead.
To connect to a Microsoft Azure Blob Storage blob container, you will need to provide:
The blob container name
The storage account name
A secret key
A path prefix (optional)
If the credentials provided do not have access to the root of the blob container, the prefix is used to specify the path for which the credentials are valid.
A CORS policy is not required with Microsoft Azure Blob Storage.
To connect to another (S3-compatible) type of bucket, you will need to provide:
The bucket name
The bucket region
The bucket endpoint
An access key
A secret key
A path prefix (optional)
If the credentials provided do not have access to the root of the bucket, the prefix is used to specify the path for which the credentials are valid.
For other (S3-compatible) buckets, you will also need to enable CORS headers for the bucket. Please refer to your provider documentation for instructions on how to do so.
The items that you will need to set are the following:
Origin: ["https://studio.edgeimpulse.com"]
Method: ["PUT", "POST"]
Header: ["*"]
For cloud storage integration to work as expected, Edge Impulse needs to be provided with credentials that allow read, write, and delete operations. Please refer to your storage provider documentation for specifics.
In order to verify the connection to the cloud storage provider, Edge Impulse will write an .ei-portal-check
file that will be subsequently deleted. Once a bucket is successfully connected to your organization, a green dot will appear in the connected column on the buckets overview page.
Data transformation or transformation jobs refer to processes that apply specific transformations to the data within an Edge Impulse organizational dataset. These jobs are executed using Transformation blocks, which are essentially scripts packaged in Docker containers. They perform a variety of tasks on the data, enabling more advanced and customized dataset transformation and manipulation.
The transformation jobs can be chained together in Data pipelines to automate your workflows.
Only available with Edge Impulse Enterprise Plan
Try our FREE Enterprise Trial today.
You have several options to create a transformation job:
From the Data transformation page by selecting the Create job tab.
From the Custom blocks->Transformation page by selecting the "⋮" action button and selecting Run job.
From the Data page:
Depending on whether you are on a Default dataset or a Clinical dataset, the view will vary:
Again, depending on whether you are on a Default dataset or a Clinical dataset, the view will vary. The common options are the Name of the transformation job, the Transformation block used for the job.
If your Transformation block has additional custom parameters, the input fields will be displayed below in a Parameters section. For example:
Dataset type options:
Default vs. Clinical datasets
Clinical Datasets: Operate on "data items" with a strict file structure. Transformation is specified using SQL-like syntax.
Default Datasets: Resemble a typical file system with flexible structure. You can specify data for transformation using wildcards.
For more information about the two dataset types, see the dedicated Data page.
Input
After selecting your Input dataset, you can filter which files or directory you want to transform.
In default dataset formats, we use wildcard filters (in a similar format to wildcards in git). This enable you to specify patterns that match multiple files or directories within your dataset:
Asterisk ( * ): Represents any number of characters (including zero characters) in a filename or directory name. It is commonly used to match files of a certain type or files whose names follow a pattern.
Example: /folder/*.png
matches all PNG files in the /folder
directory.
Example: /data/*/results.csv
matches any results.csv file in a subdirectory under /data
.
Double Asterisk ( ** ): Used to match any number of directories, including nested ones. This is particularly useful when the structure of directories is complex or not uniformly organized.
Example: /data/**/experiment-*
matches all files or directories starting with experiment-
in any subdirectory under /data
.
Output
When you work with default datasets in Edge Impulse, you have the flexibility to define how the output from your transformation jobs is structured. There are three main rules to choose from:
No Subfolders: This rule places all transformed files directly into your specified output directory, without creating any subfolders. For example, if you transform .txt
files in /data
and choose /output
as your output directory, all transformed files will be saved directly in /output
.
Subfolder per Input Item: Here, a new subfolder is created in the output directory for each input file or folder. This keeps the output from each item organized and separate. For instance, if your input includes folders like /data/2020
, /data/2021
, and /data/2022
, and you apply this rule with /transformed
as your output directory, you will get subfolders like /transformed/2020
, /transformed/2021
, and /transformed/2022
, each containing the transformed data from the corresponding input year.
Use Full Path: This rule mirrors the entire input path when creating new sub-folders in the output directory. It's especially useful for maintaining a clear trace of where each piece of output data originated, which is important in complex directory structures. For example, if you're transforming files in /project/data/experiments
, and you choose /results
as your output directory, the output will follow the full input path, resulting in transformed data being stored in /results/project/data/experiments
.
Note: For the transformation blocks operating on files when selecting the Subfolder or Full Path option, we will use the file name without extension to create the base folder. e.g. /activity-detection/Accelerometer.csv
will be uploaded to /activity-detection-output/Accelerometer/
.
Input
When running transformation jobs using the Clinical dataset option, you can query your input files or folders in all your clinical datasets. We use a different filtering mechanism for the Clinical datasets.
Filters
You can use a language which is very similar to SQL (documentation). See more on how to query your data on the dedicated documentation page. For example you can use filters like the following:
dataset = 'Activity Detection (Clinical view)' AND file_name like 'Accelero%'
dataset = 'Activity Detection (Clinical view)' AND metadata->ei_check = 1
Import into project
Import into dataset
Number of parallel jobs
For transformation jobs operating on Data items (directory) or on Files, you can edit the number of parallel jobs to run simultaneously
Users to notify
Finally, you can select users you want to notify over email when this job finishes.
Since the creation of Edge Impulse, we have been helping customers to deal with complex data pipelines, complex data transformation methods and complex clinical validation studies.
In most cases, before even thinking about machine learning algorithms, researchers need to build quality datasets from real-world data. These data come from various devices (prototype devices being developed vs clinical/industrial-grade reference devices), have different formats (excel sheets, images, csv, json, etc...), and be stored in various places (researchers' computers, Dropbox folders, Google Drive, S3 buckets, etc...).
Dealing with such complex data infrastructure is time-consuming and expensive to develop and maintain. With the organizational data, we want to give you tools to centralize, validate and transform datasets so they can be easily imported into your projects to train your machine learning models.
Only available with Edge Impulse Enterprise Plan
Try our FREE today.
Health reference design
We have built a that describes an end-to-end ML workflow for building a wearable health product using Edge Impulse.
In this reference resign, we want to help you understand how to create a full clinical data pipeline by using a public dataset from the repository. This tutorial will guide you through the following steps:
Before we get started, you must link your organization with one or more storage buckets. Further details about how to integrate with cloud storage providers can be found in the document.
Two types of dataset structures can be used - Generic datasets (default) and Clinical datasets.
There is no required format for data files. You can upload data in any format, whether it's CSV, Parquet, or a proprietary data format.
However, to import data items to an Edge Impulse project, you will need to use the right format as our studio ingestion API only supports these formats:
JPG, PNG images
MP4, AVI video files
WAV audio files
CSV files
The default dataset structure is a file-based one, no matter the directory structure:
For example:
or:
Note that you will be able to associate the labels of your data items from the file name or the directory name when importing your data in a project.
The clinical datasets structure in Edge Impulse has three layers:
The dataset, a larger set of data items, grouped together.
Data item, an item with metadata and files attached.
Data file, the actual files.
Once you successfully linked your storage bucket to your organization, head to the Datasets tab and click on + Add new dataset:
Fill out the following form:
Click on Create dataset
Default view
The default view lets you navigate in your bucket following the directory structure. You can easily add data using the "+ New folder" button. To add new data, use the right panel - drag and drop your files and folders and it will automatically upload them to your bucket.
Clinical view
Tip: You can add two distinct datasets in Edge Impulse that point to the same bucket path, one generic and one clinical. This way you can leverage both the easy upload and the ability to query your datasets.
Go to the Actions...->Import data into a project, select the project you wish to import to and click Next, Configure how to label this data:
This will import the data into the project and optionally create a new label for each file in the dataset. This labeling step helps you keep track of different classes or categories within your data.
We also have added a data preview feature, allowing you to visualize certain types of data directly within the organization data tab.
Supported data types include tables (CSV/Parquet), images, PDFs, audio files (WAV/MP3), and text files (TXT/JSON). This feature gives you a quick overview of your data and helps ensure its integrity and correctness.
The "data campaigns" feature allows you to quickly track your experiments and your models' development progresses. It is an overview of your pipelines where you can easily extract useful information from your datasets and correlate those metrics with your model performances.
It has been primarily designed to follow clinical research data processes. In August 2023, we released this feature for every enterprise user as we see value in being able to track metrics between your datasets and your projects.
Only available with Edge Impulse Enterprise Plan
Try our FREE today.
To get started, navigate to the Data campaigns tab in your organization:
Click on + Create new dashboard.
Give your dashboard a name, and select one or more collaborators to receive the daily updates by email. If you don't want to be spammed, you can select when you want to receive these updates, either Always, On new data, changes or on error, or Never. Finally, set the last number of days shown in the graphs:
You can create as many dashboards as needed, simply click on + Create a new dashboard from the dropdown available under your current dashboard:
If you want to delete a dashboard, Click on Actions... -> Delete dashboard
Once your dashboard is created, you can add your custom campaigns. It's where you will specify which metrics are important to you and your use case. Click on Actions... -> Add campaign
Fill the form to create your campaign:
Name: Name of your data campaign.
Description: Description of your data campaign.
Campaign coordinators: Add the collaborators that are engaged with this campaign
Datasets: Select the datasets you want to visualize in your campaign. You can add several datasets.
Projects: Select the projects you want to visualize in your campaign. You can add several projects.
Pipelines: Select the pipeline that is associated with your campaign. Note that this is for reference only, it is currently not displayed in your campaign
Links: Select between the link type you need. Options are Github, Spreadsheet, Text Document, Code repository, List or Folder. Add a name and the link. This place is useful for other collaborators to have all the needed information about your project, gathered in one place under your campaign.
You can then save your data campaign and it will be added to your dashboard:
If you want to edit or delete your campaign, click on the "⋮" button on the right side of your campaign:
Custom deployment blocks are a way to extend the capabilities of Edge Impulse beyond the options built into the platform. If none of the existing blocks created by Edge Impulse fit your needs, you can create custom deployment blocks to build and export your own libraries or firmware binaries for unique project requirements.
Ready to dive in and start building? Jump to the !
The deployment block structure is shown below. Please see the overview page for more details.
The sections below define the required and optional inputs and the expected outputs for custom deployment blocks.
Deployment blocks have access to command line arguments and input files.
The following arguments will be automatically passed to your custom deployment block.
CLI arguments can also be specified using the cliArguments
property in the parameters.json
file. Alternatively, these arguments can be added (or changed) by editing the block in Studio.
Your deployment block will be passed an input directory that contains all the information required for a deployment, including: deployment metadata, the Edge Impulse SDK, the trained model (in multiple formats), and all supporting source code to run the impulse.
The input directory path is stored in the input
property under the folders
property in the deployment-metadata.json
file, which can be loaded using the --metadata <file>
argument that is passed to the deployment block.
The input directory structure is shown below.
The expected output from your custom deployment block is a ZIP file named deploy.zip
located in the output directory. This archive is what will be downloaded for the user after your block has finished building.
The output directory path is stored in the output
property under the folders
property in the deployment-metadata.json
file, which can be loaded using the --metadata <file>
argument that is passed to the deployment block.
The input and output directories listed in the deployment-metadata.json
file are located on network storage. Therefore to improve the speed of your deployment block, it is best practice to create a build directory, copy in the required items for your build, then write the output archive to the output directory.
In the example below, the app_dir
contained the build instructions and files required to compile a Linux application.
If your custom deployment block requires access to the data used to train your model, you can mount the learning block by setting the mountLearnBlock
property to true
. This will mount all files for the learning block, including the training data, under a /data
directory within your block.
Deployment blocks do not have access to the internet by default. If you need to access information outside of your block, such as project items through the Edge Impulse API, you will need to set the privileged
property to true
.
Setting the showOptimizations
property to true
will present the user with additional optimization options on the Deployment page in Studio.
Firstly, if the supportsEonCompiler
property is set to true
(see below), the user will be presented with a dropdown to select between building the deployment using the EON Compiler or standard TFLite file inputs.
Secondly, the user will be presented with quantization options, if applicable. If the user selects the quantized model option, the trained.tflite
file will be the int8
version of the model; otherwise it will be the float32
version.
However, if the showOptimizations
property is set to true
(see above), the user will have the option on the Deployment page in Studio to select between the EON Compiler or standard TFLite file inputs.
The default image that will appear for your block in the dropdown in Studio on the Deployment page is the Edge Impulse logo. If you would like to change this, you can do so by editing the block after it has been pushed to Studio.
Testing locally does not mount the learning block
If your custom deployment block requires access to the learning block files after it has been mounted, testing locally will not work as the methods to download data described below do not include the learning block data.
To speed up your development process, you can test your custom deployment block locally. There are two ways to achieve this. You will need to have Docker installed on your machine for either approach.
The first time you enter the above command, you will be asked some questions to configure the runner. Follow the prompts to complete this. If you would like to change the configuration in future, you can execute the runner command with the --clean
flag.
Using the above approach will create an ei-block-data
directory within your custom block directory. It will contain several subdirectories.
For the second method, you can use the CLI block runner or Studio to download the required data from your project, then build the Docker image and run the container directly.
You can download the data by calling the block runner with the --download-data <dir>
argument. The directory specifies the location where the downloaded data should be extracted. To make this work properly the directory needs to be named input/
. Before extraction, the data archive will first be downloaded to ei-block-data/download/
.
Alternatively, you can go to the Deployment page for your project in Studio and select Custom block
as your deployment option. This will allow you to download a ZIP file of the required input files for you deployment block. Extract this archive to a directory called input/
within your custom deployment block directory.
After downloading the required input files for your block, you can then build the Docker image and run the container.
Below are direct links to a some examples:
Custom learning blocks are a way to extend the capabilities of Edge Impulse beyond the built into the platform. If none of the existing blocks created by Edge Impulse fit your needs, you can create custom learning blocks to integrate your own model architectures for unique project requirements.
Ready to dive in and start building? Jump to the !
Custom learning blocks are available for all users
Unlike other custom blocks, which are only available to customers on the Enterprise plan, custom learning blocks are available to all users of the platform. If you are an enterprise customer, your custom learning blocks will be available in your organization. If you are not an enterprise customer, your custom learning blocks will be available in your developer profile.
Expert mode
If you only want to make small modifications to the neural network architecture or loss function, you can instead use directly in Studio, eliminating the need to create a custom learning blocks. Go to any learning block settings page, select the three dots, and select Switch to Keras (expert) mode.
The learning block structure is shown below. Please see the overview page for more details.
The sections below define the required and optional inputs and the expected outputs for custom learning blocks.
Learning blocks have access to command line arguments and training data.
In addition to the items defined by you, the following arguments will be automatically passed to your custom learning block.
The X_*.npy
files are float32 arrays in the appropriate shape. You can typically load these into your training pipeline without any modification.
The Y_*.npy
files are int32 arrays with four columns: label_index
, sample_id
, sample_slice_start_ms
, and sample_slice_end_ms
, unless the labels are bounding boxes. See below.
The X_*.npy
files follow the NHWC (batch_size, height, width, channels) format for image data.
The Y_*.npy
files are a JSON array in the form of:
Image data is formatted as NHWC
If you need your data in the channels-first, NCHW format, you will need to transpose the input data yourself before training your model.
Image data is provided to your custom learning block in the NHWC (batch_size, height, width, channels) format. If you are training a PyTorch model that requires data to be in the NCHW (batch_size, channels, height, width) format, you will need to transpose the data before training your model.
You do not need to worry about this when running on device. As long as your custom learning block outputs an ONNX model, the required transpose will be handled for you in the Edge Impulse SDK.
Image data is formatted as RGB
If you have a model that requires BGR input, you will need to transpose the first and last channels.
For models that require BGR channel format, you can have Edge Impulse automatically transpose the first and last channels by selecting the RGB->BGR
option when configuring pixel scaling for your block. See below.
Image data has pixels that are already scaled
There is no need to scale the pixel values yourself for training nor for inference on-device. If the options provided in Edge Impulse do not suit your needs, please contact us to let us know what option(s) you require.
Image data is provided to your learning block with pixels that are already scaled. Pixel scaling is handled automatically by Edge Impulse. There are several options to scale your pixels, some of which include additional processing (e.g. standardization or centering):
Pixels ranging 0..1 (not normalized)
Pixels ranging -1..1 (not normalized)
Pixels ranging -128..127 (not normalized)
Pixels ranging 0..255 (not normalized)
PyTorch (pixels ranging 0..1, then standardized using ImageNet mean/std)
RGB->BGR (pixels ranging 0..255, then centered using ImageNet mean)
This can be configured when initializing your custom learning block with the Edge Impulse CLI, and changed later in Studio if required by editing your custom learning block.
The expected output from your custom learning block is one of TFLite, ONNX, or pickled scikit-learn files.
For object detection models, it is also important to ensure that the output layer of your model is supported by Edge Impulse.
TFLite file(s):
model.tflite
- a TFLite file with float32 inputs and outputs
model_quantized_int8_io.tflite
- a quantized TFLite file with int8 inputs and outputs
saved_model.zip
- a TensorFlow saved model
At least one of the above file options is required.
ONNX file:
model.onnx
- an ONNX file with float16 or float32 inputs and outputs
Edge Impulse automatically converts this file to both unquantized and quantized TFLite files after training.
Pickled scikit-learn file:
model.pkl
- a pickled instance of the scikit-learn model
Internally Edge Impulse uses scikit-learn==1.3.2 for conversion, so pin to this scikit-learn version for best results. LightGBM (3.3.5) and XGBOOST (1.7.6) models are also supported.
Unfortunately object detection models typically don't have a standard way to go from neural network output layer to bounding boxes. Currently Edge Impulse supports the following types of output layers. The most up-to-date list can be found in the API documentation for ObjectDetectionLastLayer
.
FOMO
MobileNet SSD
NVIDIA TAO RetinaNet
NVIDIA TAO SSD
NVIDIA TAO YOLOv3
NVIDIA TAO YOLOv4
YOLOv2 for BrainChip Akida
YOLOv5 (coordinates scaled 0..1)
YOLOv5 (coordinates in absolute values)
YOLOv7
YOLOX
After pushing your custom learning block to Edge Impulse, in Studio you will notice that below the section of custom parameters that you have exposed for your block, there is another section titled "Advanced training settings". These settings allow you to optionally adjust the train/validation split, split on a metadata key, and profile the int8 version of your model.
Most learning blocks built in the Edge Impulse (e.g. classifier, regression, or FOMO blocks) can be edited locally and then pushed back to Edge Impulse as a custom block. This is great if you want to make heavy modifications to these training pipelines, for example to do custom data augmentation. To download a block, go to any learning block settings page in your project, click the three dots, and select Edit block locally. Once downloaded, follow the instructions in the README file.
The train_input.json
file is not available when training locally
If your script needs information that is contained within train_input.json
, you will not be able to train locally. You will either need to push your block to Edge Impulse to train and test in Studio or alter your training script such that you can pass in that information (or eliminate it all together).
To speed up your development process, you can test and train your custom learning block locally. There are two ways to achieve this. You will need to have Docker installed on your machine for either approach.
For the additional arguments, you will need to provide the data directory (/home
), an output directory (e.g. /home/out
), and any other parameters required for your script.
Using the above approach will create an ei-block-data
directory within your custom block directory. It will contain a subdirectory with the associated project ID as the name - this is the directory that gets mounted into the container as /home
.
The first time you enter the above command, you will be asked some questions to configure the runner. Follow the prompts to complete this. If you would like to change the configuration in future, you can execute the runner command with the --clean
flag.
For the second method, you can use the block runner to download the required data from your project, then build the Docker image and run the container directly. The advantage of this approach is that you do not need to go through the feature generation and data splitting process each time you want to train your block. If your data changes, you can download it again.
Below are direct links to some examples:
Custom AI labeling blocks are a way to extend the feature within Edge Impulse. If none of the blocks created by Edge Impulse that are built into the platform fit your needs, you can modify them or develop from scratch to create a custom AI labeling block. This allows you to integrate your own models or prompts for unique project requirements.
Ready to dive in and start building? Jump to the !
AI labeling blocks are an extension of transformation blocks operating in standalone
mode and, as such, follow the same structure without being able to pass a directory or file directly to your scripts. Please see the overview page for more details.
The sections below define the required and optional inputs and the expected outputs for custom AI labeling blocks.
AI labeling blocks have access to environment variables, command line arguments, and mounted storage buckets.
The following environment variables are accessible inside of AI labeling blocks. Environment variable values are always stored as strings.
You can also define your own environment variables to pass to your custom block using the requiredEnvVariables
key in the metadata section of the parameters.json
file. You will then be prompted for the associated values for these keys when pushing the block to Edge Impulse using the CLI. Alternatively, these values can be added (or changed) by editing the block in Studio.
In addition to the items defined by you, specific arguments will be automatically passed to your AI labeling block.
Along with the transformation block arguments, the following AI labeling specific arguments are passed as well.
Additional CLI arguments can also be specified using the CLI arguments field when editing the block in Studio.
The mount point can be changed by editing the block in Studio after pushing.
There are no required outputs from AI labeling blocks. In general, all changes are applied to data using API calls inside the block itself.
AI labeling blocks can run in "preview" mode, which is triggered when a user clicks Label preview data
within an AI labeling action configuration. When a user is previewing label changes, the changes are staged and not applied directly.
For preview mode, the --propose-actions <job-id>
argument is passed into your block. When you see this option, you should not apply changes directly to the data samples (e.g. via raw_data_api.set_sample_bounding_boxes
or raw_data_api.set_sample_structured_labels
) but rather use the raw_data_api.set_sample_proposed_changes
API call.
AI labeling blocks are not supported by the edge-impulse-blocks runner
CLI tool
AI labeling blocks are not currently supported by the blocks runner in the Edge Impulse CLI. To test you custom AI labeling block, you will need to build the Docker image and run the container directly. You will need to pass any environment variables or command line arguments required by your script to the container when you run it.
Below are direct links to some examples:
Upload portals are a secure way to let external parties upload data to your datasets. Through an upload portal they get an easy user interface to add data, but they have no access to the content of the dataset, nor can they delete any files. Data that is uploaded through the portal can be stored on-premise or in your own cloud infrastructure.
In this tutorial we'll set up an upload portal, show you how to add new data, and how to show this data in Edge Impulse for further processing.
Only available with Edge Impulse Enterprise Plan
Try our FREE today.
Data is stored in cloud storage. For details on how to connect a cloud storage provider to Edge Impulse, refer to the documentation.
With your storage bucket configured you're ready to set up your first upload portal. In your organization go to Data > Upload portals and choose Create new upload portal. Here, select a name, a description, the storage bucket, and a path in the storage bucket.
After your portal is created a link is shown. This link contains an authentication token, and can be shared directly with the third party.
Click the link to open the portal. If you ever forget the link: no worries. Click the ⋮
next to your portal, and choose View portal.
To upload data you can now drag & drop files or folders to the drop zone on the right, or use Create new folder to first create a folder structure. There's no limit to the amount of files you can upload here, and all files are hashed, so if you upload a file that's already present the file will be skipped.
Note: Files with the same name but with a different hash are overwritten.
Mount the portal directly into a transformation block via Custom blocks > Transformation blocks > Edit block, and select the portal under mount points.
Here's a Python script which uploads, lists and downloads data to a portal. To upload data you'll need to authenticate with a JWT token, see below this script for more info.
And here's a script to generate JWT tokens:
Custom processing blocks are a way to extend the capabilities of Edge Impulse beyond the built into the platform. If none of the existing blocks created by Edge Impulse fit your needs, you can create custom processing blocks to implement your own feature generation algorithms for unique project requirements.
Ready to dive in and start building? Jump to the !
Hosting custom processing blocks in Edge Impulse is only available on the Enterprise plan
Hosting a custom processing block in the Edge Impulse infrastructure, and making it available to everyone in your organization, is only available on the Enterprise plan. Other developers can host their custom processing block themselves and expose it to projects. See in this document.
The processing block structure is shown below. A key difference for processing blocks versus other types of blocks is that they implement an HTTP server within the application. Please see the overview page for more details.
Processing blocks are expected to implement an HTTP server to handle requests. The sections below define the required and optional inputs (requests) and the expected outputs (responses) for custom processing blocks.
Information will be provided to your custom processing block through the request headers and body.
The request body adheres to the following interfaces for the POST
methods. GET
methods do not have a request body.
The data samples that need to be processed by your script to generate features are provided as arrays in the features
property.
The axes
property provides the names of the signals for the data sample. For example, if this was an accelerometer data sample, the axes could be [ 'accX', 'accY', 'accZ' ]
. These names could be mapped to other names in the named_axes
property.
The expected response from the HTTP server in your custom processing block varies depending on the type of request.
GET /
:
A plain text response with some information about the block. For example, the response could be the block name and author.
GET /parameters
:
The parameters file returned as a JSON object.
The POST response bodies are expected to adhere to the following interfaces.
The features
property is where you return the features that were generated by processing the data sample(s).
The labels
property can be used return the names of the features you generated. For example, if you calculated the average, maximum, and minimum values of the signal, the labels could be [ 'Average', 'Maximum', 'Minimum' ]
. These labels will be used for the feature explorer.
Graphs can be of different types: linear, logarithmic, or an image. The type of graph is controlled by the type
property of a graph object.
The results of generating features on all samples can be shown in the feature explorer. If you output high-dimensional data, you can enable dimensionality reduction for the feature explorer. This will run UMAP over the data to compress the features into two dimensions. To do so, you can set the visualization
property in your parameters.json
file to dimensionalityReduction
.
The port you publish for your Docker container needs to match the port you set with the EXPOSE
instruction in your Dockerfile. If no port number is provided, the blocks runner will look for the EXPOSE
instruction in your Dockerfile and publish that port for you.
Note down the public URL that is returned in the terminal. You can use this URL to add the block to an impulse in a Studio project.
For the second method, you can run Docker and ngrok directly instead of using the CLI blocks runner tool. First, you can build the Docker image and run the container.
Then, after signing up for and installing ngrok, you can use their CLI to create a public forwarding URL. Note down the https://
forwarding address in the response. You can use this URL to add the block to an impulse in a Studio project.
With a public URL for your custom processing block, you can go into your project and add a processing block to your impulse. When the processing block selection modal pops up, go to the bottom left corner and click the Add custom block
button. In the next modal that pops up, enter your forwarding URL from above and save. The block can now be used in your project and you will be able to view the processing results, including any visualizations you have created.
One caveat for custom processing blocks is that Edge Impulse cannot automatically generate optimized code to run on-device as is done with processing blocks built into the platform. This code will need to be written by you. To help you get started, the structure is provided for you.
After exporting the C++ library from the Deployment page in Studio, you can see that a forward declaration for your custom processing block will have been created for you in the model-parameters/model_variables.h
file.
The name for the function comes from the cppType
property in your parameters.json
file.
Below are direct links to some examples:
Much functionality in Edge Impulse is based on the concept of blocks. There are existing blocks built into the platform to achieve dedicated tasks. If these pre-built blocks do not fit your needs, you can edit existing blocks or develop from scratch to create custom blocks that extend the capabilities of Edge Impulse. These include:
The sections below provide an overview of custom blocks. The details for each specific type of block can be found on its own documentation page linked above.
A block in Edge Impulse encapsulates a Docker image and provides information to the container when it is run. Different parameters, environment variables, and data will be passed in and different volumes will be mounted depending on the type of block.
The basic structure of a block is shown below. At a minimum, a custom block consists of a directory containing your scripts, a Dockerfile, and a parameters.json
file. Block specific structures are shown in their respective documentation.
The Docker container executes the scripts that you have written for your custom block. At Edge Impulse, block scripts are mostly written in Python, Javascript/Typescript, or Bash. However, these scripts can be written in any language that you are comfortable with.
The initial script that you would like to be executed is defined in the Dockerfile as the ENTRYPOINT
for the image.
Do not set the WORKDIR
argument to /home
or /data
The /home
and /data
directory paths are used by Edge Impulse. Therefore, if you set the working directory for your container to this path, your files will be overwritten and rendered inaccessible. You will notice in most examples from Edge Impulse, the argument for the WORKDIR
instruction is set to /app
.
Use the ENTRYPOINT
instruction
It is important to set the ENTRYPOINT
instruction at the end of your Dockerfile to specify the default executable for the container. This instruction is used to turn a container into a standalone executable and blocks in Edge Impulse have been designed with this in mind.
Do not use the RUN
or CMD
instructions to set the default executable. The RUN
instruction is not meant for this purpose (it's meant for building layers of an image) and the CMD
instruction is not what Edge Impulse expects.
The Dockerfile is the instruction set for building the Docker image that will be run as a container in your custom block. The documentation for each type of custom block contains links to GitHub repositories for block examples, which each contain a Dockerfile. Referencing these is a great starting point when developing your own Dockerfile.
In general, the argument you define as the ENTRYPOINT
in your Dockerfile will be your custom script. For processing blocks, however, this will be an HTTP server. In this case, you will also need to expose the port for your server using the EXPOSE
instruction.
When running in Edge Impulse, processing and learning block containers do not have network access. Make sure you don't download dependencies while running these containers, only when building the images.
In most cases, the parameter items defined in your parameters.json
file are passed to your script as command line arguments. For example, a parameter named custom-param-one
with an associated value will be passed to your script as --custom-param-one <value>
.
Processing blocks are handled differently. In the case of processing blocks, parameter items are passed as properties in the body of an HTTP request. In this case, a parameter named custom-param-one
with an associated value will be passed to the function generating features in your script as an argument named custom_param_one
. Notice the dashes have been converted to underscores.
One additional note in regards to how parameter items are passed is that items of the type secret
will be passed as environment variables instead of command line arguments.
Parameter types are enforced and validation is performed automatically when values are being entered in Studio.
From within your custom block directory, run the edge-impulse-blocks init
command and follow the prompts to initialize your block. This will do two things:
Create an .ei-block-config
file that associates the block with your organization
Create a parameters.json
file (if one does not already exist in your custom block directory)
After the parameters.json
file is created, you will want to take a look at it and make modifications as necessary. The CLI creates a basic file for you and you may want to include additional metadata and parameter items.
There are several levels of testing locally that you can do while developing your custom block:
Calling your script directly, passing it any required environment variables and arguments
Building the Docker image and running the container directly, passing it any required environment variables and arguments
Using the blocks runner tool in the Edge Impulse CLI to test the complete block
Refer to the documentation for your type of custom block for additional details about testing locally.
Custom learning blocks can be pushed to a developer profile
Unlike all other types of custom blocks, a custom learning block can be pushed to a developer profile (non-Enterprise plan account).
After initializing and testing your custom block, you can push it to Edge Impulse to make it available for use by everyone in your organization.
From within your custom block directory, run the edge-impulse-blocks push
command and follow the prompts to push your block to Edge Impulse.
If at some point you need to change configuration settings for your block that aren't being shown when you run the edge-impulse-blocks
commands, say to download data from a different project with the runner
, you can execute any of the respective commands with the --clean
flag.
If you have previously created a Docker image for a custom block and are hosting it on Docker Hub, you can create a custom block that uses this image.
To do so, go to your organization and select the item in the left sidebar menu for the type of custom block you would like to create. On that custom block page, select the + Add new <block-type> block
button (or select an existing block to edit). In the modal that pops up, configure your block as desired and in the Docker container field enter the details for your image in the username/image:tag
format.
After successfully pushing your custom block to Edge Impulse you can edit it from within Studio.
Click on AI labeling under Custom blocks. You should find your custom AI labeling block listed here. To view the configuration settings for your block and edit them, you can click on the three dots and select Edit AI labeling block.
Click on Deployment under Custom blocks. You should find your custom deployment block listed here. To view the configuration settings for your block and edit them, you can click on the three dots and select Edit block.
Organization (Enterprise plan)
Click on Machine learning under Custom blocks. You should find your custom learning block listed here. To view the configuration settings for your block and edit them, you can click on the three dots and select Edit block.
Developer profile (all other plans)
Click on your photo in the top right corner of your developer profile, select Custom ML blocks. To view the configuration settings for your block and edit them, you can click on the three dots and select Edit block.
Click on DSP under Custom blocks. You should find your custom processing block listed here. To view the configuration settings for your block and edit them, you can click on the three dots and select Edit DSP block.
Click on Synthetic data under Custom blocks. You should find your custom synthetic data block listed here. To view the configuration settings for your block and edit them, you can click on the three dots and select Edit synthetic data block.
Click on Transformation under Custom blocks. You should find your custom transformation block listed here. To view the configuration settings for your block and edit them, you can click on the three dots and select Edit transformation block.
Most blocks have the option to set the compute requests and limits (number of CPUs and memory) and some have the option to set the maximum running time duration. These items cannot, however, be configured from the parameters.json
file; they must be configured when editing the block after it has been pushed to Edge Impulse.
CBOR/JSON files in the
Tip: You can use to convert your data
See the tutorial for a deeper explanation.
With your datasets imported, you can now navigate into your dataset, create folders, , add data items and import your data to an Edge Impulse project.
The clinical view is slightly different, see for more information. This view lets you easily but to import data, you will need to set up an or upload them directly to your bucket.
After importing the data into the project, in the Next, post-sync actions step, you can configure a to automatically retrieve and trigger actions in your project:
If you need to get data into your organization, you can now do this in a few simple steps. To go further and use advanced features, query your datasets or transform your dataset, please have a look at the
Any questions, or interested in the enterprise version of Edge Impulse? for more information.
Addition queries to track: These queries are data filters that need to be written in the SQL WHERE format. See for more information. For example metadata->
age >= 18` will return the data samples from adult patients.
This dashboard shows the metrics' progress from the
The training data is already split into train and test (validation) sets. Please refer to the Data section under Inputs in the documentation for additional details.
This will enable internet access and pass in the project API key in the deployment-metadata.json
file (if a project development API key is set) that can be used to authenticate with the Edge Impulse API. Note that if you also require the project ID, this can be retrieved using the API endpoint.
If the supportsEonCompiler
property is set to true
, the inputs for the deployment block will be the version of the files; otherwise the inputs will be the TFLite version of the files.
For the first method, you can use the CLI edge-impulse-blocks runner
tool. See for additional details. The runner does not expect any command line arguments for deployment blocks. However, if your deployment block requires arguments, you can pass them as a single string using the --extra-args <args>
argument.
Edge Impulse has developed several examples of custom deployment blocks. The code for these blocks can be found in public repositories under the . Unfortunately, the repository names don't follow a naming convention. However, they can be found by going to the Edge Impulse account and searching the repositories for deploy
.
The parameters defined in your parameters.json
file will be passed as command line arguments to the script you defined in your Dockerfile as the ENTRYPOINT
for the Docker image. Please refer to the documentation for further details about creating this file, parameter options available, and examples.
Learning blocks operate on data that has already been processed by an and a . This processed data is available to your learning block in a single directory, in the NumPy format, and already split into training (train) and validation (test) datasets. By default the train/validation split is 80/20. You can change this ratio using the . The NumPy datasets can be converted to the required format (e.g. tf.data.Dataset
) for your model and batched as desired within your custom learning block training script.
In addition to the datasets, a sample_id_details.json
file (see ) is located within the data directory. The location of this directory is specified by the --data-directory <dir>
argument and its structure is shown below.
Edge Impulse will automatically convert this file to the required format. Note that arbitrary scikit-learn pipelines cannot be converted. For a list of supported model types, please refer to .
If you are testing your block locally using the edge-impulse-blocks runner
tool as described below, you can adjust the train/validation split using the --validation-set-size <size>
argument but you are unable to split using a metadata key. To profile your model after training locally, see .
After training a custom learning block locally, you can use the to get latency, RAM and ROM estimates. This is very useful as you can immediately see whether your model will fit on device or not. Additionally, you can use this API as part your experiment tracking (e.g. in Weights & Biases or MLFlow) to wield out models that won't fit your latency or memory constraints.
You can also use the to profile your model easily. See for an example on how to profile a model created in Keras.
For the first method, you can use the CLI edge-impulse-blocks runner
tool. See for additional details. The runner expects the following arguments for learning blocks.
Edge Impulse has developed several example custom learning blocks. The code for these blocks can be found in public repositories under the . The repository names typically follow the convention of example-custom-ml-<description>
. As such, they can be found by going to the Edge Impulse account and searching the repositories for example-custom-ml
.
The parameter items defined in your parameters.json
file will be passed as command line arguments to the script you defined in your Dockerfile as the ENTRYPOINT
for the Docker image. Please refer to the documentation for further details about creating this file, parameter options available, and examples.
AI labeling blocks are an extension of transformation blocks operating in standalone
mode, the arguments that are automatically passed to transformation blocks in this mode are also automatically passed to AI labeling blocks. Please refer to the documentation for further details on those parameters.
One or more buckets can be mounted inside of your block. If storage buckets exist in your organization, you will be prompted to mount the bucket(s) when initializing the block with the Edge Impulse CLI. The default mount point will be:
Edge Impulse has developed several AI labeling blocks that are built into the platform. The code for these blocks can be found in public repositories under the . The repository names typically follow the convention of ai-labeling-<description>
. As such, they can be found by going to the Edge Impulse account and searching the repositories for ai-labeling
.
If you want to process data in a portal as part of a you can either:
Mount the bucket that the portal is in, as a transformation block. This will also give you access to all other data in the bucket, very useful if you need to sync other data (see ).
If the data in your portal is already in the right format you can also directly import the uploaded data to your project. In your project view, go to , **** select 'Upload portal' and follow the steps of the wizard:
If you need a secure way for external parties to contribute data to your datasets then upload portals are the way to go. They offer a friendly user interface, upload data directly into your storage buckets, and give you an easy way to use the data directly in Edge Impulse.
Any questions, or interested in the enterprise version of Edge Impulse? for more information.
The parameters defined in your parameters.json
file will be passed to your block in the params
property. If your parameter names contain dashes, these are replaced with underscores before being added to the request body. For example, a processing block parameter named custom-processing-param
is passed as custom_processing_param
. Please refer to the documentation for further details about creating this file, parameter options available, and examples.
The results of generating features can be shown in Studio through graphs and the .
When configuring parameters for a processing block in Studio, a preview of the feature generation results for a single sample is shown. This preview can include displaying graphs. These are the graphs that you define and return in the graphs
property of the response body for the POST /run
method. Graphs should be created in your feature generation script conditionally based on the draw_graphs
property in the request body. See the interface for a graphs object in the section above.
The most convenient way to test your custom processing block before pushing it to Edge Impulse is to host it locally and then expose it to the internet so that it can be accessed by Studio. There are two ways to achieve this. You will need to have Docker and installed on your machine for either approach.
For the first method, you can use the CLI edge-impulse-blocks runner
tool. See for additional details. The runner expects the following arguments for processing blocks.
You will need to implement this function in the main.cpp
file of the C++ library. Example implementations for the processing blocks built into Edge Impulse can be found in the .
Edge Impulse has developed several processing blocks that are built into the platform. The code for these blocks can be found in a public repository under the . See below. Additional examples can also be found in the Edge Impulse account. These repository names typically follow the convention of example-custom-processing-block-<description>
. As such, they can be found by searching the repositories for example-custom-processing
.
If you want to leverage GPU compute for your custom learning blocks, you will need to make sure to install the CUDA packages. You can refer to the repository to see an example Dockerfile that installs these packages.
A parameters.json
file is to be included at the root of your custom block directory. This file describes the block itself and identifies the parameter items that will be exposed for configuration in Studio and, in turn, passed to the script you defined in your Dockerfile as the ENTRYPOINT
. See for more details.
The first steps to developing a custom block are to write your scripts and Dockerfile. Once those are completed, you can initialize the block, test it locally, and push it to Edge Impulse using the tool in the Edge Impulse CLI.
To use the blocks runner tool in the Edge Impulse CLI, run the edge-impulse-blocks runner
command from within your custom block directory and follow the prompts to test your block locally. See .
Once pushed successfully, your block will appear in your organization or, if it is a custom learning block and you are not on the Enterprise plan, in your developer profile. See for images showing each block type after being pushed to Edge Impulse.
--metadata <file>
Always
Provides the file path for deployment-metadata.json
as a string. The deployment-metadata.json
file contains details about the impulse being deployed. See deployment-metadata.json.
download/
Download directory for the archive of required input files for the deployment block.
<project-id>/input/
The input files archive will be automatically extracted to this location.
<project-id>/output/
Where the output from your build script is expected to be written.
--info-file <file>
Always
Provides the file path for train_input.json
as a string. The train_input.json
file contains configuration details for model training options. See train_input.json.
--data-directory <dir>
Always
Provides the directory path for training/validation datasets as a string.
--out-directory <dir>
Always
Provides the directory path to the output directory as a string. This is where block output needs to be written.
--epochs <value>
Conditional
Passed if no custom parameters are provided. Provides the number of epochs for model training as an integer.
--learning-rate <value>
Conditional
Passed if no custom parameters are provided. Provides the learning rate for model training as a float.
--epochs <number>
If not provided, you will be prompted to enter a value.
--learning-rate <learningRate>
If not provided, you will be prompted to enter a value.
--validation-set-size <size>
Defaults to 0.2 but can be overwritten.
--input-shape <shape>
Automatically identified but can be overwritten.
--extra-args <args>
Additional arguments for your script.
EI_API_ENDPOINT
Always
The API base URL: https://studio.edgeimpulse.com/v1
EI_API_KEY
Always
The organization API key with member privileges: ei_2f7f54...
EI_INGESTION_HOST
Always
The host for the ingestion API: edgeimpulse.com
EI_ORGANIZATION_ID
Always
The ID of the organization that the block belongs to: 123456
EI_PROJECT_ID
Always
The ID of the project: 123456
EI_PROJECT_API_KEY
Always
The project API key: ei_2a1b0e...
--data-ids-file <file>
Always
Provides the file path for id.json
as a string. The ids.json
file lists the data sample IDs to operate on as integers. See ids.json.
--propose-actions <job-id>
Conditional
Only passed when the user wants to preview label changes. If passed, label changes should be staged and not directly applied. Provides the job ID as an integer. See preview mode.
GET
/
Requesting general information about the processing block.
GET
/parameters
Requesting the parameters.json
file for the block.
POST
/run
Requesting features be generated for a single data sample.
POST
/batch
Requesting features be generated for multiple data samples.
x-ei-project-id
Conditional
Provided with GET /run
or GET /batch
requests. The ID of the project.
x-ei-sample-id
Conditional
Provided with GET /run
request. The ID of the sample to be processed.
x-ei-sample-ids
Conditional
Provided with GET /batch
request. A list of IDs of data samples to be processed.
Linear
linear
Logarithmic
logarithmic
Image
image
--port <number>
The port to publish for your Docker container.
In this section, we will show how to synchronize research data with a bucket in your organizational dataset. The goal of this step is to gather data from different sources and sort them to obtain a sorted dataset. We will then validate this dataset in the next section.
Only available with Edge Impulse Enterprise Plan
Try our FREE Enterprise Trial today.
The reference design described in the health reference design PPG-DaLiA DOI 10.24432/C53890 is a publicly available dataset for PPG-based heart rate estimation. This multimodal dataset features physiological and motion data, recorded from both a wrist- and a chest-worn device, of 15 subjects while performing a wide range of activities under close to real-life conditions. The included ECG data provides heart rate ground truth. The included PPG- and 3D-accelerometer data can be used for heart rate estimation, while compensating for motion artefacts. Details can be found in the dataset's readme-file.
File Name
Description
S1_activity.csv
Data containing labels of the activities.
S1_quest.csv
Data from the questionnaire, detailing the subjects' attributes.
ACC.csv
Data from 3-axis accelerometer sensor. The accelerometer is configured to measure acceleration in the range [-2g, 2g]. Therefore, the unit in this file is 1/64g. Data from x, y, and z axis are respectively in the first, second, and third column.
BVP.csv
Blood Volume Pulse (BVP) signal data from photoplethysmograph.
EDA.csv
Electrodermal Activity (EDA) data expressed as microsiemens (μS).
tags.csv
Tags for the data, e.g., Stairs, Soccer, Cycling, Driving, Lunch, Walking, Working, Clean Baseline, No Activity.
HR.csv
Heart Rate (HR) data, as measured by the wearable device. Average heart rate extracted from the BVP signal. The first row is the initial time of the session expressed as a Unix timestamp in UTC. The second row is the sample rate expressed in Hz.
IBI.csv
Inter-beat Interval (IBI) data. Time between individual heartbeats extracted from the BVP signal. No sample rate is needed for this file. The first column is the time (relative to the initial time) of the detected inter-beat interval expressed in seconds (s). The second column is the duration in seconds (s) of the detected inter-beat interval (i.e., the distance in seconds from the previous beat).
TEMP.csv
Data from temperature sensor expressed in degrees on the Celsius (°C) scale.
info.txt
Metadata about the participant, e.g., Age, Gender, Height, Weight, BMI.
You can download the complete set of subject 1 files here:
We've mimicked a proper research study, and have split the data up into two locations.
Initial subject files (ACC.csv, BVP.csv, EDA.csv, HR.csv, IBI.csv, TEMP.csv, info.txt, S1_activity.csv, tags.csv) live in the company data lake in S3. The data lake uses an internal structure with non-human readable IDs for each participant (e.g. Subject 1 as S1_E4
for anonymized data):
Other files are uploaded by the research partner to an upload portal. The files are prefixed with the subject ID:
with the directory S2_E4
indicating that this data is from the second subject in the study, or prefixing the files with S2_
(e.g. S2_activity.csv
).
This is a manual step that some countries regulations may require, this example is for reference, but not needed or used in this example.
To create the mapping between the study ID, subjects name, and the internal data lake ID we can use a study master sheet. It contains information about all participants, ID mapping, and metadata. E.g.:
Notes: This master sheet was made using a Google Sheet but can be anything. All data (data lake, portal, output) are hosted in an Edge Impulse S3 bucket but can be stored anywhere (see below).
Data is stored in cloud storage buckets that are hosted in your own infrastructure. To configure a new storage bucket, head to your organization, choose Data > Buckets, click Add new bucket, and fill in your access credentials. For additional details, refer to Cloud data storage. Our solution engineers are also here to help you set up your buckets.
With the storage bucket in place you can create your first dataset. Datasets in Edge Impulse have three layers:
Datasets in Edge Impulse have three layers:
Dataset: A larger set of data items grouped together.
Data item: An item with metadata and Data file attached.
Data file: The actual files.
There are three ways of uploading data into your organization. You can either:
Upload data directly to the storage bucket (recommended method). In this case use Add data... > Add dataset from bucket and the data will be discovered automatically.
Upload data through the Edge Impulse API.
Upload the files through the Upload Portals.
The sorter is the first step of the research pipeline. Its job is to fetch the data from all locations (here: internal data lake, portal, metadata from study master sheet) and create a research dataset in Edge Impulse. It does this by:
Creating a new structure in S3 like this:
Syncing the S3 folder with a research dataset in your Edge Impulse organization (like PPG-DaLiA Activity Study 2024
).
Updating the metadata with the metadata from the master sheet (Age
, BMI
, etc...). Read on how to add and sync S3 data
With the data sorted we then:
Need to verify that the data is correct (see validate your research data)
Combine the data into a single Parquet file. This is essentially the contract we have for our dataset. By settling on a standard format (strong typed, same column names everywhere) this data is now ready to be used for ML, new algorithm development, etc. Because we also add metadata for each file here we're very quickly building up a valuable R&D datastore.
No required format for data files
There is no required format for data files. You can upload data in any format, whether it's CSV, Parquet, or a proprietary data format.
Parquet is a columnar storage format that is optimized for reading and writing large datasets. It is particularly useful for data that is stored in S3 buckets, as it can be read in parallel and is highly compressed. That is why we are converting the data to Parquet in the transform block code.
See Parquet for more information. or an example in our Create a Transform Block Doc
All these steps can be run through different transformation blocks and executed one after the other using data pipelines.
Transformation blocks take raw data from your organizational datasets and convert the data into a different dataset or files that can be loaded in an Edge Impulse project. You can use transformation blocks to only include certain parts of individual data files, calculate long-running features like a running mean or derivatives, or efficiently generate features with different window lengths. Transformation blocks can be written in any language, and run on the Edge Impulse infrastructure.
No required format for data files
There is no required format for data files. You can upload data in any format, whether it's CSV, Parquet, or a proprietary data format.
Parquet is a columnar storage format that is optimized for reading and writing large datasets. It is particularly useful for data that is stored in S3 buckets, as it can be read in parallel and is highly compressed.
The PPG-DaLiA dataset is a multimodal collection featuring physiological and motion data recorded from 15 subjects.
In this tutorial we build a Python-based transformation block that loads Parquet files, we process the dataset by calculating features and transforming it into a unified schema suitable for machine learning. If you haven't done so, go through synchronizing clinical data with a bucket first.
Only available with Edge Impulse Enterprise Plan
Try our FREE Enterprise Trial today.
You'll need:
The Edge Impulse CLI.
If you receive any warnings that's fine. Run edge-impulse-blocks
afterwards to verify that the CLI was installed correctly.
PPG-DaLiA CSV files: Download files like ACC.csv
, HR.csv
, EDA.csv
, etc., which contain sensor data.
Transformation blocks use Docker containers, a virtualization technique that lets developers package up an application with all dependencies in a single package. If you want to test your blocks locally you'll also need (this is not a requirement):
Docker desktop installed on your machine.
To build a transformation block open a command prompt or terminal window, create a new folder, and run:
This will prompt you to log in, and enter the details for your block. E.g.:
Then, create the following files in this directory:
2.1 - Dockerfile
We're building a Python based transformation block. The Dockerfile describes our base image (Python 3.7.5), our dependencies (in requirements.txt
) and which script to run (transform.py
).
Note: Do not use a WORKDIR
under /home
! The /home
path will be mounted in by Edge Impulse, making your files inaccessible.
ENTRYPOINT vs RUN / CMD
If you use a different programming language, make sure to use ENTRYPOINT
to specify the application to execute, rather than RUN
or CMD
.
2.2 - requirements.txt
This file describes the dependencies for the block. We'll be using pandas
and pyarrow
to parse the Parquet file, and numpy
to do some calculations.
2.3 - transform.py
This file includes the actual application. Transformation blocks are invoked with three parameters (as command line arguments):
--in-file
or --in-directory
- A file (if the block operates on a file), or a directory (if the block operates on a data item) from the organizational dataset. In this case the unified_data.parquet
file.
--out-directory
- Directory to write files to.
--hmac-key
- You can use this HMAC key to sign the output files. This is not used in this tutorial.
--metadata
- Key/value pairs containing the metadata for the data item, plus additional metadata about the data item in the dataItemInfo
key. E.g.:
{ "subject": "AAA001", "ei_check": "1", "dataItemInfo": { "id": 101, "dataset": "Human Activity 2022", "bucketName": "edge-impulse-tutorial", "bucketPath": "janjongboom/human_activity/AAA001/", "created": "2022-03-07T09:20:59.772Z", "totalFileCount": 14, "totalFileSize": 6347421 } }
Add the following content. This takes in the Parquet file, groups data by their label, and then calculates the RMS over the X, Y and Z axes of the accelerometer.
toggle to expand code
Docker
You can also build the container locally via Docker, and test the block. The added benefit is that you don't need any dependencies installed on your local computer, and can thus test that you've included everything that's needed for the block. This requires Docker desktop to be installed.
To build the container and test the block, open a command prompt or terminal window and navigate to the source directory. First, build the container:
Then, run the container (make sure unified_data.parquet
is in the same directory):
Seeing the output
This process has generated a new Parquet file in the out/
directory containing the RMS of the X, Y and Z axes. If you inspect the content of the file (e.g. using parquet-tools) you'll see the output:
If you don't have parquet-tools
installed, you can install it via:
Then, run:
This will show you the metadata and the columns in the file:
code output block:
With the block ready we can push it to your organization. Open a command prompt or terminal window, navigate to the folder you created earlier, and run:
This packages up your folder, sends it to Edge Impulse where it'll be built, and finally is added to your organization.
The transformation block is now available in Edge Impulse under Data transformation > Transformation blocks.
If you make any changes to the block, just re-run edge-impulse-blocks push
and the block will be updated.
Next, upload the unified_data.parquet
file, by going to Data > Add data... > Add data item, setting name as 'Gestures', dataset to 'Transform tutorial', and selecting the Parquet file.
This makes the unified_data.parquet
file available from the Data page.
With the Parquet file in Edge Impulse and the transformation block configured you can now create a new job. Go to Data, and select the Parquet file by setting the filter to dataset = 'Transform tutorial'
.
Click the checkbox next to the data item, and select Transform selected (1 file). On the 'Create transformation job' page select 'Import data into Dataset'. Under 'output dataset', select 'Same dataset as source', and under 'Transformation block' select the new transformation block.
Click Start transformation job to start the job. This pulls the data in, starts a transformation job and finally uploads the data back to your dataset. If you have multiple files selected the transformations will also run in parallel.
You can now find the transformed file back in your dataset.
Transformation blocks are a powerful feature which let you set up a data pipeline to turn raw data into actionable machine learning features. It also gives you a reproducible way of transforming many files at once, and is programmable through the Edge Impulse API so you can automatically convert new incoming data. If you're interested in transformation blocks or any of the other enterprise features, let us know!
Updating metadata from a transformation block
You can update the metadata of blocks directly from a transformation block by creating a ei-metadata.json
file in the output directory. The metadata is then applied to the new data item automatically when the transform job finishes. The ei-metadata.json
file has the following structure:
Some notes:
If action
is set to add
the metadata keys are added to the data item. If action
is set to replace
all existing metadata keys are removed.
Environmental variables
Transformation blocks get access to the following environmental variables, which let you authenticate with the Edge Impulse API. This way you don't have to inject these credentials into the block. The variables are:
EI_API_KEY
- an API key with 'member' privileges for the organization.
EI_ORGANIZATION_ID
- the organization ID that the block runs in.
EI_API_ENDPOINT
- the API endpoint (default: https://studio.edgeimpulse.com/v1).
Custom parameters
You can specify custom arguments or parameters to your block by adding a parameters.json file in the root of your block directory. This file describes all arguments for your training pipeline, and is used to render custom UI elements for each parameter. For example, this parameters file:
Renders the following UI when you run the transformation block:
And the options are passed in as command line arguments to your block:
For more information, and all options see parameters.json.
You can upload your existing data samples and datasets to your project directly through the Edge Impulse Studio Uploader.
The uploader signs local files and uploads them to the ingestion service. This is useful to upload existing data samples and entire datasets, or to migrate data between Edge Impulse instances.
Edge Impulse Datasets
Need inspiration? Check out our Edge Impulse datasets collection that contains publicly available datasets collected, generated or curated by Edge Impulse or its partners.
These datasets highlight specific use cases, helping you understand the types of data commonly encountered in projects like object detection, audio classification, and visual anomaly detection.
The uploader currently handles these types of files:
.cbor
- Files in the Edge Impulse Data Acquisition format. The uploader will not resign these files, only upload them.
.json
- Files in the Edge Impulse Data Acquisition format. The uploader will not resign these files, only upload them.
.csv
- Files in the Edge Impulse Comma Separated Values (CSV) format. If you have configured the "CSV wizard", the settings will be used to parse your CSV files.
.wav
- Lossless audio files. It's recommended to use the same frequency for all files in your data set, as signal processing output might be dependent on the frequency.
.jpg
and .png
- Image files. It's recommended to use the same ratio for all files in your data set.
.mp4
and .avi
- Video file. You can then from the studio split this video file into images at a configurable frame per second.
info.labels
- JSON-like file (without the .json
extension). You can use it to add metadata and for custom labeling strategies (single-label vs multi-label, float values labels, etc...). See Edge Impulse exporter format
The uploader currently handles these types of image dataset annotation formats:
Need more?
If none of these above choices are suitable for your project, you can also have a look at the Transformation blocks to parse your data samples to create a dataset supported by Edge Impulse. See custom transformation blocks.
To upload data using the uploader, go to the Data acquisition page and click on the uploader button as shown in the image below:
Bounding boxes?
If you have existing bounding boxes for your images dataset, make sure your project's labeling method is set to Bounding Boxes (object detection), you can change this parameter in your project's dashboard.
Then you need to upload any label files with your images. You can upload object detection datasets in any supported annotation format. Select both your images and the labels file when uploading to apply the labels. The uploader will try to automatically detect the right format.
Select individual files: This option let you select multiple individual files within a single folder. If you want to upload images with bounding boxes, make sure to also select the label files.
Select a folder: This option let you select one folder, including all the subfolders.
Select which category you want to upload your dataset into. Options can be training
, testing
or perform an 80/20 split between your data samples.
If needed, you can always perform a split later from your project's dashboard.
When a labeling method is not provided, the labels are automatically inferred from the filename through the following regex: ^[a-zA-Z0-9\s-_]+
. For example: idle.01 will yield the label idle
.
Thus, if you want to use labels (string values) containing float values (e.g. "0.01", "5.02", etc...), automatic labeling won't work.
To bypass this limitation, you can make an info.labels
JSON file containing your dataset files' info. We also support adding metadata to your samples. See below to understand the Edge Impulse Exporter format.
info.labels
files)The Edge Impulse Exporter acquisition format provides a simple and intuitive way to store files and associated labels. Folders containing data in this format will take the following structure:
The subdirectories contain files in any Edge Impulse-supported format (see above). Each file represents a sample and is associated with its respective labels in the info.labels
file.
The info.labels
file (can be located in each subdirectory or at the folder root) provides detailed information about the labels. The file follows a JSON format, with the following structure:
version
: Indicates the version of the label format.
files
: A list of objects, where each object represents a supported file format and its associated labels.
path
: The path or file name.
category
: Indicates whether the image belongs to the training or testing set.
label
(optional): Provides information about the labeled objects.
type
: Specifies the type of label - unlabeled
, label
, multi-label
label
(optional): The actual label or class name of the sample.
labels
(optional): The labels in the multi-label format:
label
: Label for the given period.
startIndex
: Timestamp in milliseconds.
endIndex
: Timestamp in milliseconds.
metadata
(Optional): Additional metadata associated with the image, such as the site where it was collected, the timestamp or any useful information.
boundingBoxes
(Optional): A list of objects, where each object represents a bounding box for an object within the image.
label
: The label or class name of the object within the bounding box.
x
, y
: The coordinates of the top-left corner of the bounding box.
width
, height
: The width and height of the bounding box.
The Studio Uploader will automatically detect the info.labels
file:
Want to try it yourself? You can export any dataset from Edge Impulse public projects once you cloned it.
Image datasets can be found in a range of different formats. Different formats have different directory structures, and require annotations (or labels) to follow a particular structure. We support uploading data in many different formats in the Edge Impulse Studio.
Image datasets usually consist of a bunch of image files, and one (or many) annotation files, which provide labels for the images. Image datasets may have annotations that consist of:
A single-label: each image has a single label
Bounding boxes: used for object detection; images contain 'objects' to be detected, given as a list of labeled 'bounding boxes'
When you upload an image dataset, we try to automatically detect the format of that data (in some cases, we cannot detect it and you will need to manually select it).
Once the format of your dataset has been selected, click on Upload Data and let the Uploader parse your dataset:
Leave the data unlabeled, you can manually label your data sample in the studio.
The Edge Impulse object detection acquisition format provides a simple and intuitive way to store images and associated bounding box labels. Folders containing data in this format will take the following structure:
The subdirectories contain image files in JPEG or PNG format. Each image file represents a sample and is associated with its respective bounding box labels in the bounding_boxes.labels
file.
The bounding_boxes.labels
file in each subdirectory provides detailed information about the labeled objects and their corresponding bounding boxes. The file follows a JSON format, with the following structure:
version
: Indicates the version of the label format.
files
: A list of objects, where each object represents an image and its associated labels.
path
: The path or file name of the image.
category
: Indicates whether the image belongs to the training or testing set.
(optional) label
: Provides information about the labeled objects.
type
: Specifies the type of label (e.g., a single label).
label
: The actual label or class name of the object.
(Optional) metadata
: Additional metadata associated with the image, such as the site where it was collected, the timestamp or any useful information.
boundingBoxes
: A list of objects, where each object represents a bounding box for an object within the image.
label
: The label or class name of the object within the bounding box.
x
, y
: The coordinates of the top-left corner of the bounding box.
width
, height
: The width and height of the bounding box.
bounding_boxes.labels
example:
Want to try it yourself? Check this cubes on a conveyor belt dataset in Edge Impulse Object Detection format. You can also retrieve this dataset from this Edge Impulse public project. Data exported from an object detection project in the Edge Impulse Studio is exported in this format.
The COCO JSON (Common Objects in Context JSON) format is a widely used standard for representing object detection datasets. It provides a structured way to store information about labeled objects, their bounding boxes, and additional metadata.
A COCO JSON dataset can follow this directory structure:
The _annotations.coco.json
file in each subdirectory provides detailed information about the labeled objects and their corresponding bounding boxes. The file follows a JSON format, with the following structure:
Categories
The "categories" component defines the labels or classes of objects present in the dataset. Each category is represented by a dictionary containing the following fields:
id
: A unique integer identifier for the category.
name
: The name or label of the category.
(Optional) supercategory
: A higher-level category that the current category belongs to, if applicable. This supercategory
is not used or imported by the Uploader.
Images
The "images" component stores information about the images in the dataset. Each image is represented by a dictionary with the following fields:
id
: A unique integer identifier for the image.
width
: The width of the image in pixels.
height
: The height of the image in pixels.
file_name
: The file name or path of the image file.
Annotations
The "annotations" component contains the object annotations for each image. An annotation refers to a labeled object and its corresponding bounding box. Each annotation is represented by a dictionary with the following fields:
id
: A unique integer identifier for the annotation.
image_id
: The identifier of the image to which the annotation belongs.
category_id
: The identifier of the category that the annotation represents.
bbox
: A list representing the bounding box coordinates in the format [x, y, width, height].
(Optional) area
: The area (in pixels) occupied by the annotated object.
(Optional) segmentation
: The segmentation mask of the object, represented as a list of polygons.
(Optional) iscrowd
: A flag indicating whether the annotated object is a crowd or group of objects.
Edge Impulse uploader currently doesn't import the area
, segmentation
, iscrowd
fields.
_annotations.coco.json
example:
Want to try it yourself? Check this cubes on a conveyor belt dataset in the COCO JSON format.
The OpenImage dataset provides object detection annotations in CSV format. The _annotations.csv
file is located in the same directory of the images it references. A class-descriptions.csv
mapping file can be used to give short description or human-readable classes from the MID LabelName
.
An OpenImage CSV dataset usually has this directory structure:
Annotation Format:
Each line in the CSV file represents an object annotation.
The values in each line are separated by commas.
CSV Columns:
The CSV file typically includes several columns, each representing different attributes of the object annotations.
The common columns found in the OpenImage CSV dataset include:
ImageID
: An identifier or filename for the image to which the annotation belongs.
Source
: The source or origin of the annotation, indicating whether it was manually annotated or obtained from other sources.
LabelName
: The class label of the object.
Confidence
: The confidence score or probability associated with the annotation.
XMin, YMin, XMax, YMax
: The coordinates of the bounding box that encloses the object, usually represented as the top-left (XMin, YMin) and bottom-right (XMax, YMax) corners.
IsOccluded, IsTruncated, IsGroupOf, IsDepiction, IsInside
: Binary flags indicating whether the object is occluded, truncated, a group of objects, a depiction, or inside another object.
Currently, Edge Impulse only imports these fields:
Class Labels:
Each object in the dataset is associated with a class label.
The class labels in the OpenImage dataset are represented as LabelName
in the CSV file.
The LabelName
correspond to specific object categories defined in the OpenImage dataset's ontology (MID).
Note that Edge Impulse does not enforce this ontology, if you have an existing dataset using the MID LabelName, simply provide a class-description.csv
mapping file to see your classes in Edge Impulse Studio.
Bounding Box Coordinates:
The bounding box coordinates define the normalized location and size of the object within the image.
The coordinates are represented as the X and Y pixel values for the top-left corner (XMin, YMin) and the bottom-right corner (XMax, YMax) of the bounding box.
class-descriptions.csv
mapping file:
To be ingested in Edge Impulse the mapping file name must end with *class-descriptions.csv
Here is an example of the mapping file: https://github.com/openimages/dataset/blob/main/dict.csv
_annotations.csv
example:
Want to try it yourself? Check this cubes on a conveyor belt dataset in the OpenImage CSV format.
The Pascal VOC (Visual Object Classes) format is another widely used standard for object detection datasets. It provides a structured format for storing images and their associated annotations, including bounding box labels.
A Pascal VOC dataset can follow this directory structure:
The Pascal VOC dataset XML format typically consists of the following components:
Image files: The dataset includes a collection of image files, usually in JPEG or PNG format. Each image represents a sample in the dataset.
Annotation files: The annotations for the images are stored in XML files. Each XML file corresponds to an image and contains the annotations for that image, including bounding box labels and class labels.
Class labels: A predefined set of class labels is defined for the dataset. Each object in the image is assigned a class label, indicating the category or type of the object.
Bounding box annotations: For each object instance in an image, a bounding box is defined. The bounding box represents the rectangular region enclosing the object. It is specified by the coordinates of the top-left corner, width, and height of the box.
Additional metadata: Pascal VOC format allows the inclusion of additional metadata for each image or annotation. This can include information like the source of the image, the author, or any other relevant details. The Edge Impulse uploader currently doesn't import these metadata.
The structure of an annotation file in Pascal VOC format typically follows this pattern:
cubes.23im33f2.xml
:
Want to try it yourself? Check this cubes on a conveyor belt dataset in the Pascal VOC format.
The Plain CSV format is a very simple format: a CSV annotation file is stored in the same directory as the images. We support both "Single Label" and "Object Detection" labeling methods for this format.
An Plain CSV dataset can follow this directory structure:
Annotation Format:
Each line in the CSV file represents an object annotation.
The values in each line are separated by commas.
CSV Columns (Single Label):
The Plain CSV format (single Label) just contains the file_name and the class:
file_name
: The filename of the image.
class_name
: The class label or category of the image.
_annotations_single_label.csv
example:
CSV Columns (Object Detection):
This Plain CSV format is similar to the TensorFlow Object Detection Dataset format. In this format, the CSV file contains the following columns:
file_name
: The filename of the image.
classes
: The class label or category of the object.
xmin
: The x-coordinate of the top-left corner of the bounding box.
ymin
: The y-coordinate of the top-left corner of the bounding box.
xmax
: The x-coordinate of the bottom-right corner of the bounding box.
ymax
: The y-coordinate of the bottom-right corner of the bounding box.
Each row represents an annotated object in an image. In the following example, there are three objects in cubes_training_0.jpg: a blue, a green and a red cube, two objects in cubes_training_1.jpg, etc... The bounding box coordinates are specified as the top-left corner (xmin, ymin) and the bottom-right corner (xmax, ymax).
_annotations_bounding_boxes.csv
example:
Want to try it yourself? Check this cubes on a conveyor belt dataset in the Plain CSV (object detection) format.
The YOLO TXT format is a specific text-based annotation format mostly used in conjunction with the YOLO object detection algorithm. This format represents object annotations for an image in a plain text file.
File Structure:
Each annotation is represented by a separate text file.
The text file has the same base name as the corresponding image file.
The file extension is .txt
.
Example:
Annotation Format:
Each line in the TXT file represents an object annotation.
Each annotation line contains space-separated values representing different attributes.
The attributes in each line are ordered as follows: class_label
, normalized bounding box coordinates (center_x
, center_y
, width
, height
).
Class label:
The class label represents the object category or class.
The class labels are usually represented as integers, starting from 0 or 1.
Each class label corresponds to a specific object class defined in the dataset.
Normalized Bounding Box Coordinates:
The bounding box coordinates represent the location and size of the object in the image.
The coordinates are normalized to the range [0, 1], where (0, 0) represents the top-left corner of the image, and (1, 1) represents the bottom-right corner.
The normalized bounding box coordinates include the center coordinates (center_x, center_y) of the bounding box and its width and height.
The center coordinates (center_x, center_y) are relative to the width and height of the image, where (0, 0) represents the top-left corner, and (1, 1) represents the bottom-right corner.
The width and height are also relative to the image size.
Here's an example of a YOLO TXT annotation file format for a single object:
For instance: cubes-23im33f2.txt
Each line represent a given normalized bounding box for the corresponding cubes-23im33f2.jpg
image.
Mapping the Class Label:
The classes.txt
, classes.names
or data.yaml
(used by Roboflow YOLOv5 PyTorch export format) files contain configuration values used by the model to locate images and map class names to class_id
s.
For example with the cubes on a conveyor belt dataset with the classes.txt
file:
Want to try it yourself? Check this cubes on a conveyor belt dataset in the YOLOv5 format.
Only available on the Enterprise plan
This feature is only available on the Enterprise plan. Review our or sign up for our free today.
Only available on the Enterprise plan
This feature is only available on the Enterprise plan. Review our or sign up for our free today.
Custom transformation blocks are a way to extend the capabilities of Edge Impulse beyond the transformation blocks built into the platform. If none of the existing blocks created by Edge Impulse fit your needs, you can create custom transformation blocks to integrate your own data pre-processing for unique project requirements.
Ready to dive in and start building? Jump to the examples!
Only available on the Enterprise plan
This feature is only available on the Enterprise plan. Review our or sign up for our free today.
The transformation block structure is shown below. Please see the custom blocks overview page for more details.
The sections below define the required and optional inputs and the expected outputs for custom transformation blocks.
Transformation blocks have access to environment variables, command line arguments, and mounted storage buckets.
The following environment variables are accessible inside of transformation blocks. Environment variable values are always stored as strings.
EI_API_ENDPOINT
Always
The API base URL: https://studio.edgeimpulse.com/v1
EI_API_KEY
Always
The organization API key with member privileges: ei_2f7f54...
EI_INGESTION_HOST
Always
The host for the ingestion API: edgeimpulse.com
EI_LAST_SUCCESSFUL_RUN
Always
The last time the block was successfully run, if a part of a data pipeline: 1970-01-01T00:00:00.000Z
EI_ORGANIZATION_ID
Always
The ID of the organization that the block belongs to: 123456
EI_PROJECT_ID
Conditional
Passed if the transformation block is a data source for a project. The ID of the project: 123456
EI_PROJECT_API_KEY
Conditional
Passed if the transformation block is a data source for a project. The project API key: ei_2a1b0e...
You can also define your own environment variables to pass to your custom block using the requiredEnvVariables
property in the parameters.json
file. You will then be prompted for the associated values for these properties when pushing the block to Edge Impulse using the CLI. Alternatively, these values can be added (or changed) by editing the block in Studio after pushing.
The parameter items defined in your parameters.json
file will be passed as command line arguments to the script you defined in your Dockerfile as the ENTRYPOINT
for the Docker image. Please refer to the parameters.json documentation for further details about creating this file, parameter options available, and examples.
In addition to the items defined by you, the following arguments will be automatically passed to your custom transformation block.
--in-file <file>
Conditional
Passed if operation mode is set to file
. Provides the file path as a string. This is the file to be processed by the block.
--in-directory <dir>
Conditional
Passed if operation mode is set to directory
. Provides the directory path as a string. This is the directory to be processed by the block.
--out-directory <dir>
Conditional
Passed if operation mode is set to either file
or directory
. Provides the directory path to the output directory as a string. This is where block output needs to be written.
--hmac-key <key>
Conditional
Passed if operation mode is set to either file
or directory
. Provides a project HMAC key as a string, if it exists, otherwise '0'
.
--metadata <metadata>
Conditional
Passed if operation mode is set to either file
or directory
, the pass in metadata property (indMetadata
) is set to true, and the metadata exists. Provides the metadata associated with data item as a stringified JSON object.
--upload-category <category>
Conditional
Passed if operation mode is set to file
or directory
and the transformation job is configured to import the results into a project. Provides the upload category (split
, training
, or testing
) as a string.
--upload-label <label>
Conditional
Passed if operation mode is set to file
or directory
and the transformation job is configured to import the results into a project. Provides the upload label as a string.
CLI arguments can also be specified using the cliArguments
property in the parameters.json
file. Alternatively, these arguments can be added (or changed) by editing the block in Studio.
Lastly a user can be prompted for extra CLI arguments when configuring a transformation job if the allowExtraCliArguments
property is set to true
.
One or more cloud data storage buckets can be mounted inside of your block. If storage buckets exist in your organization, you will be prompted to mount the bucket(s) when initializing the block with the Edge Impulse CLI. The default mount point will be:
The mount point can be changed by editing your parameters.json
file before pushing the block to Edge Impulse or editing the block in Studio after pushing.
There are no required outputs from transformation blocks. In general, for blocks operating in file
or directory
mode, new data is written to the directory given by the --out-directory <dir>
argument. For blocks operating in standalone
mode, any actions are typically achieved using API calls inside the block itself.
Transformation blocks can operate in one of three modes: file
, directory
, or standalone
.
As the name implies, file transformation blocks operate on files. When configuring a transformation job, the user will select a list of files to transform. These files will be individually passed to and processed by the script defined in your transformation block. File transformation blocks can be run in multiple processing jobs in parallel.
Each file will be passed to your block using the --in-file <file>
argument.
As the name implies, directory transformation blocks operate on directories. When configuring a transformation job, the user will select a list of directories to transform. These directories will be individually passed to and processed by the script defined in your transformation block. Directory transformation blocks can be run in multiple processing jobs in parallel.
Each directory will be passed to your block using the --in-directory <dir>
argument.
Standalone transformation blocks are a flexible way to run generic cloud jobs that can be used for a wide variety of tasks. In standalone mode, no data is passed into your block. If you need to access your data, you will need to mount your storage bucket(s) into your block. Standalone transformation blocks are run as a single processing job; they cannot be run in multiple processing jobs in parallel.
If your custom transformation block is operating in directory
mode and transforming a clinical dataset, you can update the metadata associated with the data item after it is processed.
To do so, your custom transformation block needs to write an ei-metadata.json
file to the directory specified in the --out-directory <dir>
argument. Please refer to the ei-metadata.json documentation for further details about this file.
There are two locations within Studio that transformation blocks can be found: transformation jobs and project data sources.
Transformation blocks operating in file
or directory
mode will always been shown as an option in the block dropdown for transformation jobs. They cannot be used as a project data source.
Transformation blocks operating in standalone
mode can optionally be shown in the block dropdown for transformation jobs and/or in the block dropdown for project data sources.
file
Always
directory
Always
standalone
If showInCreateTransformationJob
property set to true
.
file
Never
directory
Never
standalone
If showInDataSources
property set to true
.
To speed up your development process, you can test your custom transformation block locally. There are two ways to achieve this. You will need to have Docker installed on your machine for either approach.
For the first method, you can use the CLI edge-impulse-blocks runner
tool. See Block runner for additional details.
If your custom transformation block is operating in either file
or directory
mode, you will be prompted for information to look up and download data (a file or a directory) for the block to operate on when using the blocks runner. This can be achieved by providing either a data item name (clinical data) or the path within a dataset for a file or directory (clinical or default data). You can also specify some of this information using the blocks runner command line arguments.
--dataset <dataset>
Transformation blocks in file
or directory
mode. Files and directories will be looked up within this dataset. If not provided, you will be prompted for a dataset name.
--data-item <data-item>
Clinical data only. Transformation blocks in directory
mode. The data item will be looked up, downloaded, and passed to the container when it is run. If not provided, you will be prompted for the information required to look up a data item.
--file <filename>
Clinical data only. Transformation blocks in file
mode. Must be used in conjunction with --data-item <data-item>
. The file will be looked up, downloaded, and passed to the container when it is run. If not provided, you will be prompted for the information required to look up a file within a data item.
--skip-download
Skips downloading the data.
--extra-args <args>
Additional arguments for your script.
Additional arguments to your script can be provided as a single string using --extra-args <args>
argument.
Using the above approach will create an ei-block-data
directory within your custom block directory. It will contain subdirectories for the data that has been downloaded.
For the second method, you can build the Docker image and run the container directly. You will need to pass any environment variables or command line arguments required by your script to the container when you run it.
If your transformation block operates in either file
or directory
mode, you will also need to create a data/
directory within your custom block directory and place your data used for testing here.
file
mode:
directory
mode:
standalone
mode:
Edge Impulse has developed several transformation blocks, some of which are built into the platform. The code for these blocks can be found in public repositories under the Edge Impulse GitHub account. The repository names typically follow the convention of example-transform-<description>
. As such, they can be found by going to the Edge Impulse account and searching the repositories for example-transform
.
Note that when using the above search term you will come across synthetic data blocks as well. Please read the repository description to identify if it is for a transformation block or a synthetic data block.
Further, several example transformation blocks have been gathered into a single repository:
In this section, you will find a health reference design that describes an end-to-end machine learning workflow for building a wearable health product using Edge Impulse.
We will utilize a publicly available dataset of PPG and accelerometer data that was collected from 15 subjects performing various activities, and emulate a clinical study to train a machine learning model that can classify activities.
The dataset selected to use in this example is the PPG-DaLiA dataset, which includes 15 subjects performing 9 activities, resulting in a total of 15 recordings. See the CSV file summary here, and read more about it in the publishers website here,
This dataset covers an activity study where data is recorded from a wearable end device (PPG + accelerometer), along with labels such as Stairs, Soccer, Cycling, Driving, Lunch, Walking, Working, Clean Baseline, and No Activity. The data is collected and validated, then written to a clinical dataset in an Edge Impulse organization, and finally imported into an Edge Impulse project where we train a classifier.
The health reference design builds transformation blocks that sync clinical data, validate the dataset, query the dataset, and transform the data to process raw data files into a unified dataset.
The design culminates in a data pipeline that handles data coming from multiple sources, data alignment, and a multi-stage pipeline before the data is imported into an Edge Impulse project.
We won't cover in detail all the code snippets, it should be straightforward to follow through, if issues are encountered our solution engineers can help you set this end-to-end ML workflow.
This health reference design section helps you understand how to create a full clinical data pipeline by:
Synchronizing clinical data with a bucket: Collect and organize data from multiple sources into a sorted dataset.
Validating clinical data: Ensure the integrity and consistency of the dataset by applying checklists.
Querying clinical data: Explore and slice data using a query system.
Transforming clinical data: Process and transform raw data into a format suitable for machine learning.
After you have completed the health reference design, you can go further by combining the individual transformation steps into a data pipeline.
Refer to the following guide to learn how to build data pipelines:
Building data pipelines: Build pipelines to automate data processing steps.
The Health Reference Design pipeline consists of the following steps:
DataProcessor: Processes raw data files for each subject.
MetadataGenerator: Extracts and attaches metadata to each subject's data.
DataCombiner: Merges all processed data into a unified dataset.
Repository containing the blocks used in this health reference design:
https://github.com/edgeimpulse/health-reference-design-public-data
The data pipeline workflow for the Health Reference Design is as follows:
Now that all transformation blocks are pushed to Edge Impulse, you can create a pipeline to chain them together.
Steps:
Access Pipelines:
In Edge Impulse Studio, navigate to your organization.
Go to Data > Pipelines.
Add a New Pipeline:
Click on + Add a new pipeline.
Name: PPG-DaLiA Data Processing Pipeline
Description: Processes PPG-DaLiA data from raw files to a combined dataset.
Configure Pipeline Steps:
Paste the following JSON configuration into the pipeline steps:
Replace the transformationBlockId values with the actual IDs of your transformation blocks.
Save the Pipeline.
Run the Pipeline:
Click on the ⋮ (ellipsis) next to your pipeline.
Select Run pipeline now.
Monitor Execution:
Check the pipeline logs to ensure each step runs successfully.
Address any errors that may occur.
Verify Output:
After completion, verify that the datasets (processed-dataset and combined-dataset) have been created and populated.
After the pipeline has successfully run, you can import the combined dataset into an Edge Impulse project to train a machine learning model.
If you didn't complete the pipeline, don't worry, this is just a demonstration. However, you can still import the processed dataset from our HRV Analysis tutorial to train a model.
Refer to the following guides to learn how to import datasets into Edge Impulse:
HRV Analysis: Analyze Heart Rate Variability (HRV) data.
Activity Recognition: Classify activities using accelerometer data.
MLOps: Implement MLOps practices in your workflow.
The Health Reference Design provides a comprehensive overview of building a wearable health product using Edge Impulse. By following the steps outlined in this guide, you will gain a practical understanding of a real-world machine learning workflow for processing and analyzing clinical data.
If you have any questions or need assistance with implementing the Health Reference Design, feel free to reach out to our sales team for product development, or share your work on our forum.
Organizational datasets contain a powerful query system which lets you explore and slice data. You control the query system through the 'Filter' text box, and you use a language which is very similar to SQL (documentation).
Only available with Edge Impulse Enterprise Plan
Try our FREE Enterprise Trial today.
For example, here are some queries that you can make:
dataset like '%PPG%'
- returns all items and files from the study.
bucket_name = 'edge-impulse-health-reference-design' AND -- labels sitting,walking
- returns data whose label is 'sitting' and 'walking, and that is stored in the 'edge-impulse-health-reference-design' bucket.
metadata->>'ei_check' = 0
- return data that have a metadata field 'ei_check' which is '0'.
created > DATE('2022-08-01')
- returns all data that was created after Aug 1, 2022.
After you've created a filter, you can select one or more data items, and select Actions...>Download selected to create a ZIP file with the data files. The file count reflects the number of files returned by the filter.
The previous queries all returned all files for a data item. But you can also query files through the same filter. In that case the data item will be returned, but only with the files selected. For example:
file_name LIKE '%.png'
- returns all files that end with .png
.
If you have an interesting query that you'd like to share with your colleagues, you can just share the URL. The query is already added to it automatically.
These are all the available fields in the query interface:
dataset
- Dataset.
bucket_id
- Bucket ID.
bucket_name
- Bucket name.
bucket_path
- Path of the data item within the bucket.
id
- Data item ID.
name
- Data item name.
total_file_count
- Number of files for the data item.
total_file_size
- Total size of all files for the data item.
created
- When the data item was created.
metadata->key
- Any item listed under 'metadata'.
file_name
- Name of a file.
file_names
- All filenames in the data item, that you can use in conjunction with CONTAINS
. E.g. find all items with file X, but not file Y: file_names CONTAINS 'x' AND not file_names CONTAINS 'y'
.
Custom synthetic data blocks are a way to extend the synthetic data feature within Edge Impulse. If none of the blocks created by Edge Impulse that are built into the platform fit your needs, you can modify them or develop from scratch to create a custom synthetic data block. This allows you to integrate your own data generation techniques for unique project requirements.
Ready to dive in and start building? Jump to the examples!
Only available on the Enterprise plan
This feature is only available on the Enterprise plan. Review our or sign up for our free today.
Synthetic data blocks are an extension of transformation blocks operating in standalone
mode and, as such, follow the same structure without being able to pass a directory or file directly to your scripts. Please see the custom blocks overview page for more details.
The sections below define the required and optional inputs and the expected outputs for custom synthetic data blocks.
Synthetic data blocks have access to environment variables, command line arguments, and mounted storage buckets.
The following environment variables are accessible inside of synthetic data blocks. Environment variable values are always stored as strings.
EI_API_ENDPOINT
Always
The API base URL: https://studio.edgeimpulse.com/v1
EI_API_KEY
Always
The organization API key with member privileges: ei_2f7f54...
EI_INGESTION_HOST
Always
The host for the ingestion API: edgeimpulse.com
EI_ORGANIZATION_ID
Always
The ID of the organization that the block belongs to: 123456
EI_PROJECT_ID
Always
The ID of the project: 123456
EI_PROJECT_API_KEY
Always
The project API key: ei_2a1b0e...
You can also define your own environment variables to pass to your custom block using the requiredEnvVariables
property in the parameters.json
file. You will then be prompted for the associated values for these properties when pushing the block to Edge Impulse using the CLI. Alternatively, these values can be added (or changed) by editing the block in Studio.
The parameter items defined in your parameters.json
file will be passed as command line arguments to the script you defined in your Dockerfile as the ENTRYPOINT
for the Docker image. Please refer to the parameters.json documentation for further details about creating this file, parameter options available, and examples.
In addition to the items defined by you, specific arguments will be automatically passed to your synthetic data block.
Synthetic data blocks are an extension of transformation blocks operating in standalone
mode, the arguments that are automatically passed to transformation blocks in this mode are also automatically passed to synthetic data blocks. Please refer to the custom transformation blocks documentation for further details on those parameters.
Along with the transformation block arguments, the following synthetic data specific arguments are passed as well.
--synthetic-data-job-id <job-id>
Always
Provides the job ID as an integer. The job ID must be passed as the x-synthetic-data-job-id
header value when uploading data to Edge Impulse through the ingestion API.
Additional CLI arguments can also be specified using the CLI arguments field when editing the block in Studio.
One or more cloud data storage buckets can be mounted inside of your block. If storage buckets exist in your organization, you will be prompted to mount the bucket(s) when initializing the block with the Edge Impulse CLI. The default mount point will be:
The mount point can be changed by editing the block in Studio after pushing.
There are no required outputs from synthetic data blocks. In general, the data that is generated is uploaded to Edge Impulse using the data ingestion API.
When uploading synthetic data to Edge Impulse using the ingestion API, you will need to include the the x-synthetic-data-job-id
header in your request. The value for this header is the job ID provided to your block through the --synthetic-data-job-id <job-id>
argument.
Synthetic data blocks are not supported by the edge-impulse-blocks runner
CLI tool
Synthetic data blocks are not currently supported by the blocks runner in the Edge Impulse CLI. To test you custom synthetic data block, you will need to build the Docker image and run the container directly. You will need to pass any environment variables or command line arguments required by your script to the container when you run it.
Edge Impulse has developed several synthetic data blocks, some of which are built into the platform. The code for these blocks can be found in public repositories under the Edge Impulse GitHub account. The repository names typically follow the convention of example-transform-<description>
. As such, they can be found by going to the Edge Impulse account and searching the repositories for example-transform
.
Note that when using the above search term you will come across transformation blocks as well. Please read the repository description to identify if it is for a synthetic data block or a transformation block.
Below are direct links to a some examples:
Only available with Edge Impulse Enterprise Plan
Try our FREE Enterprise Trial today.
You can optionally show a check mark in the list of data items, and show a check list for data items. This can be used to quickly view which data items are complete (if you need to capture data from multiple sources) or whether items are in the right format.
Checklists look trivial, but are actually very powerful as they give quick insights in dataset issues. Missing these issues until after the study is done can be very expensive.
Checklists are written to ei-metadata.json
and are automatically being picked up by the UI.
Checklists are driven by the metadata for a data item. Set the ei_check
metadata item to 0
or 1
to show a check mark in the list. To show an item in the checklist, set an ei_check_KEYNAME
metadata item to 0
or 1
.
To query for items with or without a check mark, use a filter in the form of:
To make it easy to create these lists on the fly you can set these metadata items directly from a transformation block
For the reference design described and used in the previous pages, the combiner takes in a data item, and writes out:
A checklist, e.g.:
✔ - PPG file present
✔ - Accelerometer file present
✘ - Correlation between HR/PPG HR is at least 0.5
If the checklist is OK, a combined.parquet
file.
A hr.png
file with the correlation between HR found from PPG, and HR from the reference device. This is useful for two reasons:
If the correlation is too low we're looking at the wrong file, or data is missing.
Verify if the PPG => HR algorithm actually works.
This makes it easy to quickly see if the data is in the right format, and if the data is complete. If the checklist is not OK, the data item is not used in the training set.
After creating your Edge Impulse Studio project, you will be directed to the project's dashboard. The dashboard gives a quick overview of your project such as your project ID, the number of devices connected, the amount of data collected, the preferred labeling method, among other editable properties. You can also enable some additional capabilities to your project such as collaboration, making your project public, and showcasing your public projects using Markdown READMEs as we will see.
The figure below shows the various sections and widgets of the dashboard that we will cover here.
The Getting Started section is here to help. You can choose from 3 different options to get started:
Add existing data: When selecting this option, you can then choose to Upload data from your computer or to Add storage bucket
Collect new data: When selecting this option, the getting started guide will ask you to either Scan QR code to connect to your phone, Connect to your computer, or to Connect your device or developer board. Make sure that your device or development board is flashed with the Edge Impulse official firmware.
Upload your model: This option will change the default workflow. See BYOM to learn more about how to import your existing model to Edge Impulse studio.
You have two options for your project's visibility:
Private:
Community Plan: 2 private projects
Professional Plan: 10 private projects
Enterprise Plan: Unlimited private projects
Public:
When a project is public, all of your data, block configurations, intermediate results, and final models will be shared with the world.
Your project will be publicly accessible and can be cloned with a single click with the provided URL:
List your project in the public project repository
If you want to reference your project in the Edge Impulse public project repository, you can go to the versioning page, create a version and check the Publish this version under the Apache 2.0 license.
This is a great way to share your work with the community.
When you have a trained model available in your project, this card will appear to let you test your model with your phone or your computer. This is particularly useful to validate the behaviour of your model before integrating it into your targeted embedded firmware.
You can invite up to three collaborators to join and contribute to your public project.
Private projects in the Community Plan and Professional Plan cannot invite collaborators.
Private projects belonging to an organization can have unlimited collaborators (based on the number of licensed user seats). See the Enterprise Plan.
To add a collaborator, go to your project's dashboard and find the "Collaborators" widget. Click the '+' icon and type the username or e-mail address of the other user. The user will be invited to create an Edge Impulse account if it doesn't exist.
The user will be automatically added to the project and will get an email notification inviting them to start contributing to your project. To remove a user, simply click on the three dots beside the user then tap ‘Delete’ and they will be automatically removed.
The project README enables you to explain the details of your project in a short way. Using this feature, you can add visualizations such as images, GIFs, code snippets, and text to your project in order to bring your colleagues and project viewers up to speed with the important details of your project. In your README you might want to add things like:
What the project does
Why the project is useful
Motivations of the project
How to get started with the project
What sensors and target deployment devices you used
How you plan to improve your project
Where users can get help with your project
To create your first README, navigate to the "about this project" widget and click "add README"
For more README inspiration, check out the public Edge Impulse project tutorials below:
The project info widget shows the project's specifications such as the project ID, labeling method, and latency calculations for your target device.
The project ID is a unique numerical value that identifies your project. Whenever you have any issue with your project on the studio, you can always share your project ID on the forum for assistance from edge Impulse staff.
On the labeling method dropdown, you need to specify the type of labeling your dataset and model expect. This can be either one label per data item or bounding boxes. Bounding boxes only work for object detection tasks in the studio. Note that if you interchange the labeling methods, learning blocks will appear to be hidden when building your impulse.
One of the amazing Edge Impulse superpowers is the latency calculation component. This is an approximate time in milliseconds that the trained model and DSP operations are going to take during inference based on the selected target device. This hardware in the loop approach ensures that the target deployment device compute resources are not underutilized or over-utilized. It also saves developers' time associated with numerous inference iterations back and forth the studio in search of optimum models.
In the Block Output section, you can download the results of the DSP and ML operations of your impulse.
The downloadable assets include the extracted features, Tensorflow SavedModel, and both quantized and unquantized LiteRT (previously Tensorflow Lite) models. This is particularly helpful when you want to perform other operations to the output blocks outside the Edge Impulse studio. For example, if you need a TensorflowJS model, you will just need to download the TensorFlow saved model from the dashboard and convert it to TensorFlowJS model format to be served on a browser.
Only available with Edge Impulse Professional and Enterprise Plans
Try our Professional Plan or FREE Enterprise Trial today.
This section consists of editable parameters that directly affect the performance of the studio when building your impulse. Depending on the selected or available settings, your jobs can either be fast or slow.
The use of GPU for training and Parallel DSP jobs is currently an internal experimental feature that will be soon released.
To bring even more flexibility to projects, the administrative zone gives developers the power to enable other additional features that are not found in edge impulse projects by default. Most of these features are usually advanced features intended for organizations or sometimes experimental features.
To activate these features you just need to check the boxes against the specific features you want to use and click save experiments.
The danger zone widget consists of irrevocable actions that let you to:
Perform train/test split. This action re-balances your dataset by splitting all your data automatically between the training and testing set and resets the categories for all data.
Launch the getting started wizard. This will remove all data, and clear out your impulse.
Transfer ownership. This action is available for users who have one or more organizations linked with their accounts. With it, you can start working on a project with your user profile and then transfer the ownership to your organization.
Delete your project. This action removes all devices, data, and impulses from your project.
Delete all data in this project.
The data explorer is a visual tool to explore your dataset, find outliers or mislabeled data, and to help label unlabeled data. The data explorer first tries to extract meaningful features from your data (through signal processing and neural network embeddings) and then uses a dimensionality reduction algorithm to map these features to a 2D space. This gives you a one-look overview of your complete dataset.
The Data explorer tab is available for audio classification, image classification and regression projects only.
To access the data explorer head to Data acquisition, click Data explorer, then select a way to generate the data explorer. Depending on you data you'll see three options:
Using a pre-trained model - here we use a large neural network trained on a varied dataset to generate the embeddings. This works very well if you don't have any labeled data yet, or want to look at new clusters of data. This option is available for keywords and for images.
Using your trained impulse - here we use the neural network block in your impulse to generate the embeddings. This typically creates even better visualizations, but will fail if you have completely new clusters of data as the neural network hasn't learned anything about them. This option is only available if you have a trained impulse.
Using the preprocessing blocks in your impulse - here we skip the embeddings, and just use your selected signal processing blocks to create the data explorer. This creates a similar visualization as the feature explorer but in a 2D space and with extra labeling tools. This is very useful if you don't have any labeled data yet, or if you have new clusters of data that your neural network hasn't learned yet.
Then click Generate data explorer to create the data explorer. If you want to make a different choice after creating the data explorer click ⋮ in the top right corner and select Clear data explorer.
Want to see examples of the same dataset visualized in different ways? Scroll down!
To view an item in your dataset just click on any of the dots (some basic information appears on hover). Information about the sample, and a preview of the data item appears at the bottom of the data explorer. You can click Set label (or l on your keyboard) to set a new label for the data item, or press Delete item (or d on your keyboard) to remove the data item. These changes are queued until you click Save labels (at the top of the data explorer).
The data explorer marks unlabeled data in gray (with an 'Unlabeled' label). To label this data you click on any gray dot. To then set a label by clicking the Set label button (or by pressing l
on your keyboard) and enter a label. Other unlabeled data in the vicinity of this item will automatically be labeled as well. This way you can quickly label clustered data.
To upload unlabeled data you can either:
Use the upload UI and select the 'Leave data unlabeled' option.
Select the items in your dataset under Data acquisition, select all relevant items, click Edit labels and set the label to an empty string.
When uploading data through the ingestion API, set the x-no-label
header to 1, and the x-label
to an empty string.
Or, if you want to start from scratch, click the three dots on top of the data explorer, and select Clear all labels
.
The data explorer uses a three-stage process:
It runs your data through an input and a DSP block - like any impulse.
It passes the result of 1) through part of a neural network. This forces the neural network to compress the DSP output even further, but to features that are highly specialized to distinguish the exact type of data in your dataset (called 'embeddings').
The embeddings are passed through t-SNE, a dimensionality reduction algorithm.
So what are these embeddings actually? Let's imagine you have the model from the Continuous motion recognition tutorial. Here we slice data up in 2-second windows and run a signal processing step to extract features. Then we use a neural network to classify between motions. This network consists of:
33 input features (from the signal processing step)
A layer with 20 neurons
A layer with 10 neurons
A layer with 4 neurons (the number of different classes)
While training the neural network we try to find the mathematical formula that best maps the input to the output. We do this by tweaking each neuron (each neuron is a parameter in our formula). The interesting part is that each layer of the neural network will start acting like a feature extracting step - just like our signal processing step - but highly tuned for your specific data. For example, in the first layer, it'll learn what features are correlated, in the second it derives new features, and in the final layer, it learns how to distinguish between classes of motions.
In the data explorer we now cut off the final layer of the neural network, and thus we get the derived features back - these are called "embeddings". Contrary to features we extract using signal processing we don't really know what these features are - they're specific to your data. In essence, they provide a peek into the brain of the neural network. Thus, if you see data in the data explorer that you can't easily separate, the neural network probably can't either - and that's a great way to spot outliers - or if there's unlabeled data close to a labeled cluster they're probably very similar - great for labeling unknown data!
Here's an example of using the data explorer to visualize a very complex computer vision dataset (distinguishing between the four cats of one of our infrastructure engineers).
For less complex datasets, or lower-dimensional data you'll typically see more separation, even without custom models.
If you have any questions about the data explorer or embeddings, we'd be happy to help on the forums or reach out to your solutions engineer. Excited? Talk to us to get access to the data explorer, and finally be able to label all that sensor data you've collected!
All collected data for each project can be viewed on the Data acquisition tab. You can see how your data has been split for train/test set as well as the data distribution for each class in your dataset. You can also send new sensor data to your project either by file upload, WebUSB, Edge Impulse API, or Edge Impulse CLI.
Organization data
Since the creation of Edge Impulse, we have been helping our customers deal with complex data pipelines, complex data transformation methods and complex clinical validation studies.
The organizational data gives you tools to centralize, validate and transform datasets so they can be easily imported into your projects.
See the Organization data documentation.
The panel on the right allows you to collect data directly from any fully supported platform:
Through WebUSB.
Using the Edge Impulse CLI daemon.
From the Edge Impulse for Linux CLI.
The WebUSB and the Edge Impulse daemon work with any fully supported device by flashing the pre-built Edge Impulse firmware to your board. See the list of fully supported boards.
When using the Edge Impulse for Linux CLI, run edge-impulse-linux --clean
and it will add your platform to the device list of your project. You will then will be able to interact with it from the Collect data panel.
Need more?
If your device is not in the officially supported list, you can also collect data using the CLI data forwarder by directly writing the sensor values over a serial connection. The "data forwarder" then signs the data and sends it to the ingestion service.
Edge Impulse also supports different data sample formats and dataset annotation formats (Pascal VOC, YOLO TXT, COCO JSON, Edge Impulse Object Detection, OpenImage CSV) that you can import into your project to build your edge AI models:
Upload portals (Enterprise feature)
Edge Impulse Datasets
Need inspiration? Check out our Edge Impulse datasets collection that contains publicly available datasets collected, generated or curated by Edge Impulse or its partners.
These datasets highlight specific use cases, helping you understand the types of data commonly encountered in projects like object detection, audio classification, and visual anomaly detection.
For time-series data samples (including audio), you can visualize the time-series graphs on the right panel with a dark-blue background:
If you are dealing with multi-label data samples. Here is the corresponding preview:
Preview the values of tabular non-time-series & pre-processed data samples:
Raw images can be directly visualized from the preview:
For object detection projects, we can overlay the corresponding bounding boxes:
Raw videos (.mp4) can be directly visualized from the preview. Please note that you will need to split the videos into frames as we do not support training on videos files:
List view
Grid view
The train/test split is a technique for training and evaluating the performance of machine learning algorithms. It indicates how your data is split between training and testing samples. For example, an 80/20 split indicates that 80% of the dataset is used for model training purposes while 20% is used for model testing.
This section also shows how your data samples in each class are distributed to prevent imbalanced datasets which might introduce bias during model training.
Manually navigating to some categories of data can be time-consuming, especially when dealing with a large dataset. The data acquisition filter enables the user to filter data samples based on some criteria of choice. This can be based on:
Label - class to which a sample represents.
Sample name - unique ID representing a sample.
Signature validity
Enabled and disabled samples
Length of sample - duration of a sample.
The filtered samples can then be manipulated by editing labels, deleting, and moving from the training set to the testing set (and vice versa), a shown in the image above.
The data manipulations above can also be applied at the data sample level by simply navigating to the individual data sample by clicking on "⋮" and selecting the type of action you might want to perform on the specific sample. This might be renaming, editing its label, disabling, cropping, splitting, downloading, and even deleting the sample when desired.
Single label
Multi-label
See Data Acquisition -> Multi-label -> Edit multi-label samples for more information.
To crop a data sample, go to the sample you want to crop and click ⋮, then select Crop sample. You can specify a length, or drag the handles to resize the window, then move the window around to make your selection.
Made a wrong crop? No problem, just click Crop sample again and you can move your selection around. To undo the crop, just set the sample length to a high number, and the whole sample will be selected again.
Besides cropping you can also split data automatically. Here you can perform one motion repeatedly, or say a keyword over and over again, and the events are detected and can be stored as individual samples. This makes it easy to very quickly build a high-quality dataset of discrete events. To do so head to Data Acquisition, record some new data, click, and select Split sample. You can set the window length, and all events are automatically detected. If you're splitting audio data you can also listen to events by clicking on the window, the audio player is automatically populated with that specific split.
Samples are automatically centered in the window, which might lead to problems on some models (the neural network could learn a shortcut where data in the middle of the window is always associated with a certain label), so you can select "Shift samples" to automatically move the data a little bit around.
Splitting data is - like cropping data - non-destructive. If you're not happy with a split just click Crop sample and you can move the selection around easily.
The labeling queue and the auto-labeler will only appear on your data acquisition page if you are dealing with object detection tasks.
If you are not dealing with an object detection task, you can simply change the Labeling method configuration by going to Dashboard > Project info > Labeling method and clicking the dropdown and selecting "one label per data item" as shown in the image below.
Also, see our Label image data using GPT-4o tutorial to see how to leverage the power of LLMs to automatically label your data samples based on simple prompts.
There is a wide variety of devices that you can connect to your Edge Impulse project. These devices can help you collect datasets for your project, test your trained ML model and even deploy your ML model directly to your development board with a pre-built binary application (for fully supported development platforms).
On the Devices tab, you'll find a list of all your connected devices and a guide on how to connect new devices that are currently supported by Edge Impulse.
To connect a new device, click on the Connect a new device button on the top right of your screen.
You will get a pop-up with multiple options of devices you can connect to your Edge Impulse project. Available options include:
The target configuration tool allows you to define your Target device and Application budget according to your project's requirements. This flow is designed to help you optimize your impulse, processing, learn block, or imported model for your specific target hardware, ensuring that your impulse will run efficiently on your device or custom architecture.
The configuration form can be accessed from the top-level navigation. The form allows you to select from a range of processor types, architectures, and clock rates. For a custom device, you could for example select Low-end MCU and specify the clock rate, RAM, ROM, and maximum allowed latency for your application.
By default, the form shows 'Cortex-M4F 80MHz' as the target device. You can change this by clicking on Change Target device. You can select from a range of processor types, architectures, and clock rates. For a custom device, you could for example select Low-end MCU and specify the clock rate, RAM, ROM, and maximum allowed latency for your application.
Lets walk you through some of the current options for configuring your device and application budget:
Target Device: Select the type of target device you are configuring from options like "Custom" or specific development boards.
Processor Type Selection: Selecting a processor type dynamically adjusts available architecture options and fields to suit your hardware:
For Low-end MCU: This option allows you to specify clock rate, RAM, and ROM, suitable for 'Cortex-M' architectures.
For AI Accelerators: Selecting this disables the clock rate field, reflecting the unique requirements of AI accelerator devices.
Custom Device Configuration: Choosing to configure a custom device opens fields to precisely define its capabilities, ensuring your project setup is accurately tailored to your hardware.
The form allows you to select from a range of processor types, architectures, and clock rates. For a custom device, you could for example select Low-end MCU and specify the clock rate, RAM, ROM, and maximum allowed latency for your application.
Custom: Select this for custom hardware specifications or devices not listed in Edge Impulse, allowing for a customized hardware profile. Selection Options
Processor Type & Architecture
Choose from a variety of processor types and architectures. Your selection determines which options and fields are available to accurately configure your device. Estimations for GPU, AI accelerator, or NPU devices are not computed using clock speed, or but rather the device's unique capabilities.
Processor Type: Selections range from various processor types. Choosing GPU, AI accelerator' or NPU deactivates the clock speed option, as it's irrelevant for device estimation.
Processor Architecture (Optional): Specify your device's architecture to refine its configuration (e.g., Cortex-M0+, Cortex-M4F, Cortex-M7).
Clock Rate (Optional): Set the clock rate for relevant processor types to estimate operational capabilities accurately. The units shown will be indicated by the | MHz | GHz as relevant to the scale of processor. As previously stated the clock rate field is disabled for GPU, AI accelerator, or NPU devices.
Accelerator: If the device supports hardware acceleration, select from available options such as Arm Cortex-U55, NVIDIA Jetson Nano, and others.
Device ID (Optional): Provide a unique identifier for your custom device model or chip architecture variant for easy recognition and setup.
Custom Device Name (Optional): Provide a unique name for your custom device to easily identify it in your project.
Application Budget - RAM, ROM, and Latency
The application budget section allows you to specify the maximum allowed latency, RAM, and ROM for your application. These values are used to estimate the performance of your model on your target device.
RAM: Specify the amount of RAM available on your device in kilobytes (kB).
ROM: Specify the amount of ROM available on your device in kilobytes (kB).
Latency: Specify the maximum allowed latency for your application in milliseconds (ms).
Save Target: Save your custom device and application budget configuration to apply it to your project.
After customizing your target device and application budget, click Save target. With the target device set, navigate to the EON Tuner to see the configuration in action. The target device can be seen at the top level of navigation on all screens within your project. Your custom device name (e.g., 'my first mcu') and the specified parameters (100 ms latency, 256 kB RAM, 1024 kB ROM) are visible. The target device configuration is also taken into account during the performance estimation for deployment.
Once saved the target device can be seen at the top level of navigation on all screens within your project. Your custom device name (e.g., 'my first mcu') and the specified parameters (100 ms latency, 256 kB RAM, 1024 kB ROM) are visible. The target device configuration is also taken into account during the performance estimation for deployment.
The target-driven flow in Edge Impulse Studio allows you to configure your target device and application budget according to your project's requirements. This flow is designed to help you optimize your impulse for your specific target hardware, ensuring that your impulse will run efficiently on your device.
We hope this feature is helpful, and intuitive. If you have any questions or suggestions, feel free to reach out to us at forum.edgeimpulse.com. We're always happy to hear from you!
Only available on the Enterprise plan
This feature is only available on the Enterprise plan. Review our or sign up for our free today.
When you have initalized and finished testing your block locally, you will want to push it to Edge Impulse. The procedure to push your block to Edge Impulse is described in the overview page. Please refer to that documentation for details.
When you have initalized and finished testing your block locally, you will want to push it to Edge Impulse. The procedure to push your block to Edge Impulse is described in the overview page. Please refer to that documentation for details.
When you have initalized and finished testing your block locally, you will want to push it to Edge Impulse. The procedure to push your block to Edge Impulse is described in the overview page. Please refer to that documentation for details.
When you have initalized and finished testing your block locally, you will want to push it to Edge Impulse. The procedure to push your block to Edge Impulse is described in the overview page. Please refer to that documentation for details.
When you are finished developing your block locally, you will want to initialize it. The procedure to initialize your block is described in the overview page. Please refer to that documentation for details.
When you are finished developing your block locally, you will want to initialize it. The procedure to initialize your block is described in the overview page. Please refer to that documentation for details.
When you are finished developing your block locally, you will want to initialize it. The procedure to initialize your block is described in the overview page. Please refer to that documentation for details.
When you are finished developing your block locally, you will want to initialize it. The procedure to initialize your block is described in the overview page. Please refer to that documentation for details.
No common issues have been identified thus far. If you encounter an issue, please reach out on the or, if you are on the Enterprise plan, through your support channels.
No common issues have been identified thus far. If you encounter an issue, please reach out on the or, if you are on the Enterprise plan, through your support channels.
No common issues have been identified thus far. If you encounter an issue, please reach out on the or, if you are on the Enterprise plan, through your support channels.
No common issues have been identified thus far. If you encounter an issue, please reach out on the or, if you are on the Enterprise plan, through your support channels.
No common issues have been identified thus far. If you encounter an issue, please reach out on the or, if you are on the Enterprise plan, through your support channels.
No common issues have been identified thus far. If you encounter an issue, please reach out on the or, if you are on the Enterprise plan, through your support channels.
When you are finished developing your block locally, you will want to initialize it. The procedure to initialize your block is described in the overview page. Please refer to that documentation for details.
When you have initalized and finished testing your block locally, you will want to push it to Edge Impulse. The procedure to push your block to Edge Impulse is described in the overview page. Please refer to that documentation for details.
When you are finished developing your block locally, you will want to initialize it. The procedure to initialize your block is described in the overview page. Please refer to that documentation for details.
When you have initalized and finished testing your block locally, you will want to push it to Edge Impulse. The procedure to push your block to Edge Impulse is described in the overview page. Please refer to that documentation for details.
No common issues have been identified thus far. If you encounter an issue, please reach out on the or, if you are on the Enterprise plan, through your support channels.
You can change the default view (list) to a grid view to quickly overview your datasets by clicking on the icon.
After you have pushed your block to Edge Impluse, it can be used in the same way as any other built-in block.
After you have pushed your block to Edge Impluse, it can be used in the same way as any other built-in block.
After you have pushed your block to Edge Impluse, it can be used in the same way as any other built-in block.
After you have pushed your block to Edge Impluse, it can be used in the same way as any other built-in block.
After you have pushed your block to Edge Impluse, it can be used in the same way as any other built-in block.
After you have pushed your block to Edge Impluse, it can be used in the same way as any other built-in block.
In this tutorial, we will explore how to label image data using GPT-4o, a powerful language model developed by OpenAI. GPT-4o is capable of generating accurate and meaningful labels for images, making it a valuable tool for image classification tasks. By leveraging the capabilities of GPT-4o, we can automate the process of labeling image data, saving time and effort in data preprocessing.
We packaged in a "pre-built Transformation block" (available for all Enterprise plans), an innovative method to distill LLM knowledge.
This pre-built transformation block can be found under the Data sources tab in the Data acquisition view.
The block takes all your unlabeled image files and asks GPT-4o to label them based on your prompt - and we automatically add the reasoning as metadata to your items!
Your prompt should return a single label, e.g.
The GPT-4o model processes images and assigns labels based on the content, filtering out any images that do not meet the quality criteria.
Navigate to the Data acquisition page and add images to your project's dataset. In the video tutorial above, we show how to collect a video recorded directly from a phone, upload it to Edge Impulse and split the video into individual frames.
In the Data sources tab, add the "Label image data using GPT-4o" block:
OpenAI API key: Add your OpenAI API key. This value will be stored as a secret, and won't be shown again.
Prompt: Your prompt should return a single label. For example:
Disable samples w/ label: If a certain label is output, disable the data item - these are excluded from training. Multiple labels are accepted, separate them with a coma.
Max. no. of samples to label: Number of samples to label.
Concurrency: Number of samples to label in parallel.
Auto-convert videos: If set, all videos are automatically split into individual images before labeling.
To edit your configuration, you need to update the json-like steps of your block:
Then, run the block to automatically label the frames.
And here is an example of the returned logs:
Use the labeled data to train a machine learning model. See the end-to-end tutorial Image classification.
In the video tutorial, we deployed the trained model to an MCU-based edge device - the Arduino Nicla Vision.
The small model we tested this on performed exceptionally well, identifying toys in various scenes quickly and accurately. By distilling knowledge from the large LLM, we created a specialized, efficient model suitable for edge deployment.
The latest multimodal LLMs are incredibly powerful but too large for many practical applications. At Edge Impulse, we enable the transfer of knowledge from these large models to smaller, specialized models that run efficiently on edge devices.
Our "Label image data using GPT-4o" block is available for enterprise customers, allowing you to experiment with this technology.
For further assistance, visit our forum.
Blog post: Label image data using GPT-4o blog post
In today’s tech world, CI/CD (Continuous Integration/Continuous Deployment) is crucial for delivering fully tested and up-to-date software or firmware to your customers. This tutorial will guide you through integrating Edge Impulse Studio with GitHub workflows, enabling seamless build and deployment of your Edge Impulse model into your workflow.
Edge Impulse provides a comprehensive REST API for seamless integration with third-party services, allowing for the automation of tasks within Edge Impulse Studio. The GitHub Action we created available here simplifies the process of building and deploying models into your workflow.
This example was adapted from the Edge Impulse Blog - Integrate Your GitHub Workflow with Edge Impulse Studio By Mateusz Majchrzycki.
GitHub repository for your firmware source code.
Edge Impulse project created in the Studio.
Obtain Project ID and API Key
Navigate to your Edge Impulse project in the Studio.
Select "Dashboard" from the left pane, then click on "Keys" at the top.
Note down the Project ID and Project API Key.
Add GitHub Action to Your Workflow
Open your workflow YAML file in your GitHub repository.
Add the following code to your workflow YAML file:
Replace ${{ secrets.PROJECT_ID }} and ${{ secrets.API_KEY }} with your actual Edge Impulse Project ID and API Key.
Extract the Model and SDK
After the build and deployment action, you may want to extract the model and SDK.
Use the following example code in your workflow:
Customize Deployment Type (Optional)
Here's an example of downloading the Arduino library:
Real-world Use Case
Utilize the GitHub Action for CI/CD purposes.
For example, testing public examples to prevent breaking changes.
Here's an example of using the Action with Nordic Semiconductor/Zephyr inference example:
6. Notification for Workflow Errors
Thanks to GitHub Actions notification, the person responsible for the commit that created an error in workflow will be notified.
Integrating Edge Impulse Studio with GitHub workflows enhances your CI/CD pipeline by automating the build and deployment process of your Edge Impulse models. This simplifies the development and testing of firmware, ensuring its accuracy and reliability. GitHub repository for your firmware source code. Edge Impulse project created in the Studio.
By default, the GitHub Action downloads the C++ library. You can customize the deployment type using the deployment_type input parameter. We can use a simple Python script here