1 of 100

Documentation

Getting started

Welcome to Edge Impulse! We enable professional developers and researchers to create the next generation of intelligent products with Edge AI. In this documentation, you'll find user guides, tutorials, and API documentation. If at any point you have questions, visit our .

If you are a beginner, an advanced embedded engineer, an ML engineer, or a data scientist, you may want to use Edge Impulse differently. We have tailored Edge Impulse to suit your needs. Check out the following getting-started guides for a smooth start:

If you're new to the idea of embedded machine learning, or machine learning in general, you may enjoy our quick articles: and

Developer Plan

Edge Impulse offers a highly capable and feature rich Developer plan, for free! If you are a professional developer, hobbyist, maker, student, innovator, or the like who wants to develop edge AI applications without the need for enterprise level features, this is the plan for you.

Enterprise Plan

For startups and enterprises looking to scale edge ML algorithm development from prototype to production, we offer an with additional functionality and access to your own . This plan includes all of the tools needed to go from data collection to model deployment, such as a robust dataset builder to future-proof your data, integrations with all major cloud vendors, dedicated technical support, custom DSP and ML capabilities, and full access to the Edge Impulse APIs to automate your algorithm development.

Suitable for any type of edge AI application

We have some great tutorials, but you have full freedom in the models that you design in Edge Impulse. You can plug in new signal processing blocks and completely new neural networks. See and .

Edge Impulse Datasets

Need inspiration? Check out our that contains publicly available datasets collected, generated or curated by Edge Impulse or its partners.

These datasets highlight specific use cases, helping you understand the types of data commonly encountered in projects like object detection, audio classification, and visual anomaly detection.

API Documentation

You can access any feature in the Edge Impulse Studio through the . We also have the if you want to send data directly, and we have an open to control devices from the Studio.

Community

Edge Impulse offers a thriving community of engineers, developers, researchers, and machine learning experts. Connect with like-minded professionals, share your knowledge, and collaborate to enhance your embedded machine-learning projects. Head to the to ask questions or share your awesome ideas!

Public projects

Think your model is awesome, and want to share it with the world? Go to Dashboard and click Make this project public. This will make your whole project - including all data, machine learning models, and visualizations - available, and can be viewed and cloned by anyone with the URL.

We reference all the public projects here: . If you need some inspiration, just clone a project and fine-tune it to your needs!

For beginners

Welcome to Edge Impulse! If you're new to the world of edge machine learning, you've come to the right place. This guide will walk you through the essential steps to get started with Edge Impulse, a suite of engineering tools for building, training, and deploying machine learning models on edge devices.

Check out our Edge AI Fundamentals course to learn more about edge computing, machine learning, and edge MLOps.

Why Edge Impulse, for beginners?

Edge Impulse empowers you to bring intelligence to your embedded projects by enabling devices to understand and respond to their environment. Whether you want to recognize sounds, identify objects, or detect motion, Edge Impulse makes it accessible and straightforward. Here's why beginners like you are diving into Edge Impulse:

No Coding Required: You don't need to be a coding expert to use Edge Impulse. Our platform provides a user-friendly interface that guides you through the process - this includes many optimized preprocessing and learning blocks, various neural network architectures, and pre-trained models and can generate ready-to-flash binaries to test your models on real devices.
Edge Computing: Your machine learning models are optimized to run directly on your edge devices, ensuring low latency and real-time processing.
Support for Various Sensors: Edge Impulse supports a wide range of sensors, from accelerometers and microphones to cameras, making it versatile for different projects.
Community and Resources: You're not alone on this journey. Edge Impulse offers a supportive community and extensive documentation to help you succeed.

Getting started in a few steps

Ready to begin? Follow these simple steps to embark on your Edge Impulse journey:

Start by creating an Edge Impulse account. It's free to get started, and you'll gain access to all the tools and resources you need.

2. Create a project

Once you're logged in, create your first project. Give it a name that reflects your project's goal, whether it's recognizing sounds, detecting objects, or something entirely unique.

3. Collect/import data

To teach your device, you need data. Edge Impulse provides user-friendly tools for collecting data from your sensors, such as recording audio, capturing images, or reading sensor values. We recommend using a hardware target from this list or your smartphone to start collecting data when you begin with Edge Impulse.

You can also import existing datasets or clone a public project to get familiar with the platform.

Edge Impulse Datasets

Need inspiration? Check out our Edge Impulse datasets collection that contains publicly available datasets collected, generated or curated by Edge Impulse or its partners.

These datasets highlight specific use cases, helping you understand the types of data commonly encountered in projects like object detection, audio classification, and visual anomaly detection.

4. Label your data

Organize your data by labeling it. For example, if you're working on sound recognition, label audio clips with descriptions like "dog barking" or "car horn." You can label your data as you collect it or add labels later, our data explorer is also particularly useful to understand your data.

5. Pre-process your data and train your model

This is where the magic happens. Edge Impulse offers an intuitive model training process through processing blocks and learning blocks. You don't need to write complex code; the platform guides you through feature extraction, model creation, and training.

6. Run the inference on a device

After training your model, you can easily export your model to run in a web browser or on your smartphone, but you can also run it on a wide variety of edge devices, whether it's a Raspberry Pi, Arduino, or other compatible hardware. We also provide ready-to-flash binaries for all the officially supported hardware targets. You don't even need to write embedded code to test your model on real devices!

If you have a device that is not supported, no problem, you can export your model as a C++ library that runs on any embedded device. See Running your impulse locally for more information.

7. Go further

Building Edge AI solutions is an iterative process. Feel free to try our organization hub to automate your machine-learning pipelines, collaborate with your colleagues, and create custom blocks.

Tutorials and resources for beginners

The end-to-end tutorials are perfect for learning how to use Edge Impulse Studio. Try the tutorials:

Motion recognition + anomaly detection,
Keyword spotting,
Sound recognition,
Image classification,
Object detection using bounding boxes (size and location),
Object detection using centroids (location)

These will let you build machine-learning models that detect things in your home or office.

Join the Edge Impulse Community

Remember, you're not alone on your journey. Join the Edge Impulse community to connect with other beginners, experts, and enthusiasts. Share your experiences, ask questions, and learn from others who are passionate about embedded machine learning.

Now that you have a roadmap, it's time to explore Edge Impulse and discover the exciting possibilities of embedded machine learning. Let's get started!

For ML practitioners

Welcome to Edge Impulse! Whether you are a machine learning engineer, MLOps engineer, data scientist, or researcher, we have developed professional tools to help you build and optimize models to run efficiently on any edge device.

In this guide, we'll explore how Edge Impulse empowers you to bring your expertise and your own models to the world of edge AI using either the Edge Impulse Studio, our visual interface, and the Edge Impulse Python SDK, available as a pip package.

Why Edge Impulse, for ML practitioners?

Flexibility: You can choose to work with the tools they are already familiar with and import your models, architecture, and feature processing algorithms into the platform. This means that you can leverage your existing knowledge and workflows seamlessly. Or, for those who prefer an all-in-one solution, Edge Impulse provides enterprise-grade tools for your entire machine-learning pipeline.

Optimized for edge devices: Edge Impulse is designed specifically for deploying machine learning models on edge devices, which are typically resource-constrained, from low-power MCUs up to powerful edge GPUs. We provide tools to optimize your models for edge deployment, ensuring efficient resource usage and peak performance. Focus on developing the best models, we will provide feedback on whether they can run on your hardware target!

Data pipelines: We developed a strong expertise in complex data pipelines (including clinical data) while working with our customers. We support data coming from multiple sources, in any format, and provide tools to perform data alignment and validation checks. All of this in customizable multi-stage pipelines. This means you can build gold-standard labeled datasets that can then be imported into your project to train your models.

Getting started in a few steps

In this getting started guide, we'll walk you through the two different approaches to bringing your expertise to edge devices. Either starting from your dataset or from an existing model.

First, start by creating your .

Start with existing data

You can import data using , , or our . These allow you to easily upload and manage your existing data samples and datasets to Edge Impulse Studio.

We currently accept various file types, including .cbor, .json, .csv, .wav, .jpg, .png, .mp4, and .avi.

If you are working with image datasets, the Studio uploader and the CLI uploader currently handle these types of : Edge Impulse object detection, COCO JSON, Open Images CSV, Pascal VOC XML, Plain CSV, and YOLO TXT.

Organization data

Since the creation of Edge Impulse, we have been helping our customers deal with complex data pipelines, complex data transformation methods and complex clinical validation studies.

The organizational data gives you tools to centralize, validate and transform datasets so they can be easily imported into your projects.

See the documentation.

To visualize how your labeled data items are clustered, use the feature available for most dataset types, where we apply dimensionality reduction techniques (t-SNE or PCA) on your embeddings.

To extract features from your data items, either choose an available (MFE, MFCC, spectral analysis using FFT or Wavelets, etc.) or from your expertise. These can be written in any language.

Similarly, to train your machine learning model, you can choose from different (Classification, Anomaly Detection, Regression, Image or Audio Transfer Learning, Object Detection). In most of these blocks, we expose the Keras API in an . You can also bring your own architecture/training pipeline as a .

Each block will provide on-device performance information showing you the estimated RAM, flash, and latency.

Start with an existing model

If you already have been working on different models for your Edge AI applications, Edge Impulse offers an easy way to upload your models and profile them. This way, in just a few minutes, you will know if your model can run on real devices and what will be the on-device performances (RAM, flash usage, and latency).

You can do this directly from the or using .

Edge Impulse Python SDK is available as a pip package:

From there, you can profile your existing models:

And then directly generate a customizable library or any other supported

Run the inference on a device

You can easily in a .eim format, a Linux executable that contains your signal processing and ML code, compiled with optimizations for your processor or GPU. This executable can then be called with our . We have inferencing libraries and examples for Python, Node.js, C++, and Go.

If you target MCU-based devices, you can generate ready-to-flash binaries for all the officially supported hardware targets. This method will let you test your model on real hardware very quickly.

In both cases, we will provide profiling information about your models so you can make sure your model will fit your edge device constraints.

Tutorials and resources, for ML practitioners

End-to-end tutorials

If you want to get familiar with the full end-to-end flow using Edge Impulse Studio, please have a look at our :

To understand the full potential of Edge Impulse, see our that describes an end-to-end ML workflow for building a wearable health product using Edge Impulse. It handles data coming from multiple sources, data alignment, and a multi-stage pipeline before the data is imported into an Edge Impulse project.

Edge Impulse Python SDK tutorials

While the Edge Impulse Studio is a great interface for guiding you through the process of collecting data and training a model, the Python SDK allows you to programmatically Bring Your Own Model (BYOM), developed and trained on any platform:

Other useful resources

(access Keras API in the studio)

Integrations

For embedded engineers

Welcome to Edge Impulse! When we started Edge Impulse, we initially focused on developing a suite of engineering tools designed to empower embedded engineers to harness the power of machine learning on edge devices. As we grew, we also started to develop advanced tools for ML practitioners to ease the collaboration between teams in organizations.

In this getting started guide, we'll walk you through the essential steps to dive into Edge Impulse and leverage it for your embedded projects.

Why Edge Impulse, for embedded engineers?

Embedded systems are becoming increasingly intelligent, and Edge Impulse is here to streamline the integration of machine learning into your hardware projects. Here's why embedded engineers are turning to Edge Impulse:

Extend hardware capabilities: Edge Impulse can extend hardware capabilities by enabling the integration of machine learning models, allowing edge devices to process complex tasks, recognize patterns, and make intelligent decisions that are complex to develop using rule-based algorithms.
Open-source export formats: Exported models and libraries contain both digital signal processing code and machine learning models, giving you full explainability of the code.
Powerful integrations: Edge Impulse provides complete and documented integrations with various hardware platforms, allowing you to focus on the application logic rather than the intricacies of machine learning.
Support for diverse sensors: Whether you're working with accelerometers, microphones, cameras, or custom sensors, Edge Impulse accommodates a wide range of data sources for your projects.
Predict on-device performances: Models trained in Edge Impulse run directly on your edge devices, ensuring real-time decision-making with minimal latency. We provide tools to ensure the DSP and models developed with Edge Impulse can fit your device constraints.
Device-aware optimization: You have full control over model optimization, enabling you to tailor your machine-learning models to the specific requirements and constraints of your embedded systems. Our EON tuner can help you select the best model by training many different variants of models only from an existing dataset and your device constraints!

Getting started in a few steps

Ready to embark on your journey with Edge Impulse? Follow these essential steps to get started:

Start by creating your Edge Impulse account. Registration is straightforward, granting you immediate access to the comprehensive suite of tools and resources.

2. Create a Project

Upon logging in, initiate your first project. Select a name that resonates with your project's objectives. If you already which hardware target or system architecture you will be using, you can set it up directly in the dashboard's project info section. This will help you to make sure your model fits your device constraints.

3. Data Collection and Labeling

We offer various methods to collect data from your sensors or to import datasets (see Data acquisition for all methods). For the officially supported hardware targets, we provide binaries or simple steps to attach your device to Edge Impulse Studio and collect data from the Studio. However, as an embedded engineer, you might want to collect data from sensors that are not necessarily available on these devices. To do so, you can use the Data forwarder and print out your sensor values over serial (up to 8kHz) or use our C Ingestion SDK, a portable header-only library (designed to reliably store sampled data from sensors at a high frequency in very little memory).

4. Pre-process your data and train your model

Edge Impulse offers an intuitive model training process through processing blocks and learning blocks. You don't need to write Python code to train your model; the platform guides you through feature extraction, model creation, and training. Customize and fine-tune your blocks for optimal performance on your hardware. Each block will provide on-device performance information showing you the estimated RAM, flash, and latency.

5. Run the inference on a device

This is where the fun start, you can easily export your model as ready-to-flash binaries for all the officially supported hardware targets. This method will let you test your model on real hardware very quickly.

In addition, we also provide a wide variety of export methods to easily integrate your model with your application logic. See C++ library to run your model on any device that supports C++ or our guides for Arduino library, Cube.MX CMSIS-PACK, DRP-AI library, DRP-AI TVM i8 library, OpenMV library, Ethos-U library, Meta TF model, Simplicity Studio Component, Tensai Flow library, TensorRT library, TIDL-RT library, etc...

The C++ inferencing library is a portable library for digital signal processing and machine learning inferencing, and it contains native implementations for both processing and learning blocks in Edge Impulse. It is written in C++11 with all dependencies bundled and can be built on both desktop systems and microcontrollers. See Inferencing SDK documentation.

6. Go further

Building Edge AI solutions is an iterative process. Feel free to try our organization hub to automate your machine-learning pipelines, collaborate with your colleagues, and create custom blocks.

Tutorials and resources, for embedded engineers

End-to-end tutorials

If you want to get familiar with the full end-to-end flow, please have a look at our end-to-end tutorials:

Motion recognition + anomaly detection,
Keyword spotting,
Sound recognition,
Image classification,
Object detection using bounding boxes (size and location),
Object detection using centroids (location)

Advanced inferencing tutorials

In the advanced inferencing tutorials section, you will discover useful techniques to leverage our inferencing libraries or how you can use the inference results in your application logic:

Continuous audio sampling
Multi-impulse
Count objects using FOMO

Other useful resources

Join the Edge Impulse Community

Edge Impulse offers a thriving community of embedded engineers, developers, and experts. Connect with like-minded professionals, share your knowledge, and collaborate to enhance your embedded machine-learning projects.

Now that you have a roadmap, it's time to explore Edge Impulse and discover the exciting possibilities of embedded machine learning. Let's get started!

Frequently asked questions (FAQ)

Data

Does Edge impulse integrate with cloud storage services?

Yes. The enterprise version of Edge Impulse can integrate directly with your cloud storage provider to access and transform data.

How is the labeling of the data performed?

Using the Edge Impulse Studio data acquisition tools (like the or ), you can collect data samples manually with a pre-defined label.

If you have a dataset that was collected outside of Edge Impulse, you can upload your dataset using the , , , or . You can then use the Edge Impulse Studio to split up your data into labeled chunks, crop your data samples, and more to create high quality machine learning datasets.

Processing

What signal processing is available in Edge Impulse?

A big part of Edge Impulse are the processing blocks, as they clean up the data, and extract important features from your data before passing it to a machine learning model.

The source code for these processing blocks can be found on GitHub: (and you can build as well).

How does the feature explorer visualize data that has more that 3 dimensions?

Edge Impulse uses (a dimensionality reduction algorithm) to project high dimensionality input data into a 2 or 3 dimensional space. This even works for extremely high dimensionality data such as images.

Learning

What frameworks does Edge Impulse use to train the machine learning models?

We use a wide variety of tools, depending on the machine learning model. For neural networks we typically use TensorFlow and Keras, for object detection models we use TensorFlow with Google's Object Detection API, and for 'classic' non-neural network machine learning algorithms we mainly use sklearn. For neural networks you can see (and modify) the Keras code by clicking ⋮, and selecting Switch to expert mode.

Can I use a model that has been trained outside of Edge Impulse?

Yes you can! Check out our documentation on to see how to import your model into your Edge Impulse project, and using the !

Deployment

What are the minimum hardware requirements to run the Edge Impulse inferencing library on my embedded device?

The minimum hardware requirements for the embedded device depends on the use case. Anything from a Cortex-M0+ for vibration analysis to Cortex-M4F for audio, Cortex-M7 for image classification to Cortex-A for object detection in video should work.

View our for more details.

What is the typical power consumption of the Edge Impulse machine learning processes on my device?

Simple answer: To get an indication of time per inference we show performance metrics in every DSP and ML block in Studio. Multiply this by the active power consumption of your MCU to get an indication of power cost per inference.

More complicated answer: It depends. Normal techniques to conserve power still apply to ML, so try to do as little as possible (do you need to classify every second, or can you do it once a minute?), be smart about when to run inference (can there be an external trigger like a motion sensor before you run inference on a camera?), and collect data in a lower power mode (don't run at full speed when sampling low-resolution data, and see if your sensor can use an interrupt to wake your MCU - rather than polling).

Also see .

What engine does Edge Impulse use to compile the impulse?

It depends on the hardware.

For general-purpose MCUs we typically use EON Compiler with TFLite Micro and additional kernels (including hardware optimization, e.g. via CMSIS-NN, ESP-NN).

On Linux, if you run the impulse on CPU, we use LiteRT (previously Tensorflow Lite).

For accelerators we use a wide variety of other runtimes, e.g. hardcoded network in silicon for Syntiant, custom SNN-based inference engine for Brainchip Akida, DRP-AI for Renesas RZV2L, etc...

Is there a downside to enabling the EON Compiler?

The compiles your neural networks to C++ source code, which then gets compiled into your application. This is great if you need the lowest RAM and ROM possible (EON typically uses 30-50% less memory than LiteRT (previously Tensorflow Lite) but you also lose some flexibility to update your neural networks in the field - as it is now part of your firmware).

By disabling EON we place the full neural network (architecture and weights) into ROM, and load it on demand. This increases memory usage, but you could just update this section of the ROM (or place the neural network in external flash, or on an SD card) to make it easier to update.

What is the .eim model format for Edge Impulse for Linux?

See on the Edge Impulse for Linux pages.

Can I use an unsupported development board or a custom PCB (with a different microcontroller or microprocessor) with Edge Impulse?

Yes! A "supported board" simply means that there is an official or community-supported firmware that has been developed specifically for that board that helps you collect data and run impulses. Edge Impulse is designed to be extensible to computers, smartphones, and a nearly endless array of microcontroller build systems.

You can collect data from you custom board and upload it to Edge Impulse in a variety of ways. For example:

Transmitting data to the
Using the SDK
By (e.g. CBOR, JSON, CSV, WAV, JPG, PNG)

Your trained model can be deployed as part a . It requires some effort, but most build systems will work with our C++ library, as long as that build system has a C++ compiler and there is enough flash/RAM on your device to run the library (which includes the DSP block and model).

Other

How can I cite Edge Impulse in scientific publications?

If you use Edge Impulse in a scientific publication, we would appreciate citations to the following paper:

Tutorials

End-to-end tutorials

This section provides detailed end-to-end tutorials to help you get started with Edge Impulse:

Edge Impulse Datasets

Need inspiration? Check out our Edge Impulse datasets collection that contains publicly available datasets collected, generated or curated by Edge Impulse or its partners.

These datasets highlight specific use cases, helping you understand the types of data commonly encountered in projects like object detection, audio classification, and visual anomaly detection.

Computer vision

Image classification

In this tutorial, you'll use machine learning to build a system that can recognize objects in your house through a camera - a task known as image classification - connected to a microcontroller. Adding sight to your embedded devices can make them see the difference between poachers and elephants, do quality control on factory lines, or let your RC cars drive themselves. In this tutorial you'll learn how to collect images for a well-balanced dataset, how to apply transfer learning to train a neural network, and deploy the system to an embedded device.

At the end of this tutorial, you'll have a firm understanding of how to classify images using Edge Impulse.

There is also a video version of this tutorial:

You can view the finished project, including all data, signal processing and machine learning blocks here: .

1. Prerequisites

For this tutorial, you'll need a .

If you don't have any of these devices, you can also upload an existing dataset through the . After this tutorial you can then deploy your trained machine learning model as a C++ library and run it on your device.

2. Building a dataset

In this tutorial we'll build a model that can distinguish between two objects in your house - we've used a plant and a lamp, but feel free to pick two other objects. To make your machine learning model see it's important that you capture a lot of example images of these objects. When training the model these example images are used to let the model distinguish between them. Because there are (hopefully) a lot more objects in your house than just lamps or plants, you also need to capture images that are neither a lamp or a plant to make the model work well.

Capture the following amount of data - make sure you capture a wide variety of angles and zoom levels:

50 images of a lamp.
50 images of a plant.
50 images of neither a plant nor a lamp - make sure to capture a wide variation of random objects in the same room as your lamp or plant.

You can collect data from the following devices:

- for all other officially supported boards with camera sensors.

Or you can capture your images using another camera, and then upload them by going to Data acquisition and clicking the 'Upload' icon.

Afterwards you should have a well-balanced dataset listed under Data acquisition in your Edge Impulse project. You can switch between your training and testing data with the two buttons above the 'Data collected' widget.

3. Designing an impulse

With the training set in place you can design an impulse. An impulse takes the raw data, adjusts the image size, uses a preprocessing block to manipulate the image, and then uses a learning block to classify new data. Preprocessing blocks always return the same values for the same input (e.g. convert a color image into a grayscale one), while learning blocks learn from past experiences.

For this tutorial we'll use the 'Images' preprocessing block. This block takes in the color image, optionally makes the image grayscale, and then turns the data into a features array. If you want to do more interesting preprocessing steps - like finding faces in a photo before feeding the image into the network -, see the tutorial. Then we'll use a 'Transfer Learning' learning block, which takes all the images in and learns to distinguish between the three ('plant', 'lamp', 'unknown') classes.

In the studio go to Create impulse, set the image width and image height to 96, and add the 'Images' and 'Transfer Learning (Images)' blocks. Then click Save impulse.

Configuring the processing block

To configure your processing block, click Images in the menu on the left. This will show you the raw data on top of the screen (you can select other files via the drop down menu), and the results of the processing step on the right. You can use the options to switch between 'RGB' and 'Grayscale' mode, but for now leave the color depth on 'RGB' and click Save parameters.

This will send you to the 'Feature generation' screen. In here you'll:

Resize all the data.
Apply the processing block on all this data.
Create a 3D visualization of your complete dataset.

Click Generate features to start the process.

Afterwards the 'Feature explorer' will load. This is a plot of all the data in your dataset. Because images have a lot of dimensions (here: 96x96x3=27,648 features) we run a process called 'dimensionality reduction' on the dataset before visualizing this. Here the 27,648 features are compressed down to just 3, and then clustered based on similarity. Even though we have little data you can already see some clusters forming (lamp images are all on the right), and can click on the dots to see which image belongs to which dot.

Configuring the transfer learning model

With all data processed it's time to start training a neural network. Neural networks are a set of algorithms, modeled loosely after the human brain, that are designed to recognize patterns. The network that we're training here will take the image data as an input, and try to map this to one of the three classes.

It's very hard to build a good working computer vision model from scratch, as you need a wide variety of input data to make the model generalize well, and training such models can take days on a GPU. To make this easier and faster we are using transfer learning. This lets you piggyback on a well-trained model, only retraining the upper layers of a neural network, leading to much more reliable models that train in a fraction of the time and work with substantially smaller datasets.

To configure the transfer learning model, click Transfer learning in the menu on the left. Here you can select the base model (the one selected by default will work, but you can change this based on your size requirements), optionally enable data augmentation (images are randomly manipulated to make the model perform better in the real world), and the rate at which the network learns.

Set:

Number of training cycles to 20.
Learning rate to 0.0005.
Data augmentation: enabled.
Minimum confidence rating: 0.7.

Important: If you're using a development board with less memory, like the click Choose a different model and select MobileNetV1 96x96 0.25. This is a smaller transfer learning model.

And click Start training. After the model is done you'll see accuracy numbers, a confusion matrix and some predicted on-device performance on the bottom. You have now trained your model!

4. Validating your model

With the model trained let's try it out on some test data. When collecting the data we split the data up between a training and a testing dataset. The model was trained only on the training data, and thus we can use the data in the testing dataset to validate how well the model will work in the real world. This will help us ensure the model has not learned to overfit the training data, which is a common occurrence.

To validate your model, go to Model testing, select the checkbox next to 'Sample name' and click Classify selected. Here we hit 89% accuracy, which is great for a model with so little data.

To see a classification in detail, click the three dots next to an item, and select Show classification. This brings you to the Live classification screen with much more details on the file (if you collected data with your mobile phone you can also capture new testing data directly from here). This screen can help you determine why items were misclassified.

5. Running the model on your device

With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the preprocessing steps, neural network weights, and classification code - in a single C++ library that you can include in your embedded software.

To run your impulse on either the OpenMV camera or your phone, follow these steps:

OpenMV Cam H7 Plus:
Mobile phone: just click Switch to classification mode at the bottom of your phone screen.

For other boards: click on Deployment in the menu. Then under 'Build firmware' select your development board, and click Build. This will export the impulse, and build a binary that will run on your development board in a single step. After building is completed you'll get prompted to download a binary. Save this on your computer.

Flashing the device

When you click the Build button, you'll see a pop-up with text and video instructions on how to deploy the binary to your particular device. Follow these instructions. Once you are done, we are ready to test your impulse out.

Running the model on the device

We can connect to the board's newly flashed firmware over serial. Open a terminal and run:

To also see a preview of the camera, run:

To run continuous (without a pause every 2 seconds), but without the preview, run:

Congratulations! You've added sight to your sensors. Now that you've trained your model you can integrate your impulse in the firmware of your own embedded device, see . There are examples for Mbed OS, Arduino, STM32CubeIDE, and any other target that supports a C++ compiler. Note that the model we trained in this tutorial is relatively big, but you can choose a smaller transfer learning model.

Or if you're interested in more, see our tutorials on or . If you have a great idea for a different project, that's fine too. Edge Impulse lets you capture data from any sensor, build to extract features, and you have full flexibility in your Machine Learning pipeline with the learning blocks.

We can't wait to see what you'll build! 🚀

Object detection

Object detection tasks take an image and output information about the class and number of objects, position, (and, eventually, size) in the image.

Edge Impulse provides, by default, two different model architectures to perform object detection, MobileNetV2 SSD FPN-Lite uses bounding boxes (objects location and size) and FOMO uses centroids (objects location only).

Want to compare the two models?

See

Bounding Boxes

(bounding boxes) Can run on systems starting from Linux CPUs up to powerful GPUs

Centroid

(centroids) Can run on high-end MCUs, Linux CPUs, and GPUs

Visual anomaly detection

Visual regression

Audio

Time-series

Environmental (Sensor fusion)

Neural networks are not limited to working with one type of data at a time. One of their biggest advantages is that they are incredibly flexible with the type of input data, so long as the format and ordering of that data stays the same from training to inference. As a result, we can use them to perform sensor fusion for a variety of tasks.

Sensor fusion is the process of combining data from different types of sensors or similar sensors mounted in different locations, which gives us more information to make decisions and classifications. For example, you could use temperature data with accelerometer data to get a better idea of a potential anomaly!

In this tutorial, you will learn how to use Edge Impulse to perform sensor fusion on the Arduino Nano 33 BLE Sense.

Example Project: You can find the dataset and impulse used throughout this tutorial in .

Multi-impulse vs multi-model vs sensor fusion

Running multi-impulse refers to running two separate projects (different data, different DSP blocks and different models) on the same target. It will require modifying some files in the EI-generated SDKs. See the

Running multi-model refers to running two different models (same data, same DSP block but different tflite models) on the same target. See how to run a motion classifier model and an anomaly detection model on the same device in .

Sensor fusion refers to the process of combining data from different types of sensors to give more information to the neural network. To extract meaningful information from this data, you can use the same DSP block (like in this tutorial), multiples DSP blocks, or use neural networks embeddings like this tutorial.

1. Prerequisites

For this tutorial, you'll need a .

2. Building a dataset

For this demo, we'll show you how to identify different environments by using a fusion of temperature, humidity, pressure, and light data. In particular, I'll have the Arduino board identify different rooms in my house as well as outside. Note that the we assume that the environment is static--if I turn out lights or the outside temperature changes, the model will not work. However, it demonstrates how we can combine different sensor data with machine learning to do classification!

As we will be collecting data from our Arduino board connected to a computer, it helps to have a laptop that you can move to different rooms.

Create a new project on the Edge Impulse studio.

Connect the Arduino Nano 33 BLE to your computer. Follow the to upload the Edge Impulse firmware to the board and connect it to your project.

Go to Data acquisition. Under Record new data, select your device and set the label to bedroom. Change Sensor to Environmental + Interactional, set the Sample length to 10000 ms and Frequency to 12.5Hz.

Stand in one of your rooms with your Arduino board (and laptop). Click Start sampling and slowly move the board around while data is collected. After sampling is complete, you should see a new data plot with a different line for each sensor.

Variations

Try to stand in different parts of each room while collecting data.

Repeat this process to record about 3 minutes of data for the bedroom class. Try to stand in a different spot in the room while collecting data--we want a robust dataset that represents the features of each room. Head to another room and repeat data collection. Continue doing this until you have around 3 minutes of data for each of the following classes:

Bedroom
Hallway
Outside

You are welcome to try other rooms or locations. For this demo, I found that my bedroom, kitchen, and living room all exhibited similar environmental and lighting properties, so the model struggled to tell them apart.

Head to Dashboard and scroll down to Danger zone. Click Perform train/test split and follow the instructions in the pop-up window to split your dataset into training and testing groups. When you're done, you can head back to Data acquisition to see that your dataset has been split. You should see about 80% of your samples in Training data and about 20% in Test data.

4. Design an Impulse

An impulse is a combination of preprocessing (DSP) blocks followed by machine learning blocks. It will slice up our data into smaller windows, use signal processing to extract features, and then train a machine learning model. Because we are using environmental and light data, which are slow-moving averages, we will use the for preprocessing.

Head to Create impulse. Change the Window increase to 500 ms. Add a Flatten block. Notice that you can choose which environmental and interactional sensor data to include. Deselect proximity and gesture, as we won't need those to detect rooms. Add a Classification (Keras) learning block

Click Save impulse.

5. Configure the Flatten block

Head to Flatten. You can select different samples and move the window around to see what the DSP result will look like for each set of features to be sent to the learning block.

The Flatten block will compute the average, minimum, maximum, root-mean square, standard deviation, skewness, and kurtosis of each axis (e.g. temperature, humidity, brightness, etc.). With 7 axes and 7 features computed for each axis, that gives us 49 features for each window being sent to the learning block. You can see these computed features under Processed features.

Click Save parameters. On the next screen, select Calculate feature importance and click Generate features.

After a few moments, you should be able to explore the features of your dataset to see if your classes are easily separated into categories.

You can also look at the Feature importance section to get an idea of which features are the most important in determining class membership. You can read more about feature importance .

Interestingly enough, it looks like temperature and red light values were the most important features in determining the location of the Arduino board.

6. Configure the neural network

With our dataset collected and features processed, we can train our machine learning model. Click on NN Classifier. Change the Number of training cycles to 300 and click Start training. We will leave the neural network architecture as the default for this demo.

During training, parameters in the neural network's neurons are gradually updated so that the model will try to guess the class of each set of data as accurately as possible. When training is complete, you should see a Model panel appear on the right side of the page.

The Confusion matrix gives you an idea of how well the model performed at classifying the different sets of data. The top row gives the predicted label and the column on the left side gives the actual (ground-truth) label. Ideally, the model should predict the classes correctly, but that's not always the case. You want the diagonal cells from the top-left to the bottom-right to be as close to 100% as possible.

If you see a lot of confusion between classes, it means you need to gather more data, try different features, use a different model architecture, or train for a longer period of time (more epochs). See to learn about ways to increase model performance.

The On-device performance provides some statistics about how the model will likely run on a particular device. By default, an Arm Cortex-M4F running at 80 MHz is assumed to be your target device. The actual memory requirements and run time may vary on different platforms.

7. Model testing

Rather than simply assume that our model will work when deployed, we can run inference on our test dataset as well as on live data.

First, head to Model testing, and click Classify all. After a few moments, you should see results from your test set.

You can click on the three dots next to an item and select Show classification. This will give you a classification result screen where you can see results information in more detail.

Additionally, we can test the impulse in a real-world environment to make sure the model has not overfit the training data. To do that, head to Live classification. Make sure your device is connected to the Studio and that the Sensor, Sample length, and Frequency match what we used to initially capture data.

Click Start sampling. A new sample will be captured from your board, uploaded, and classified. Once complete, you should see the classification results.

In the example above, we sampled 10 seconds of data from the Arduino. This data is split into 1-second windows (the window moves over 0.5 seconds each time), and the data in that window is sent to the DSP block. The DSP block computes the 49 features that are then sent to the trained machine learning model, which performs a forward pass to give us our inference results.

As you can see, the inference results from all of the windows claimed that the Arduino board was in the bedroom, which was true! This is great news for our model--it seems to work even on unseen data.

8. Running the impulse on your device

Now that we have an impulse with a trained model and we've tested its functionality, we can deploy the model back to our device. This means the impulse can run locally without an internet connection to perform inference!

Edge Impulse can package up the entire impulse (preprocessing block, neural network, and classification code) into a single library that you can include in your embedded software.

Click on Deployment in the menu. Select the library that you would like to create, and click Build at the bottom of the page.

Running your impulse locally

See to learn how to deploy your impulse to a variety of platforms.

9. Conclusion

Well done! You've trained a neural network to determine the location of a development board based on a fusion of several sensors working in tandem. Note that this demo is fairly limited--as the daylight or temperature changes, the model will no longer be valid. However, it hopefully gives you some ideas about how you can mix and match sensors to achieve your machine learning goals.

If you're interested in more, see our tutorials on or . If you have a great idea for a different project, that's fine too. Edge Impulse lets you capture data from any sensor, build to extract features, and you have full flexibility in your Machine Learning pipeline with the learning blocks.

We can't wait to see what you'll build! 🚀

Data

Data ingestion

Collecting image data from the Studio

This page is part of Image classification and describes how you can use development boards with an integrated camera to import image data into Edge Impulse.

First, make sure your device is connected on the Devices page in the Edge Impulse Studio. Then, head to Data acquisition, and under 'Record new data', set a label and select 'Camera' as a sensor (most devices have multiple resolutions). This shows you a nice preview of the camera. Then click Start sampling.

A few moments later - depending on the speed of the development board and the resolution - you'll now have an image collected!

Do this until you have captured 30 images per class from a variety of angles. Also make sure to vary the things you capture for the unknown class.

Collecting image data with your mobile phone

This page is part of and describes how you can use your mobile phone to import image data into Edge Impulse.

To add your phone to your project, go to the Devices page, select Connect a new device and select Use your mobile phone. A QR code will pop up. Scan this code with your phone and your phone will pop up on the devices screen.

1. Collecting images

With your phone connected to your project, it's time to start capturing some images and build our dataset. We have a special UI for collecting images quickly, on your phone choose Collecting images?.

On your phone a permission prompt will show up, and then the viewfinder will be displayed. Set the label (in the top corner) to 'lamp', point your camera at your lamp and press Capture.

Afterwards the photo shows up in the studio on the Data acquisition page.

Do this until you have captured 30 images per class from a variety of angles. Also make sure to vary the things you capture for the unknown class.

2. Alternative: upload data directly

Alternatively you can also capture your dataset directly through a different app, and then upload the data directly to Edge Impulse There are both options to do this visually (click the 'Upload' icon on the data acquisition screen), or via the CLI. You can find instructions here: . In this case it's highly recommended to you use square images, as the transfer learning model expects these; and you probably want to resize these images before uploading them to make sure training remains fast.

Collecting image data with the OpenMV Cam H7 Plus

This page is part of Image classification and describes how you can use the OpenMV Cam H7 Plus to build a dataset, and import the data into Edge Impulse.

1. Setting up your environment

To set up your OpenMV camera, and collect some data:

Install the OpenMV IDE.
Follow the OpenMV hardware setup guide to clean the sensor and focus the lens.
Connect a micro-USB cable to the camera, and open the OpenMV IDE. The camera should automatically update to the latest firmware.
Verify that the camera can capture live images, by clicking on the Connect button in the bottom left corner, then pressing Play to run the application.

A live feed from your camera will be displayed in the top right corner of the IDE.

2. Collecting images

Once your camera is up and running, it's time to start capturing some images and build our dataset.

First, set up a new dataset via Tools -> Dataset Editor, select New Dataset.

This opens the 'Dataset editor' panel on the left side, and the 'dataset capture script' in the main panel of the IDE. Here, create three classes: "plant", "lamp" and "unknown". It's important to add an unknown class that contains random images which are neither lamps nor plants.

As we'll build a model that takes in square images, change the 'Dataset capture script' to read:

import sensor, image, time

sensor.reset()
sensor.set_pixformat(sensor.RGB565) # Modify as you like.
sensor.set_framesize(sensor.QVGA) # Modify as you like.
sensor.set_windowing((240, 240)) # Modify as you like.
sensor.skip_frames(time = 2000)

clock = time.clock()

while(True):
    clock.tick()
    img = sensor.snapshot()
    print(clock.fps())

Now you can capture data for the three classes.

Click the Play icon to run the 'dataset capture script' on your OpenMV camera.
Select one of the classes by clicking on the folder name in the 'Dataset editor'.
Take a snap by clicking the Capture data (camera icon) button.

Do this until you have captured 30 images per class from a variety of angles. Also make sure to vary the things you capture for the unknown class.

3. Sending the dataset to Edge Impulse

To import the dataset into Edge Impulse go to Tools > Dataset Editor > Export > Upload to Edge Impulse project.

Then, choose the project name, and the split between training and testing data (recommended to keep this to 80/20).

A duplicate check runs when you upload new data, so you can upload your dataset multiple times (for example, when you've added new files) without adding the same data twice.

Training and testing data split

The split between training and testing data is based on the hash of the file in order to have a deterministic process. As a consequence you may not have a perfect 80/20 split between training and testing, but this process ensures samples are always placed in the same category.

Our dataset now appears under the Data acquisition section of our project.

You can now go back to the Image classification tutorial to build your machine learning model.

Ingest multi-labeled data using the API

This guide provides a quick walk-through on how to upload and update time-series data with multiple labels using the Edge Impulse API.

Getting Started

Prerequisites

API Key - Obtain from your project settings on Edge Impulse.
Example Github repository - Clone the following repository, it contains the scripts and the example files:

Setup Your Environment

Export your API key to use in the upload scripts:

Prepare Your Data

Data File: JSON formatted time-series data. See the
structured_labels.labels File: JSON with structured labels. See the

Upload Data

Use the upload.sh script to send your data and labels to Edge Impulse:

Update Existing Samples

To update a sample, run update-sample.sh with the required project and sample IDs:

🚀

We hope this tutorial has helped you to understand how to ingest multi-label data samples to your Edge Impulse project. If you have any questions, please reach out to us on our .

Synthetic data

Synthetic datasets are a collection of data artificially generated rather than being collected from real-world observations or measurements. They are created using algorithms, simulations, or mathematical models to mimic the characteristics and patterns of real data. Synthetic datasets are a valuable tool to generate data for experimentation, testing, and development when obtaining real data is challenging, costly, or undesirable.

You might want to generate synthetic datasets for several reasons:

Cost Efficiency: Creating synthetic data can be more cost-effective and efficient than collecting large volumes of real data, especially in resource-constrained environments.

Data Augmentation: Synthetic datasets allow users to augment their real-world data with variations, which can improve model robustness and performance.

Data Diversity: Synthetic datasets enable the inclusion of uncommon or rare scenarios, enriching model training with a wider range of potential inputs.

Privacy and Security: When dealing with sensitive data, synthetic datasets provide a way to train models without exposing real information, enhancing privacy and security.

You can generate synthetic data directly from Edge Impulse using the Synthetic Data tab in the Data acquisition view. This tab provides a user-friendly interface to generate synthetic data for your projects. You can create synthetic datasets using a variety of tools and models.

We have put together the following tutorials to help you get started with synthetic datasets generation:

Using integrated models directly available inside Edge Impulse Studio

DALL-E Image Generation Block: Generate image datasets using Dall·E using the .
Whisper Keyword Spotting Generation Block: Generate keyword-spotting datasets using the . Ideal for keyword spotting and speech recognition applications.
Eleven Labs Sound Generation Block: Generate sound datasets using the . Ideal for generating realistic sound effects for various applications.

Note that you will need an API Key/Access Token from the different providers to run the model used to generate the synthetic data.

If you want to create your own synthetic data block, see .

Generate audio datasets using Eleven Labs

Generate audio data using the Eleven Labs Sound Effects models. This integration allows you to generate realistic sound effects for your projects, such as glass breaking, car engine revving, or other custom sounds. You can customize the sound prompts and generate high-quality audio samples for your datasets.

This integration allows you to expand your datasets with sounds that may be difficult or expensive to record naturally. This approach not only saves time and money but also enhances the accuracy and reliability of the models we deploy on edge devices.

Example: Glass Breaking Sound

In this tutorial, we focus on a practical application that can be used in a smart security system, or in a factory to detect incidents, such as detecting the sounds of glass breaking.

There is also a video version of this guide:

Getting Started

You will also need an Eleven Labs account and API Key.

Navigate to Data Acquisition: Once you're in your project, navigate to the Data Acquisition section, go to Synthetic data and select the ElevenLabs Synthetic Audio Generator data source.

Step 1: Get your ElevenLabs API Key

First, get your Eleven Labs API Key. Navigate to the Eleven Labs web interface to get your key and optionally test your prompt.

Step 2: Parameters

Here we will be trying to collect a glass-breaking sound or impact.

Prompt: "glass breaking"
Simple prompts are just that: they are simple, one-sided prompts where we try to get the AI to generate a single sound effect. This could be, for example, “person walking on grass” or “glass breaking.” These types of prompts will generate a single type of sound effect with a few variations either in the same generation or in subsequent generations. All in all, they are fairly simple.
There are a few ways to improve these prompts, however, and that is by adding a little bit more detail. Even if they are simple prompts, they can be made to give better output by improving the prompt itself. For example, something that sometimes works is adding details like “high-quality, professionally recorded footsteps on grass, sound effects foley.” It can require some experimentation to find a good balance between being descriptive and keeping it brief enough to have AI understand the prompt. e.g. high quality audio of window glass breaking
Label: The label of the generated audio sample.
Prompt influence: Between 0 and 1, this setting ranges from giving the AI more creativity in how it interprets the prompt to telling the AI to be more strict in following the exact prompt that you’ve given. 1 being more creative.
Number of samples: Number of samples to generate
Minimum length (seconds): Minimum length of generated audio samples. Audio samples will be padded with silence to minimum length. It also determines how long your generations should be. Depending on what you set this as, you can get quite different results. For example, if I write “kick drum” and set the length to 11 seconds, I might get a full drum loop with a kick drum in it, but that might not be what I want. On the other hand, if I set the length to 1 second, I might just get a one-shot with a single instance of a kick drum.
Frequency (Hz): Audio frequency, ElevenLabs generates data at 44100Hz, so any other value will be resampled.
Upload to category: Data will be uploaded to this category in your project.

See ElevenLabs API documentation for more information

Step 3: Generate samples

Once you've set up your prompt, and api key, run the pipeline to generate the sound samples. You can then view the output in the Data Acquisition section.

Benefits of Using Generative AI for Sound Generation

Enhance Data Quality: Generative AI can create high-quality sound samples that are difficult to record naturally.
Increase Dataset Diversity: Access a wide range of sounds to enrich your training dataset and improve model performance.
Save Time and Resources: Quickly generate the sound samples you need without the hassle of manual recording.
Improve Model Accuracy: High-quality, diverse sound samples can help fill gaps in your dataset and enhance model performance.

Conclusion

By leveraging generative AI for sound generation, you can enhance the quality and diversity of your training datasets, leading to more accurate and reliable edge AI models. This innovative approach saves time and resources while improving the performance of your models in real-world applications. Try out the Eleven Labs block in Edge Impulse today and start creating high-quality sound datasets for your projects.

Labeling

Label audio data using your existing models

This example comes from the Edge Impulse Linux Inferencing Python SDK that has been slightly modify to upload the raw data back to Edge Impulse based on the inference results.

To run the example:

Clone this repository:

git clone [email protected]:edgeimpulse/example-active-learning-linux-python-sdk.git

Install the dependencies:

pip3 install -r requirements.txt

Grab your the API key of the project you want to upload the inferred results raw data:

Past the new key in the EI_API_KEY variable in the audio-classify-export.py file. Alternatively, load it from your environment variable:

export EI_API_KEY='ei_xxxx'

Download your modelfile.eim:

edge-impulse-linux-runner --download modelfile.eim

Run the script:

python3 audio-classify-export.py modelfile.eim yes,no 0.6 0.8

Here are the arguments you can set:

modelfile.eim, path the model.eim
yes,no, labels to upload, separated by comas, no space
0.6, low confidence threshold
0.8, high confidence threshold
<audio_device_ID, optional>

In a keyword spotting model, it can give the following results:

python3 audio-classify-export.py modelfile.eim yes,no 0.6 0.8
['modelfile.eim', 'yes,no', '0.6', '0.8']
{'model_parameters': {'axis_count': 1, 'frequency': 16000, 'has_anomaly': 0, 'image_channel_count': 0, 'image_input_frames': 0, 'image_input_height': 0, 'image_input_width': 0, 'input_features_count': 16000, 'interval_ms': 0.0625, 'label_count': 4, 'labels': ['no', 'noise', 'unknown', 'yes'], 'model_type': 'classification', 'sensor': 1, 'slice_size': 4000, 'use_continuous_mode': True}, 'project': {'deploy_version': 29, 'id': 10487, 'name': 'Keywords Detection', 'owner': 'Demo Team'}}
Loaded runner for "Demo Team / Keywords Detection"
0 --> MacBook Pro Microphone
2 --> Microsoft Teams Audio
3 --> Descript Loopback Recorder
4 --> ZoomAudioDevice
Type the id of the audio device you want to use: 
0
selected Audio device: 0

Result (0 ms.) no: 0.18 noise: 0.16     unknown: 0.20   yes: 0.46
Result (0 ms.) no: 0.13 noise: 0.58     unknown: 0.22   yes: 0.07
Result (0 ms.) no: 0.00 noise: 0.89     unknown: 0.10   yes: 0.01
Result (0 ms.) no: 0.00 noise: 0.01     unknown: 0.04   yes: 0.95
Result (0 ms.) no: 0.00 noise: 0.82     unknown: 0.10   yes: 0.07
Result (0 ms.) no: 0.02 noise: 0.77     unknown: 0.13   yes: 0.08
Result (0 ms.) no: 0.01 noise: 0.14     unknown: 0.26   yes: 0.59
Result (0 ms.) no: 0.07 noise: 0.76     unknown: 0.15   yes: 0.01
Result (0 ms.) no: 0.04 noise: 0.24     unknown: 0.11   yes: 0.61       Uploading sample to Edge Impulse...
Successfully uploaded audio to Edge Impulse.

Result (0 ms.) no: 0.02 noise: 0.93     unknown: 0.04   yes: 0.00
Result (0 ms.) no: 0.01 noise: 0.67     unknown: 0.32   yes: 0.01
Result (0 ms.) no: 0.02 noise: 0.18     unknown: 0.23   yes: 0.57
Result (0 ms.) no: 0.07 noise: 0.70     unknown: 0.22   yes: 0.01
Result (0 ms.) no: 0.03 noise: 0.83     unknown: 0.12   yes: 0.02
Result (0 ms.) no: 0.24 noise: 0.44     unknown: 0.21   yes: 0.11
Result (0 ms.) no: 0.23 noise: 0.25     unknown: 0.42   yes: 0.10
Result (0 ms.) no: 0.04 noise: 0.76     unknown: 0.18   yes: 0.02
Result (0 ms.) no: 0.16 noise: 0.67     unknown: 0.12   yes: 0.05
Result (0 ms.) no: 0.12 noise: 0.81     unknown: 0.06   yes: 0.01
Result (0 ms.) no: 0.54 noise: 0.24     unknown: 0.12   yes: 0.10
Result (0 ms.) no: 0.01 noise: 0.91     unknown: 0.05   yes: 0.03
Result (0 ms.) no: 0.65 Uploading sample to Edge Impulse...
Successfully uploaded audio to Edge Impulse.
noise: 0.08     unknown: 0.17   yes: 0.10
Result (0 ms.) no: 0.00 noise: 0.96     unknown: 0.03   yes: 0.00
Result (0 ms.) no: 0.04 noise: 0.80     unknown: 0.13   yes: 0.03
Result (0 ms.) no: 0.03 noise: 0.27     unknown: 0.16   yes: 0.54
Result (0 ms.) no: 0.05 noise: 0.66     unknown: 0.15   yes: 0.14
Result (0 ms.) no: 0.08 noise: 0.74     unknown: 0.14   yes: 0.04
Result (0 ms.) no: 0.01 noise: 0.87     unknown: 0.11   yes: 0.02
Result (0 ms.) no: 0.01 noise: 0.87     unknown: 0.06   yes: 0.06
...

Label image data using GPT-4o

In this tutorial, we will explore how to label image data using GPT-4o, a powerful language model developed by OpenAI. GPT-4o is capable of generating accurate and meaningful labels for images, making it a valuable tool for image classification tasks. By leveraging the capabilities of GPT-4o, we can automate the process of labeling image data, saving time and effort in data preprocessing.

We packaged in a "pre-built " (available for all Enterprise plans), an innovative method to distill LLM knowledge.

This pre-built transformation block can be found under the tab in the Data acquisition view.

The block takes all your unlabeled image files and asks GPT-4o to label them based on your prompt - and we automatically add the reasoning as metadata to your items!

Your prompt should return a single label, e.g.

How to use it

The GPT-4o model processes images and assigns labels based on the content, filtering out any images that do not meet the quality criteria.

Step 1: Data Collection

Navigate to the page and add images to your project's dataset. In the video tutorial above, we show how to collect a video recorded directly from a phone, upload it to Edge Impulse and split the video into individual frames.

Step 2: Add the labeling block

In the tab, add the "Label image data using GPT-4o" block:

Step 4: Configure the labeling block

OpenAI API key: Add your OpenAI API key. This value will be stored as a secret, and won't be shown again.
Prompt: Your prompt should return a single label. For example:

Disable samples w/ label: If a certain label is output, disable the data item - these are excluded from training. Multiple labels are accepted, separate them with a coma.
Max. no. of samples to label: Number of samples to label.
Concurrency: Number of samples to label in parallel.
Auto-convert videos: If set, all videos are automatically split into individual images before labeling.

Optional: Editing your labeling block

To edit your configuration, you need to update the json-like steps of your block:

Step 5: Execute

Then, run the block to automatically label the frames.

And here is an example of the returned logs:

Step 6: Train your model

Use the labeled data to train a machine learning model. See the end-to-end tutorial .

Step 7: Deployment

In the video tutorial, we deployed the trained model to an MCU-based edge device - the Arduino Nicla Vision.

Results

The small model we tested this on performed exceptionally well, identifying toys in various scenes quickly and accurately. By distilling knowledge from the large LLM, we created a specialized, efficient model suitable for edge deployment.

Conclusion

The latest multimodal LLMs are incredibly powerful but too large for many practical applications. At Edge Impulse, we enable the transfer of knowledge from these large models to smaller, specialized models that run efficiently on edge devices.

Our "Label image data using GPT-4o" block is available for enterprise customers, allowing you to experiment with this technology.

For further assistance, visit our .

Examples & Resources

Blog post:

Feature extraction

Machine learning

In this ML & data engineering section, you will discover useful techniques to train your models, generate synthetic datasets, or to perform advanced feature extraction:

Edge Impulse Python SDK

Synthetic data

Learn about synthetic data and how to integrate synthetic data models into your Edge Impulse project with the following guides:

The following tutorials detail how to work with synthetic datasets in Edge Impulse:

Other

Classification with multiple 2D input features

Neural networks can work with multiple types of data simultaneously, allowing for sophisticated sensor fusion techniques. Here we demonstrate how to use a single model to perform classification using two separate sets of two-dimensional (2D) input features.

For this tutorial, the two sets of input features are assumed to be spectrograms that are passed through two different convolutional branches of a neural network. The two branches of the network are then combined and passed through a final dense output layer for classification. In the general case, however, there could be more than two branches and the input features could represent different sensor data, channels, or outputs from feature extractors.

This example is for when you would like to pass sets of input features to independent branches of a neural network and then combine the results for a final classification.

If you plan on passing all input features through a fully connected network, then these steps are unnecessary; Edge Impulse automatically concatenates output features from processing blocks and learning blocks operate on the concatenated array.

Example use cases

By using this architecture, you can perform sophisticated sensor fusion tasks, potentially improving classification accuracy and robustness compared to single-input models.

This type of model is particularly useful in situations such as:

Combining data from different sensor types (e.g. accelerometer and gyroscope)
Handling output from multiple feature extractors (e.g. time-domain and frequency-domain features)
Analyzing data from sensors in different locations

Model architecture

The architecture in this tutorial consists of separating the model input into two sets of 2D input features (input layer and reshape layers), passing them through independent convolutional branches, and then combining the results for classification (concatenate layer and dense layer). See below for additional details.

Input layer

The input layer accepts a one-dimensional (1D) input of length input_length. The input_length argument is available if you are using the pre-built by Edge Impulse and modifying them through . By default, Edge Impulse flattens the output of all before passing them to the learning block, so the shape is a 1D array.

Reshape layers

The model input is first split into two halves and then reshaped back into 2D formats. In this example, we assumed spectrogram inputs with 50 columns (time frames) each. We calculated the number of rows based on the number of columns and channels, then used the row and column information for the reshape layer.

Convolutional branches

Each half of the input goes through its own convolutional branch that contains multiple layers:

Conv2D layers (8 and 16 filters)
MaxPooling2D layers for downsampling
Dropout layers for regularization
Flatten layer to prepare for concatenation

Output layers

Finally, the outputs of the two independent branches are combined with a concatenation layer. This combined output is then passed through a final dense layer with a softmax activation to perform classification.

Keras model

The following code snippet can be copied and pasted into a using the expert mode feature within Studio.

Extending the model architecture

To extend this model architecture to handle more inputs, you can:

Split the input into more sections (e.g. thirds or quarters instead of halves).
Create additional convolutional branches for each new input section.
Include the new branches in the concatenation step before the final dense layer.

Conclusion

The architecture defined in this tutorial allows the model to process two separate sets of 2D input features independently before combining them for a final classification. Each branch can learn model parameters specific to its input, which can be particularly useful when dealing with different types of sensor data or feature extractors.

Inferencing & post-processing

In the section, you will discover useful techniques to leverage our inferencing libraries or how you can use the inference results in your application logic:

Lifecycle management

At Edge Impulse, we recognize that the lifecycle of your impulse is dynamic. As data grows, unanticipated factors, or drift occurs retraining and redeployment becomes essential. Many of our partners have already starting to address this with integrations to our platform, or documenting details for implementation on aspects like OTA updates to your impulse, and Lifecycle Management. We have put together this section to help you understanding and explore how to create your own implementation of a Lifecycle Management system.

MLOps

MLOps is a set of practices that combines Machine Learning, DevOps, and Data Engineering. The goal of MLOps is to streamline and automate the machine learning lifecycle, including integration, testing, releasing, deployment, and infrastructure management.\

Continuous Integration, Continuous Deployment and Continuous Learning

Continuous Learning is a key concept in the domain of Machine Learning Operations (MLOps), which is a set of practices that combines Machine Learning, DevOps, and Data Engineering. Here is an example of the process:

OTA Infrastructure

In this section we will explore how firmware updates and other scenarios are currently addressed, with traditional OTA. It should help you to get started planning your own updated impulse across a range of platforms. Starting with platform-specific examples like Arduino Cloud, Nordic nRF Connect SDK, Zephyr, and Golioth, Particle Workbench and Blues Wireless.

Finally we will explore building an end-to-end example on the Espressif IDF. By covering a cross section of platforms we hope to provide a good overview of the process and how it can be applied to your own project.

With more generic examples like Arduino, Zephyr and C++ which can be applicable to all other vendors.

These OTA Model Update Tutorials tutorials will help you to get started.

Closing the Loop

Edge AI solutions are typically not just about deploying once; it’s about building a Lifecycle Management ecosystem. You can configure your device to send labeled data back to Edge Impulse for ongoing model refinement, and leverage our version control to track your model performance over time.

This bidirectional data flow can be established with a straightforward call to our ingestion API you can explore how to collect data from your board in the following tutorial:

Collect Data from Board

By integrating these elements, you establish an Lifecycle Management cycle, where the impulse is not static but evolves, learns, and adapts. This adaptation is can be as simple as adding new data to the existing model, or as complex as retraining the model with new data and deploying a new model to the device. Based on metrics you can define, you can trigger this process automatically, or manually. In the esp-idf example, we will explore how to trigger this process manually, and conditionally based on metrics.

Espressif IDF end-to-end example

Conclusion

We hope this section has helped you to understand the process of Lifecycle Management and how to implement it in your own project. If you have any questions, please reach out to us on our forum.

CI/CD with GitHub Actions

Introduction

In today’s tech world, CI/CD (Continuous Integration/Continuous Deployment) is crucial for delivering fully tested and up-to-date software or firmware to your customers. This tutorial will guide you through integrating Edge Impulse Studio with GitHub workflows, enabling seamless build and deployment of your Edge Impulse model into your workflow.

Edge Impulse provides a comprehensive REST API for seamless integration with third-party services, allowing for the automation of tasks within Edge Impulse Studio. The GitHub Action we created available here simplifies the process of building and deploying models into your workflow.

This example was adapted from the Edge Impulse Blog - Integrate Your GitHub Workflow with Edge Impulse Studio By Mateusz Majchrzycki.

Prerequisites

GitHub repository for your firmware source code.
Edge Impulse project created in the Studio.

Steps

Obtain Project ID and API Key

Navigate to your Edge Impulse project in the Studio.
Select "Dashboard" from the left pane, then click on "Keys" at the top.
Note down the Project ID and Project API Key.

Add GitHub Action to Your Workflow

Open your workflow YAML file in your GitHub repository.

Add the following code to your workflow YAML file:

- name: Build and deploy Edge Impulse Model
 uses: edgeimpulse/build-deploy@v1
 id: build-deploy
 with:
  project_id: ${{ secrets.PROJECT_ID }}
  api_key: ${{ secrets.API_KEY }}

Replace ${{ secrets.PROJECT_ID }} and ${{ secrets.API_KEY }} with your actual Edge Impulse Project ID and API Key.

Extract the Model and SDK

After the build and deployment action, you may want to extract the model and SDK.

Use the following example code in your workflow:

- name: Extract the Model and SDK
 run: |
  mkdir temp
  unzip -q "${{ steps.build-deploy.outputs.deployment_file_name }}" -d temp
  mv temp/edge-impulse-sdk/ .
  mv temp/model-parameters/ .
  mv temp/tflite-model/ .
  rm -rf "${{ steps.build-deploy.outputs.deployment_file_name }}"
  rm -rf temp/

Customize Deployment Type (Optional)

By default, the GitHub Action downloads the C++ library. You can customize the deployment type using the deployment_type input parameter. We can use a simple Python script here

Here's an example of downloading the Arduino library:

- name: Build and deploy Edge Impulse Model
 uses: edgeimpulse/build-deploy@v1
 id: build-deploy
 with:
  project_id: ${{ secrets.PROJECT_ID }}
  api_key: ${{ secrets.API_KEY }}
  deployment_type: "arduino"

Real-world Use Case

Utilize the GitHub Action for CI/CD purposes.
For example, testing public examples to prevent breaking changes.
Here's an example of using the Action with Nordic Semiconductor/Zephyr inference example:

```yaml
- name: Build and deploy EI Model
 uses: ./.github/actions/build-deploy
 id: build-deploy
 with:
  project_id: ${{ secrets.PROJECT_ID }}
  api_key: ${{ secrets.API_KEY }}
- name: Extract the EI Model
 run: |
  mkdir ei-model
  unzip -q "${{ steps.build-deploy.outputs.deployment_file_name }}" -d ei-model
  mv ei-model/edge-impulse-sdk/ .
  mv ei-model/model-parameters/ .
  mv ei-model/tflite-model/ .
  rm -rf "${{ steps.build-deploy.outputs.deployment_file_name }}"
  rm -rf ei-model/
- name: Build test app for nRF52840DK
 run: |
  docker run --rm -v $PWD:/app zephyr-ncs-1.9.1:latest west build -b nrf52840dk_nrf52840
```

6. Notification for Workflow Errors

Thanks to GitHub Actions notification, the person responsible for the commit that created an error in workflow will be notified.

Conclusion

Integrating Edge Impulse Studio with GitHub workflows enhances your CI/CD pipeline by automating the build and deployment process of your Edge Impulse models. This simplifies the development and testing of firmware, ensuring its accuracy and reliability. GitHub repository for your firmware source code. Edge Impulse project created in the Studio.

OTA model updates

Introduction

Starting with platform-specific examples like Arduino Cloud, Nordic nRF Connect SDK / Zephyr and Golioth, Particle Workbench and Blues Wireless. Finally we will explore building an end-to-end example on the Espressif IDF.

By covering a cross section of platforms we hope to provide a good overview of the process and how it can be applied to your own project. With more generic examples like Arduino, Zephyr and C++ which can be applicable to all other vendors.

These tutorials will help you to get started with the following platforms:

Prerequisites

Edge Impulse Account: If you haven't got one, .
Trained Impulse: If you're new, follow one of our

Overview

Edge Impulse recognises the need for OTA model updates, as the process is commonly referred to although we are going to be updating the impulse which includes more than just a model, a complete review of your infrastructure is required. Here is an example of the process:

Detect a change

The initiation of an update to your device can be as straightforward as a call to our API to verify the availability of a new deployment. This verification can be executed either by a server or a device with adequate capabilities. Changes can be dependent on a range of factors, including but not limited to the last modified date of the project, the performance of the model, or the release version of the project e.g. last modified date of the project endpoint:

Download the latest impulse

After we inquire about the last modification, and if an update is available, proceed to download the latest build through:

We could add further checking for impulse model performance, project release version tracking or other metrics to ensure the update is valid. However in this series we will try to keep it simple and focus on the core process. Here is an example of a more complete process:

Identify components that influence change: Determine the components of your project that require updates. This could be based on performance metrics, data drift, or new data incorporation.
Retrain: Focus on retraining based on the identified components of your project.
Test and Validate: Before deploying the updated components, ensure thorough testing and validation to confirm their performance before sending the update.
Deploy Updated Components: Utilize available OTA mechanisms to deploy the updated components to your devices. Ensure seamless integration with the existing deployment that remains unchanged.
Monitor on device Performance: Post-deployment, continuously monitor the performance of the updated model to ensure it meets the desired objectives. See Lifecycle Management for more details.

The aim will be to make sure your device is always equipped with the most recent and efficient impulse, enhancing performance and accuracy.

Conclusion

We hope this section has helped you to understand the process of OTA model updates and how to implement it in your own project. If you have any questions, please reach out to us on our .

with Docker on Allxon

Allxon provides essential remote device management solutions to simplify and optimize edge AI device operations. As an AI/IoT ecosystem enabler, connecting hardware (IHV), software (ISV), and service providers (SI/MSP), Allxon serves as the catalyst for fast, seamless connectivity across all systems.

Allxon Over-the-Air (OTA) deployment works perfectly with Edge Impulse OTA model update on NVIDIA Jetson devices. This tutorial guides you through the steps to deploy a new impulse on multiple devices.

Introduction

This guide demonstrates how to deploy and manage Edge Impulse models on NVIDIA Jetson devices using Allxon's Over-the-Air (OTA) deployment capabilities. Allxon provides essential remote device management solutions to streamline and optimize edge AI device operations.

Prerequisites

Before you begin, ensure you have the following:

Updated impulse as a Docker container from Edge Impulse.
Get Allxon officially supported devices(https://www.allxon.com/).
Create an Allxon account.

Getting Started with Allxon

Allxon's services are compatible with a variety of hardware models. Follow these steps to complete the required preparations.

Add a Device to Allxon Portal

Install Allxon Agent: Use the command prompt to install the Allxon Agent on your device.
Pair Your Device: Follow the instructions to add your device to Allxon Portal.

Once added, your devices will appear in the Allxon Portal for management and monitoring.

Allxon OTA Deployment

To perform an OTA deployment, ensure you have your updated Impulse deployed as a Docker container from Edge Impulse.

Steps to Deploy

Generate OTA Artifact: Use the Allxon CLI to generate the OTA artifact.
Deploy OTA Artifact: Follow the Deploy OTA Artifact guide to complete the deployment.

Example Scripts

Below are example scripts to help you set up the OTA deployment.

ota_deploy.sh

#!/bin/bash
set -e
mkdir -p /opt/allxon/tmp/core/appies_ota/model_logs/
./install.sh > /opt/allxon/tmp/core/appies_ota/model_logs/log.txt 2>&1
echo "Model deployment has started. Please check /opt/allxon/tmp/core/appies_ota/model_logs/log.txt for progress."

install.sh


#!/bin/bash
docker run --rm --privileged --runtime nvidia \
 -v /dev/bus/usb/001/002:/dev/video0 \
 -p 80:80 \
 public.ecr.aws/z9b3d4t5/inferencecontainer:73d6ea64bf931f338de5183438915dc390120d5d \
 --api-key <replace with your project api key e.g. ei_07XXX > \
 --run-http-server 1337 &

Two minor modifications have been made to the standard Docker command from Edge Impulse docker deploy:

The -it option has been removed from the Docker command to avoid an error related to the lack of standard input during deployment. An & has been added to the end of the Docker command to send the process to the background.

Conclusion

By following these steps, you can efficiently deploy and manage Edge Impulse models on NVIDIA Jetson devices using Docker through Allxon. This setup leverages Allxon's remote management capabilities to streamline the process of updating and maintaining your edge AI devices.

We hope this section has helped you understand the process of Lifecycle Management and how to implement it in your own project. If you have any questions, please reach out to us on our forum.

with Docker on NVIDIA Jetson

Introduction

Welcome to the tutorial series on OTA Model Updates with Edge Impulse Docker Deploy on Jetson Nano! In this series, we will explore how to update machine learning models over-the-air (OTA) using Edge Impulse and Docker on the Jetson Nano platform.

Prerequisites

Before getting started, make sure you have the following prerequisites:

Jetson Nano Developer Kit
Docker installed on Jetson Nano
Edge Impulse account
Be familiar with Edge Impulse and Docker deploy

Overview

In this tutorial, we will explore how to enable GPU usage and use a camera with the Jetson Nano. We will then deploy a machine learning model using Edge Impulse and Docker on the Jetson Nano. Finally, we will update the model over-the-air (OTA) using Edge Impulse.

Step 1: Enable GPU Usage on Jetson Nano

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

After installing the toolkit, restart the Docker service:


sudo systemctl restart docker

Now you can use the GPU for machine learning tasks in Docker containers.

Step 2: Use a Camera with Jetson Nano

To use a camera with Jetson Nano, you need to install the libgstreamer and libv4l libraries. Run the following commands to install the libraries:


sudo apt-get update
sudo apt-get install -y libgstreamer1.0-0 gstreamer1.0-plugins-base gstreamer1.0-plugins-good gstreamer1.0-plugins-bad gstreamer1.0-plugins-ugly gstreamer1.0-libav libgstrtspserver-1.0-0 libv4l-0 v4l-utils

After installing the libraries, you can use a camera with Jetson Nano.

Step 3: Deploy Machine Learning Model with Edge Impulse and Docker

To deploy a machine learning model with Edge Impulse and Docker, follow these steps:

Export your model from Edge Impulse as a Docker container. Copy the generated Docker command from the deployment section. Modify the Docker command to use the GPU and camera on Jetson Nano. Run the Docker command on Jetson Nano to deploy the model.

Step 4: Update Model Over-the-Air (OTA) with Edge Impulse

To update the model over-the-air (OTA) with Edge Impulse, follow these steps:

Train a new model in Edge Impulse. Export the new model as a Docker container. Copy the generated Docker command from the deployment section and build a new Docker image.


docker build -t my_video_inference_container .
Modify the Docker command to use the GPU and camera on Jetson Nano.

```Dockerfile

FROM nvcr.io/nvidia/l4t-base:r32.5.0

# Install necessary dependencies for video streaming
RUN apt-get update && apt-get install -y \
    ffmpeg \
    v4l-utils \
    && rm -rf /var/lib/apt/lists/*

# Set environment variables
ENV DISPLAY=:0
ENV QT_X11_NO_MITSHM=1

# Expose port for streaming
EXPOSE 80

# Mount USB camera device
RUN ln -s /dev/bus/usb/001/002 /dev/video0

# Command to run your model as a Docker container
CMD ["docker", "run", "--rm", "-it", \
    "-p", "80:80", \
    "public.ecr.aws/z9b3d4t5/inference-container:73d6ea64bf931f338de5183438915dc390120d5d", \
    "--api-key", "ei_07e1e4fad55f06b30839c062076a2ad4bbc174386330493011e75566405a5603", \
    "--run-http-server", "1337"]

Run the Docker command on Jetson Nano to deploy the new model.

docker run --rm -it --privileged --runtime nvidia -v /dev/bus/usb/001/002:/dev/video0 -p 80:80 public.ecr.aws/z9b3d4t5/inference-container:73d6ea64bf931f338de5183438915dc390120d5d --api-key ei_07e1e4fad55f06b30839c062076a2ad4bbc174386330493011e75566405a5603 --run-http-server 1337

Test the new model on Jetson Nano.
Monitor the model performance and update as needed.

Summary

In this tutorial series, we explored how to update machine learning models over-the-air (OTA) using Edge Impulse and Docker on the Jetson Nano platform. We enabled GPU usage, used a camera with Jetson Nano, deployed a machine learning model, and updated the model over-the-air.

Now you can easily update your machine learning models on Jetson Nano devices using Edge Impulse and Docker.

with Nordic Thingy53 and the Edge Impulse app

Introduction

This tutorial is part of the series. If you haven't read the introduction yet, we recommend doing so .

We'll guide you through deploying updated machine learning models over-the-air (OTA) to the Nordic Thingy:53 using Edge Impulse. This process leverages the Nordic Thingy:53 app, allowing users to deploy firmware updates and facilitating on-device testing for Lifecycle Management.

Key Features of Nordic Thingy:53 OTA Updates:

User-initiated firmware deployment via the Nordic Thingy:53 app.
Remote data collection and on-device testing for machine learning models.
Seamless integration with Edge Impulse for Lifecycle Management.

Prerequisites

Edge Impulse Account: Sign up if you don't have one .
Trained Impulse: If you're new, follow one of our
Nordic Thingy:53: Have the device ready and charged.
Nordic Thingy:53 App: Installed on your smartphone or tablet.

Preparation

Begin by connecting your Nordic Thingy:53 to the Edge Impulse platform and setting it up for data collection and model deployment.

Step-by-Step Guide

1. Setting Up Nordic Thingy:53 with Edge Impulse

Connect your Nordic Thingy:53 to the Edge Impulse using the Nordic Thingy:53 app. This will be your interface for managing the device and deploying updates.

2. Collecting Data and Training the Model

Use the Nordic Thingy:53 to collect relevant data for your machine learning application.
Upload this data to Edge Impulse and train your model.

3. Deploying the Model via the Nordic Thingy:53 App

Once your model is trained and ready, use the Nordic Thingy:53 app to deploy it to the device.
The app allows you to initiate the OTA update, which downloads and installs the latest firmware containing the new model.

4. Remote On-Device Testing

Conduct remote testing through the app to evaluate the model's performance in real-world scenarios.
This step is crucial for validating the effectiveness of your machine learning model.

5. Continuous Improvement Cycle

Continuously collect new data with the Nordic Thingy:53.
Re-train your model on Edge Impulse with this new data.
Deploy these updates to the Thingy:53 via the app, maintaining the cycle of Lifecycle Management.

Conclusion

This tutorial provides a straightforward approach to implementing OTA updates and Lifecycle Management on the Nordic Thingy:53 using Edge Impulse. The user-friendly Nordic Thingy:53 app facilitates easy deployment of firmware updates, making it ideal for rapid prototyping and iterative machine learning model development.

Additional Resources

This guide helps users leverage the capabilities of the Nordic Thingy:53 for advanced IoT applications, ensuring devices are always updated with the latest intelligence and improvements.

API examples

The exposes programmatic access to most functionality in the studio and it is particularly useful when it comes to automating tasks. You can use the API to edit the labels of many samples at once, train models, or create new impulses. In addition, you can subscribe to events, such as when a new file is processed by the ingestion service. An Edge Impulse Python API bindings is also available as the edgeimpulse-api pip package.

See the following examples:

Python SDK examples

While the Edge Impulse Studio is a great interface for guiding you through the process of collecting data and training a model, the edgeimpulse Python SDK allows you to programmatically Bring Your Own Model (BYOM), developed and trained on any platform. See here for the Python SDK API reference documentation.

With the following tutorials, you will learn how to use the Edge Impulse Python SDK with a number of other machine-learning frameworks and platforms:

Using the Edge Impulse Python SDK to run EON Tuner
Using the Edge Impulse Python SDK to upload and download data
Using the Edge Impulse Python SDK with Hugging Face
Using the Edge Impulse Python SDK with SageMaker Studio
Using the Edge Impulse Python SDK with TensorFlow and Keras
Using the Edge Impulse Python SDK with Weights & Biases

Edge Impulse Studio

Organization hub

Your Edge Impulse Organization enables your team to collaborate on multiple datasets, automation, and models in a shared workspace. It provides tools to automate data preparation tasks with reusable pipelines, enabling data transformation, preparation, and analysis of sensor data at scale. Allowing anyone in your team to quickly access relevant data through familiar tools, add versions and add traceability to your machine learning models, and lets you quickly create and monitor your Edge Impulse projects for optimal on-device performance.

To get started, follow these guides:

- to add collaborators with different access rights.
- to track and visualize all your metrics over time.
- to connect a storage bucket, to learn how to deal with such complex data infrastructure and to import your data samples into your projects.
- to chain several transformation blocks and to import data into your projects.
- to run your transformation blocks and get an overview of the running jobs.
- to allow external parties to securely contribute data to your datasets.
- to match any specific use cases using dedicated cloud jobs.

Health reference design

We have built a that describes an end-to-end ML workflow for building a wearable health product using Edge Impulse. It is a good tutorial to understand how we handle complex data infrastructure and discover the organization's advanced features.

Usage metrics

Existing enterprise users or enterprise trial users can view their entitlement limits via the dashboard of their enterprise organization:

This view allows you to see your organization's current usage of total users, projects, compute time and storage limits. To increase your organization's limits, select the Request limit increase button to contact sales.

Users

Within an organization you can work on one or more projects with multiple people. These can be colleagues, outside researchers, or even members of the community. They will only get access to the specific data in the project, and not to any of the raw data in your organizational datasets.

To invite a user in an organization, click on the "Add user button, enter the email address and select the role:

Organization Users vs Project Users

It is important to note that there are two types of users in Edge Impulse: Project Users and Organization Users.

Organization Users, typically holding roles like Admin, are responsible for the overarching management and customization of organizational elements, including datasets, storage buckets, and white label attributes. These users also encompass the capabilities of Project Users.

Conversely, Project Users, often in roles such as Member or Guest, are limited to specific project involvement, focusing on collaboration and contributions at the project level, without access to broader organizational management functions. They are granted access only to certain project data to maintain the security of raw data in organizational datasets.

Organization User Roles

For a more granular look at the capabilities of each role, see the table below:

Admin

Admins have full rights on the organization, overseeing organizational and white label functionalities, including dataset management and storage bucket updates. They also have all the rights of a Project Member.

Full Rights on the Organization
Project User rights
Manage organization datasets
Update and add storage buckets
Verify bucket connectivity
Customize white label (where applicable) attributes like themes and information
API access for organization and white label management

Member

Members have full access on the datasets, custom blocks but cannot join a project without being invited.

Broad Access, with Restrictions on Project Joining
Project User rights
Full access to datasets and custom blocks
Can collaborate on projects, but only by invitation
Can access metrics via API

Guest

Guests have restricted access, limited to selected datasets within the projects they are associated with.

Limited Access to Selected Datasets
Project User rights
Access to selected datasets within the project they are invited to
Cannot access raw data in organizational datasets
Cannot access metrics via API

To give someone access to a project only, go to your project's dashboard, and find the "Collaborators" widget. Click the '+' icon, and type the username or e-mail address of the other user.

Data campaigns

The "data campaigns" feature allows you to quickly track your experiments and your models' development progresses. It is an overview of your pipelines where you can easily extract useful information from your datasets and correlate those metrics with your model performances.

It has been primarily designed to follow clinical research data processes. In August 2023, we released this feature for every enterprise user as we see value in being able to track metrics between your datasets and your projects.

Setting up your dashboard

To get started, navigate to the Data campaigns tab in your organization:

Click on + Create new dashboard.

Give your dashboard a name, and select one or more collaborators to receive the daily updates by email. If you don't want to be spammed, you can select when you want to receive these updates, either Always, On new data, changes or on error, or Never. Finally, set the last number of days shown in the graphs:

You can create as many dashboards as needed, simply click on + Create a new dashboard from the dropdown available under your current dashboard:

If you want to delete a dashboard, Click on Actions... -> Delete dashboard

Setting up your campaign

Once your dashboard is created, you can add your custom campaigns. It's where you will specify which metrics are important to you and your use case. Click on Actions... -> Add campaign

Fill the form to create your campaign:

Name: Name of your data campaign.

Description: Description of your data campaign.

Campaign coordinators: Add the collaborators that are engaged with this campaign

Datasets: Select the datasets you want to visualize in your campaign. You can add several datasets.

Projects: Select the projects you want to visualize in your campaign. You can add several projects.

Pipelines: Select the pipeline that is associated with your campaign.Note that this is for reference only, it is currently not displayed in your campaign

Links: Select between the link type you need. Options are Github, Spreadsheet, Text Document, Code repository, List or Folder. Add a name and the link. This place is useful for other collaborators to have all the needed information about your project, gathered in one place under your campaign.

Addition queries to track: These queries are data filters that need to be written in the SQL WHERE format. See Querying data for more information. For example metadata->age >= 18` will return the data samples from adult patients.

You can then save your data campaign and it will be added to your dashboard:

This dashboard shows the metrics' progress from the Health reference design data

If you want to edit or delete your campaign, click on the "⋮" button on the right side of your campaign:

Data pipelines

Building data pipelines is a very useful feature where you can stack several transformation blocks similar to the . They can be used in a standalone mode (just execute several transformation jobs in a pipeline), to feed a dataset or to feed a project.

The examples in the screenshots below shows how to create and use a pipeline to create the 'AMS Activity 2022' dataset.

Create a pipeline

To create a new pipeline, click on '+Add a new pipeline:

Get the steps from your transformation blocks

In your organization workspace, go to Custom blocks -> Transformation and select Run job on the job you want to add.

Select Copy as pipeline step and paste it to the configuration json file.

You can then paste the copied step directly to the respected field.

Below, you have an option to feed the data to either a organisation dataset or an Edge Impulse project

Schedule and notify

By default, your pipeline will run every day. To schedule your pipeline jobs, click on the ⋮ button and select Edit pipeline.

Once the pipeline has successfully finished, it can send an email to the Users to notify.

Run the pipeline

Once your pipeline is set, you can run it directly from the UI, from external sources or by scheduling the task.

Run the pipeline from the UI

To run your pipeline from Edge Impulse studio, click on the ⋮ button and select Run pipeline now.

Run the pipeline from code

To run your pipeline from Edge Impulse studio, click on the ⋮ button and select Run pipeline from code. This will display an overlay with curl, Node.js and Python code samples.

You will need to create an API key to run the pipeline from code.

Webhooks

Another useful feature is to create a webhook to call a URL when the pipeline has ran. It will run a POST request containing the following information:

Transformation blocks

Transformation blocks are a flexible way to pre-process, i.e. transform, your through transformation jobs before using it within a project or to create another dataset. They can also be used standalone (not operating on organizational data) as a way to run cloud processing jobs for specific actions.

You can use transformation blocks to fetch external datasets, augment and create variants of your data samples, extract metadata from config files, create helper graphs, align and interpolate measurements across sensors, remove duplicate entries, and more.

Running a transformation block

Transformation blocks can be run individually through , stacked together to run in series in , or run as a (standalone mode only) importing data into a project. Please refer to the respective documentation for details.

Operating modes

Transformation blocks operate in one of three modes: file, directory, or standalone.

File

Transformation blocks that have been configured with the file operating mode will only appear in the block dropdown for transformation jobs.

As the name implies, file transformation blocks operate on files. The directory specified in the input data path field for the transformation job is automatically scanned for files. Any files that are found are shown on the right hand side as the selected items.

Standalone

Transformation blocks that have been configured with the standalone operating mode can appear in the block dropdown for transformation jobs, for project data sources, or both depending on how the block has been set up.

As the name implies, standalone transformation blocks do not operate on any files or directories. No dataset selection fields will be shown for standalone blocks when creating a transformation job. There will be no files or directories shown on the right hand side as the selected items.

Pre-built transformation blocks

Edge Impulse has the ability to incorporate pre-built transformation blocks into the platform and make these available for all organizations to use. Pre-built transformation blocks may be added over time as recurring needs or interests emerge.

As thees blocks are added to the platform, they will be found within your organization by going to the Transformation left sidebar menu item under Custom blocks. The pre-built blocks will be listed under the Public blocks section at the bottom of the transformation blocks overview page.

Custom transformation blocks

Please refer to the documentation for details.

Troubleshooting

Additional Resources

Validating clinical data

Using Checklists

You can optionally show a check mark in the list of data items, and show a check list for data items. This can be used to quickly view which data items are complete (if you need to capture data from multiple sources) or whether items are in the right format.

Checklists look trivial, but are actually very powerful as they give quick insights in dataset issues. Missing these issues until after the study is done can be very expensive.

Checklists are written to ei-metadata.json and are automatically being picked up by the UI.

Checklists are driven by the metadata for a data item. Set the ei_check metadata item to 0 or 1 to show a check mark in the list. To show an item in the checklist, set an ei_check_KEYNAME metadata item to 0 or 1.

To query for items with or without a check mark, use a filter in the form of:

To make it easy to create these lists on the fly you can set these metadata items directly from a

Example

For the reference design described and used in the previous pages, the combiner takes in a data item, and writes out:

A checklist, e.g.:
- ✔ - PPG file present
- ✔ - Accelerometer file present
- ✘ - Correlation between HR/PPG HR is at least 0.5
If the checklist is OK, a combined.parquet file.
A hr.png file with the correlation between HR found from PPG, and HR from the reference device. This is useful for two reasons:
- If the correlation is too low we're looking at the wrong file, or data is missing.
- Verify if the PPG => HR algorithm actually works.

This makes it easy to quickly see if the data is in the right format, and if the data is complete. If the checklist is not OK, the data item is not used in the training set.

Querying clinical data

Organizational datasets contain a powerful query system which lets you explore and slice data. You control the query system through the 'Filter' text box, and you use a language which is very similar to SQL ().

For example, here are some queries that you can make:

dataset like '%PPG%' - returns all items and files from the study.
bucket_name = 'edge-impulse-health-reference-design' AND -- labels sitting,walking - returns data whose label is 'sitting' and 'walking, and that is stored in the 'edge-impulse-health-reference-design' bucket.
metadata->>'ei_check' = 0 - return data that have a metadata field 'ei_check' which is '0'.
created > DATE('2022-08-01') - returns all data that was created after Aug 1, 2022.

After you've created a filter, you can select one or more data items, and select Actions...>Download selected to create a ZIP file with the data files. The file count reflects the number of files returned by the filter.

The previous queries all returned all files for a data item. But you can also query files through the same filter. In that case the data item will be returned, but only with the files selected. For example:

file_name LIKE '%.png' - returns all files that end with .png.

If you have an interesting query that you'd like to share with your colleagues, you can just share the URL. The query is already added to it automatically.

All available fields

These are all the available fields in the query interface:

dataset - Dataset.
bucket_id - Bucket ID.
bucket_name - Bucket name.
bucket_path - Path of the data item within the bucket.
id - Data item ID.
name - Data item name.
total_file_count - Number of files for the data item.
total_file_size - Total size of all files for the data item.
created - When the data item was created.
metadata->key - Any item listed under 'metadata'.
file_name - Name of a file.
file_names - All filenames in the data item, that you can use in conjunction with CONTAINS. E.g. find all items with file X, but not file Y: file_names CONTAINS 'x' AND not file_names CONTAINS 'y'.

Select AI hardware

The target configuration tool allows you to define your Target device and Application budget according to your project's requirements. This flow is designed to help you optimize your impulse, processing, learn block, or imported model for your specific target hardware, ensuring that your impulse will run efficiently on your device or custom architecture.

The configuration form can be accessed from the top-level navigation. The form allows you to select from a range of processor types, architectures, and clock rates. For a custom device, you could for example select Low-end MCU and specify the clock rate, RAM, ROM, and maximum allowed latency for your application.

Accessing the configuration panel

By default, the form shows 'Cortex-M4F 80MHz' as the target device. You can change this by clicking on Change Target device. You can select from a range of processor types, architectures, and clock rates. For a custom device, you could for example select Low-end MCU and specify the clock rate, RAM, ROM, and maximum allowed latency for your application.

Navigate to the top-level menu in Edge Impulse Studio.
Click on Change Target Device to open the configuration panel.

Configure your target device and application budget

Lets walk you through some of the current options for configuring your device and application budget:

Target Device: Select the type of target device you are configuring from options like "Custom" or specific development boards.
Processor Type Selection: Selecting a processor type dynamically adjusts available architecture options and fields to suit your hardware:
- For Low-end MCU: This option allows you to specify clock rate, RAM, and ROM, suitable for 'Cortex-M' architectures.
- For AI Accelerators: Selecting this disables the clock rate field, reflecting the unique requirements of AI accelerator devices.
Custom Device Configuration: Choosing to configure a custom device opens fields to precisely define its capabilities, ensuring your project setup is accurately tailored to your hardware.

Special options for Custom Targets:

The form allows you to select from a range of processor types, architectures, and clock rates. For a custom device, you could for example select Low-end MCU and specify the clock rate, RAM, ROM, and maximum allowed latency for your application.

Custom: Select this for custom hardware specifications or devices not listed in Edge Impulse, allowing for a customized hardware profile. Selection Options

Processor Type & Architecture

Choose from a variety of processor types and architectures. Your selection determines which options and fields are available to accurately configure your device. Estimations for GPU, AI accelerator, or NPU devices are not computed using clock speed, or but rather the device's unique capabilities.

Processor Type: Selections range from various processor types. Choosing GPU, AI accelerator' or NPU deactivates the clock speed option, as it's irrelevant for device estimation.
Processor Architecture (Optional): Specify your device's architecture to refine its configuration (e.g., Cortex-M0+, Cortex-M4F, Cortex-M7).
Clock Rate (Optional): Set the clock rate for relevant processor types to estimate operational capabilities accurately. The units shown will be indicated by the | MHz | GHz as relevant to the scale of processor. As previously stated the clock rate field is disabled for GPU, AI accelerator, or NPU devices.
Accelerator: If the device supports hardware acceleration, select from available options such as Arm Cortex-U55, NVIDIA Jetson Nano, and others.
Device ID (Optional): Provide a unique identifier for your custom device model or chip architecture variant for easy recognition and setup.
Custom Device Name (Optional): Provide a unique name for your custom device to easily identify it in your project.

Application Budget - RAM, ROM, and Latency

The application budget section allows you to specify the maximum allowed latency, RAM, and ROM for your application. These values are used to estimate the performance of your model on your target device.

RAM: Specify the amount of RAM available on your device in kilobytes (kB).
ROM: Specify the amount of ROM available on your device in kilobytes (kB).
Latency: Specify the maximum allowed latency for your application in milliseconds (ms).

Save Target: Save your custom device and application budget configuration to apply it to your project.

After customizing your target device and application budget, click Save target. With the target device set, navigate to the EON Tuner to see the configuration in action. The target device can be seen at the top level of navigation on all screens within your project. Your custom device name (e.g., 'my first mcu') and the specified parameters (100 ms latency, 256 kB RAM, 1024 kB ROM) are visible. The target device configuration is also taken into account during the performance estimation for deployment.

Once saved the target device can be seen at the top level of navigation on all screens within your project. Your custom device name (e.g., 'my first mcu') and the specified parameters (100 ms latency, 256 kB RAM, 1024 kB ROM) are visible. The target device configuration is also taken into account during the performance estimation for deployment.

Additional functionalities:

There are some additional considerations for targets mentioned in the tabs below, particularly for their generated binaries models and deployment formats.

For Ethos enabled targets, you can select them as profiling targets. After training, the Vela compiled model will be available on the dashboard. This feature is exclusive to Ethos enabled targets. These assets are only available in our studio, when deploying and selecting the deployment format, the library will include the Vela compiled model converted to a C header. While this is sufficient for most users, those needing the binary file directly on the flash may find it inconvenient to convert it back from a C header.

Key Differences:
- C++ Library: The library will include the Vela compiled model converted to a C header.
- Vela Library: The Vela compiled model will be available on the dashboard.

C2 and IMIX users, who need Vela compiled models using other non standard inferencing engine that need to be converted to TFLite files instead of the C++ library.

Key Differences:
- Deployment Format: The deployment format will include the TFLite model.

When selecting the Neox target as a profiling target, the training pipeline involves training a float model, which is then converted to an int8 model using a custom quantizer, rather than the TensorFlow framework's quantizer.

Key Differences:
- Deployment Format: The deployment format will include the int8 model.
- Custom Quantizer: The int8 model is generated using a custom quantizer.

Summary

The target-driven flow in Edge Impulse Studio allows you to configure your target device and application budget according to your project's requirements. This flow is designed to help you optimize your impulse for your specific target hardware, ensuring that your impulse will run efficiently on your device.

We hope this feature is helpful, and intuitive. If you have any questions or suggestions, feel free to reach out to us at . We're always happy to hear from you!

Devices

There is a wide variety of devices that you can connect to your Edge Impulse project. These devices can help you collect datasets for your project, test your trained ML model and even deploy your ML model directly to your development board with a pre-built binary application (for fully supported development platforms).

On the Devices tab, you'll find a list of all your connected devices and a guide on how to connect new devices that are currently supported by Edge Impulse.

To connect a new device, click on the Connect a new device button on the top right of your screen.

You will get a pop-up with multiple options of devices you can connect to your Edge Impulse project. Available options include:

Python API bindings example

The Python SDK is built on top of the Edge Impulse Python API bindings, which is known as the edgeimpulse_api package. These are Python wrappers for all of the web API calls that you can use to interact with Edge Impulse projects programmatically (i.e. without needing to use the Studio graphical interface).

The API reference guide for using the Python API bindings can be found here.

This example will walk you through the process of using the Edge Impulse API bindings to upload data, define an impulse, process features, train a model, and deploy the impulse as a C++ library.

After creating your project and copying the API key, feel free to leave the project open in a browser window so you can watch the changes as we make API calls. You might need to refresh the browser after each call to see the changes take affect.

Important! This project will add data and remove any current features and models in a project. We highly recommend creating a new project when running this notebook! Don't say we didn't warn you if you mess up an existing project.

# Install the Edge Impulse API bindings and the requests package
!python -m pip install edgeimpulse-api requests

import json
import re
import os
import pprint
import time

import requests

# Import the API objects we plan to use
from edgeimpulse_api import (
    ApiClient,
    BuildOnDeviceModelRequest,
    Configuration,
    DeploymentApi,
    DSPApi,
    DSPConfigRequest,
    GenerateFeaturesRequest,
    Impulse,
    ImpulseApi,
    JobsApi,
    ProjectsApi,
    SetKerasParameterRequest,
    StartClassifyJobRequest,
    UpdateProjectRequest,
)

You will need to obtain an API key from an Edge Impulse project. Log into edgeimpulse.com and create a new project. Open the project, navigate to Dashboard and click on the Keys tab to view your API keys. Double-click on the API key to highlight it, right-click, and select Copy.

Note that you do not actually need to use the project in the Edge Impulse Studio. We just need the API Key.

Paste that API key string in the EI_API_KEY value in the following cell:

# Settings
API_KEY = "ei_dae2..." # Change this to your Edge Impulse API key
API_HOST = "https://studio.edgeimpulse.com/v1"
DATASET_PATH = "dataset/gestures"
OUTPUT_PATH = "."

Initialize API clients

The Python API bindings use a series of submodules, each encapsulating one of the API subsections (e.g. Projects, DSP, Learn, etc.). To use these submodules, you need to instantiate a generic API module and use that to instantiate the individual API objects. We'll use these objects to make the API calls later.

To configure a client, you generally create a configuration object (often from a dict) and then pass that object as an argument to the client.

# Create top-level API client
config = Configuration(
    host=API_HOST,
    api_key={"ApiKeyAuthentication": API_KEY}
)
client = ApiClient(config)

# Instantiate sub-clients
deployment_api = DeploymentApi(client)
dsp_api = DSPApi(client)
impulse_api = ImpulseApi(client)
jobs_api = JobsApi(client)
projects_api = ProjectsApi(client)

Initialize project

Before uploading data, we should make sure the project is in the regular impulse flow mode, rather than BYOM mode. We'll also need the project ID for most of the other API calls in the future.

Notice that the general pattern for calling API functions is to instantiate a configuration/request object and pass it to the API method that's part of the submodule. You can find which parameters a specific API call expects by looking at the call's documentation page.

API calls (links to associated documentation):

# Get the project ID, which we'll need for future API calls
response = projects_api.list_projects()
if not hasattr(response, "success") or getattr(response, "success") == False:
    raise RuntimeError("Could not obtain the project ID.")
else:
    project_id = response.projects[0].id

# Print the project ID
print(f"Project ID: {project_id}")

# Create request object with the required parameters
update_project_request = UpdateProjectRequest.from_dict({
    "inPretrainedModelFlow": False,
})

# Update the project and check the response for errors
response = projects_api.update_project(
    project_id=project_id,
    update_project_request=update_project_request,
)
if not hasattr(response, "success") or getattr(response, "success") == False:
    raise RuntimeError("Could not obtain the project ID.")
else:
    print("Project is now in impulse workflow.")

Upload dataset

We'll start by downloading the gesture dataset from this link. Note that the ingestion API is separate from the regular Edge Impulse API: the URL and interface are different. As a result, we must construct the request manually and cannot rely on the Python API bindings.

We rely on the ingestion service using the string before the first period in the filename to determine the label. For example, "idle.1.cbor" will be automatically assigned the label "idle." If you wish to set a label manually, you must specify the x-label parameter in the headers. Note that you can only define a label this way when uploading a group of data at a time. For example, setting "x-label": "idle" in the headers would give all data uploaded with that call the label "idle."

API calls used with associated documentation:

Ingestion service

# Download and unzip gesture dataset
!mkdir -p dataset/
!wget -P dataset -q https://cdn.edgeimpulse.com/datasets/gestures.zip
!unzip -q dataset/gestures.zip -d {DATASET_PATH}

def upload_files(api_key, path, subset):
    """
    Upload files in the given path/subset (where subset is "training" or
    "testing")
    """

    # Construct request
    url = f"https://ingestion.edgeimpulse.com/api/{subset}/files"
    headers = {
        "x-api-key": api_key,
        "x-disallow-duplicates": "true",
    }

    # Get file handles and create dataset to upload
    files = []
    file_list = os.listdir(os.path.join(path, subset))
    for file_name in file_list:
        file_path = os.path.join(path, subset, file_name)
        if os.path.isfile(file_path):
            file_handle = open(file_path, "rb")
            files.append(("data", (file_name, file_handle, "multipart/form-data")))

    # Upload the files
    response = requests.post(
        url=url,
        headers=headers,
        files=files,
    )

    # Print any errors for files that did not upload
    upload_responses = response.json()["files"]
    for resp in upload_responses:
        if not resp["success"]:
            print(resp)

    # Close all the handles
    for handle in files:
        handle[1][1].close()

# Upload the dataset to the project
print("Uploading training dataset...")
upload_files(API_KEY, DATASET_PATH, "training")
print("Uploading testing dataset...")
upload_files(API_KEY, DATASET_PATH, "testing")

Create an impulse

Now that we uploaded our data, it's time to create an impulse. An "impulse" is a combination of processing (feature extraction) and learning blocks. The general flow of data is:

data -> input block -> processing block(s) -> learning block(s)

Only the processing and learning blocks make up the "impulse." However, we must still specify the input block, as it allows us to perform preprocessing, like windowing (for time series data) or cropping/scaling (for image data).

Your project will have one input block, but it can contain multiple processing and learning blocks. Specific outputs from the processing block can be specified as inputs to the learning blocks. However, for simplicity, we'll just show one processing block and one learning block.

Note: Historically, processing blocks were called "DSP blocks," as they focused on time series data. In Studio, the name has been changed to "Processing block," as the blocks work with different types of data, but you'll see it referred to as "DSP block" in the API.

It's important that you define the input block with the same parameters as your captured data, especially the sampling rate! Additionally, the processing block axes names must match up with their names in the dataset.

API calls (links to associated documentation):

# To start, let's fetch a list of all the available blocks
response = impulse_api.get_impulse_blocks(
    project_id=project_id
)
if not hasattr(response, "success") or getattr(response, "success") is False:
    raise RuntimeError("Could not get impulse blocks.")

# Print the available input blocks
print("Input blocks")
print(json.dumps(json.loads(response.to_json())["inputBlocks"], indent=2))

# Print the available processing blocks
print("Processing blocks")
print(json.dumps(json.loads(response.to_json())["dspBlocks"], indent=2))

# Print the available learning blocks
print("Learning blocks")
print(json.dumps(json.loads(response.to_json())["learnBlocks"], indent=2))

# Give our impulse blocks IDs, which we'll use later
processing_id = 2
learning_id = 3

# Impulses (and their blocks) are defined as a collection of key/value pairs
impulse = Impulse.from_dict({
    "inputBlocks": [
        {
            "id": 1,
            "type": "time-series",
            "name": "Time series",
            "title": "Time series data",
            "windowSizeMs": 1000,
            "windowIncreaseMs": 500,
            "frequencyHz": 62.5,
            "padZeros": True,
        }
    ],
    "dspBlocks": [
        {
            "id": processing_id,
            "type": "spectral-analysis",
            "name": "Spectral Analysis",
            "implementationVersion": 4,
            "title": "processing",
            "axes": ["accX", "accY", "accZ"],
            "input": 1,
        }
    ],
    "learnBlocks": [
        {
            "id": learning_id,
            "type": "keras",
            "name": "Classifier",
            "title": "Classification",
            "dsp": [processing_id],
        }
    ],
})

# Delete the current impulse in the project
response = impulse_api.delete_impulse(
    project_id=project_id
)
if not hasattr(response, "success") or getattr(response, "success") is False:
    raise RuntimeError("Could not delete current impulse.")

# Add blocks to impulse
response = impulse_api.create_impulse(
    project_id=project_id,
    impulse=impulse
)
if not hasattr(response, "success") or getattr(response, "success") is False:
    raise RuntimeError("Could not create impulse.")

Configure processing block

Before generating features, we need to configure the processing block. We'll start by printing all the available parameters for the spectral-analysis block, which we set when we created the impulse above.

API calls (links to associated documentation):

# Get processing block config
response = dsp_api.get_dsp_config(
    project_id=project_id,
    dsp_id=processing_id
)

# Construct user-readable parameters
settings = []
for group in response.config:
    for item in group.items:
        element = {}
        element["parameter"] = item.param
        element["description"] = item.help
        element["currentValue"] = item.value
        element["defaultValue"] = item.default_value
        element["type"] = item.type
        if hasattr(item, "select_options") and \
            getattr(item, "select_options") is not None:
            element["options"] = [i.value for i in item.select_options]
        settings.append(element)

# Print the settings
print(json.dumps(settings, indent=2))

# Define processing block configuration
config_request = DSPConfigRequest.from_dict({
    "config": {
        "scale-axes": 1.0,
        "input-decimation-ratio": 1,
        "filter-type": "none",
        "analysis-type": "FFT",
        "fft-length": 16,
        "do-log": True,
        "do-fft-overlap": True,
        "extra-low-freq": False,
    }
})

# Set processing block configuration
response = dsp_api.set_dsp_config(
    project_id=project_id,
    dsp_id=processing_id,
    dsp_config_request=config_request
)
if not hasattr(response, "success") or getattr(response, "success") is False:
    raise RuntimeError("Could not start feature generation job.")
else:
    print("Processing block has been configured.")

Run processing block to generate features

After we've defined the impulse, we then want to use our processing block(s) to extract features from our data. We'll skip feature importance and feature explorer to make this go faster.

Generating features kicks off a job in Studio. A "job" involves instantiating a Docker container and running a custom script in the container to perform some action. In our case, that involves reading in data, extracting features from that data, and saving those features as Numpy (.npy) files in our project.

Because jobs can take a while, the API call will return immediately. If the call was successful, the response will contain a job number. We can then monitor that job and wait for it to finish before continuing.

API calls (links to associated documentation):

def poll_job(jobs_api, project_id, job_id):
    """Wait for job to complete"""

    # Wait for job to complete
    while True:

        # Check on job status
        response = jobs_api.get_job_status(
            project_id=project_id,
            job_id=job_id
        )
        if not hasattr(response, "success") or getattr(response, "success") is False:
            print("ERROR: Could not get job status")
            return False
        else:
            if hasattr(response, "job") and hasattr(response.job, "finished"):
                if response.job.finished:
                    print(f"Job completed at {response.job.finished}")
                    return response.job.finished_successful
            else:
                print("ERROR: Response did not contain a 'job' field.")
                return False

        # Print that we're still running and wait
        print(f"Waiting for job {job_id} to finish...")
        time.sleep(2.0)

# Define generate features request
generate_features_request = GenerateFeaturesRequest.from_dict({
    "dspId": processing_id,
    "calculate_feature_importance": False,
    "skip_feature_explorer": True,
})

# Generate features
response = jobs_api.generate_features_job(
    project_id=project_id,
    generate_features_request=generate_features_request,
)
if not hasattr(response, "success") or getattr(response, "success") is False:
    raise RuntimeError("Could not start feature generation job.")

# Extract job ID
job_id = response.id

# Wait for job to complete
success = poll_job(jobs_api, project_id, job_id)
if success:
    print("Features have been generated.")
else:
    print(f"ERROR: Job failed. See https://studio.edgeimpulse.com/studio/{project_id}/jobs#show-job-{job_id} for more details.")

# Optional: download NumPy features (x: training data, y: training labels)
print("Go here to download the generated features in NumPy format:")
print(f"https://studio.edgeimpulse.com/v1/api/{project_id}/dsp-data/{processing_id}/x/training")
print(f"https://studio.edgeimpulse.com/v1/api/{project_id}/dsp-data/{processing_id}/y/training")

Use learning block to train model

Now that we have trained features, we can run the learning block to train the model on those features. Note that Edge Impulse has a number of learning blocks, each with different methods of configuration. We'll be using the "keras" block, which uses TensorFlow and Keras under the hood.

You can use the get_keras and set_keras functions to configure the granular settings. We'll use the defaults for that block and just set the number of epochs and learning rate for training.

API calls (links to associated documentation):

 # Define training request
keras_parameter_request = SetKerasParameterRequest.from_dict({
    "mode": "visual",
    "training_cycles": 10,
    "learning_rate": 0.001,
    "train_test_split": 0.8,
    "skip_embeddings_and_memory": True,
})

# Train model
response = jobs_api.train_keras_job(
    project_id=project_id,
    learn_id=learning_id,
    set_keras_parameter_request=keras_parameter_request,
)
if not hasattr(response, "success") or getattr(response, "success") is False:
    raise RuntimeError("Could not start training job.")

# Extract job ID
job_id = response.id

# Wait for job to complete
success = poll_job(jobs_api, project_id, job_id)
if success:
    print("Model has been trained.")
else:
    print(f"ERROR: Job failed. See https://studio.edgeimpulse.com/studio/{project_id}/jobs#show-job-{job_id} for more details.")

Now that the model has been trained, we can go back to the job logs to find the accuracy metrics for both the float32 and int8 quantization levels. We'll need to parse the logs to find these. Because the logs are printed with the most recent events first, we'll work backwards through the log to find these metrics.

def get_metrics(response, quantization=None):
    """
    Parse the response to find the accuracy/training metrics for a given
    quantization level. If quantization is None, return the first set of metrics
    found.
    """
    metrics = None
    delimiter_str = "calculate_classification_metrics"

    # Skip finding quantization metrics if not given
    if quantization:
        quantization_found = False
    else:
        quantization_found = True

    # Parse logs
    for log in reversed(response.to_dict()["stdout"]):
        data_field = log["data"]
        if quantization_found:
            substrings = data_field.split("\n")
            for substring in substrings:
                substring = substring.strip()
                if substring.startswith(delimiter_str):
                    metrics = json.loads(substring[len(delimiter_str):])
                    break
        else:
            if data_field.startswith(f"Calculating {quantization} accuracy"):
                quantization_found = True

    return metrics

# Get the job logs for the previous job
response = jobs_api.get_jobs_logs(
    project_id=project_id,
    job_id=job_id
)
if not hasattr(response, "success") or getattr(response, "success") is False:
    raise RuntimeError("Could not get job log.")

# Print training metrics (quantization is "float32" or "int8")
quantization = "float32"
metrics = get_metrics(response, quantization)
if metrics:
    print(f"Training metrics for {quantization} quantization:")
    pprint.pprint(metrics)
else:
    print("ERROR: Could not get training metrics.")

Test the impulse

As with any good machine learning project, we should test the accuracy of the model using our holdout ("testing") set. We'll call the classify API function to make that happen and then parse the job logs to get the results.

In most cases, using int8 quantization will result in a faster, smaller model, but you will slightly lose some accuracy.

API calls (links to associated documentation):

 # Set the model quantization level ("float32", "int8", or "akida")
quantization = "int8"
classify_request = StartClassifyJobRequest.from_dict({
    "model_variants": quantization
})

# Start model testing job
response = jobs_api.start_classify_job(
    project_id=project_id,
    start_classify_job_request=classify_request
)
if not hasattr(response, "success") or getattr(response, "success") is False:
    raise RuntimeError("Could not start classify job.")

# Extract job ID
job_id = response.id

# Wait for job to complete
success = poll_job(jobs_api, project_id, job_id)
if success:
    print("Inference performed on test set.")
else:
    print(f"ERROR: Job failed. See https://studio.edgeimpulse.com/studio/{project_id}/jobs#show-job-{job_id} for more details.")

# Get the job logs for the previous job
response = jobs_api.get_jobs_logs(
    project_id=project_id,
    job_id=job_id
)
if not hasattr(response, "success") or getattr(response, "success") is False:
    raise RuntimeError("Could not get job log.")

# Print
metrics = get_metrics(response)
if metrics:
    print(f"Test metrics for {quantization} quantization:")
    pprint.pprint(metrics)
else:
    print("ERROR: Could not get test metrics.")

Deploy the impulse

Now that you've trained the model, let's build it as a C++ library and download it. We'll start by printing out the available target devices. Note that this list changes depending on how you've configured your impulse. For example, if you use a Syntiant-specific learning block, then you'll see Syntiant boards listed. We'll use the "zip" target, which gives us a generic C++ library that we can use for nearly any hardware.

The engine must be one of:

tflite
tflite-eon
tflite-eon-ram-optimized
tensorrt
tensaiflow
drp-ai
tidl
akida
syntiant
memryx

We'll use tflite, as that's the most ubiquitous.

modelType is the quantization level. Your options are:

float32
int8

In most cases, using int8 quantization will result in a faster, smaller model, but you will slightly lose some accuracy.

API calls (links to associated documentation):

# Get the available devices
response = deployment_api.list_deployment_targets_for_project_data_sources(
    project_id=project_id
)
if not hasattr(response, "success") or getattr(response, "success") is False:
    raise RuntimeError("Could not get device list.")

# Print the available devices
targets = [x.to_dict()["format"] for x in response.targets]
for target in targets:
    print(target)

# Choose the target hardware (from the list above), engine,
target_hardware = "zip"
engine = "tflite"
quantization = "int8"

# Construct request
device_model_request = BuildOnDeviceModelRequest.from_dict({
    "engine": engine,
    "modelType": quantization
})

# Start build job
response = jobs_api.build_on_device_model_job(
    project_id=project_id,
    type=target_hardware,
    build_on_device_model_request=device_model_request,
)
if not hasattr(response, "success") or getattr(response, "success") is False:
    raise RuntimeError("Could not start feature generation job.")

# Extract job ID
job_id = response.id

# Wait for job to complete
success = poll_job(jobs_api, project_id, job_id)
if success:
    print("Impulse built.")
else:
    print(f"ERROR: Job failed. See https://studio.edgeimpulse.com/studio/{project_id}/jobs#show-job-{job_id} for more details.")

# Get the download link information
response = deployment_api.download_build(
    project_id=project_id,
    type=target_hardware,
    model_type=quantization,
    engine=engine,
    _preload_content=False,
)
if response.status != 200:
    raise RuntimeError("Could not get download information.")

# Find the file name in the headers
file_name = re.findall(r"filename\*?=(.+)", response.headers["Content-Disposition"])[0].replace("utf-8''", "")
file_path = os.path.join(OUTPUT_PATH, file_name)

# Write the contents to a file
with open(file_path, "wb") as f:
    f.write(response.data)

You should have a .zip file in the same directory as this notebook. Download or move it to somewhere else on your computer and unzip it. You can now follow this guide to link and compile the library as part of an application.

Multi-impulse (C++)

Once you successfully trained or imported a model, you can use Edge Impulse to download a C++ library that bundles both your signal processing and your machine learning model. Until recently, we could only run one impulse on MCUs.

This tutorial is for advanced users. We will provide limited support on the forum until this feature is fully integrated into the platform. If you have subscribed to an Enterprise plan, you can contact our customer success or solution engineering team.

In this tutorial, we will see how to run multiple impulses using the downloaded C++ libraries of two different projects.

We have put together a custom deployment block that will automate all the processes and provide a C++ library that can be compiled and run as a standalone.

In this page, we will explain the high level concepts of how to merge two impulses. Feel free to look at the code to gain a deeper understanding.

Multi-impulse vs multi-model vs sensor fusion

Sensor fusion refers to the process of combining data from different types of sensors to give more information to the neural network. See how to use sensor fusion in this tutorial.

Also see this video (starting min 13):

Examples

Make sure you have at least two impulses fully trained.

You can use one of the following examples:

Audio + Image classification

This example can be used for an intrusion detection system. We will use a first model to detect glass-breaking sounds, if we detected this sound, we will then classify an image to see if there is a person or not in the image. In this tutorial, we will use the following public projects:

Merge multiple impulses into a single C++ Library

The source code and the generator script can be found here.

By default, the quantized version is used when downloading the C++ libraries. To use float32, add the option --float32 as an argument.

Similarly by default the EON compiled model is used, if you want to use full tflite then add the option --full-tflite and be sure to include a recent version of tensorflow lite compiled for your device architecture in the root of your project in a folder named tensorflow-lite

If you need a mix of quantized and float32, you can look at the dzip.download_model function call in generate.py and change the code accordingly.

By default, the block will download cached version of builds. You can force new builds using the --force-build option.

Locally

Retrieve API Keys of your projects and run the generate.py command as follows:

python generate.py --out-directory output --api-keys ei_0b0e...,ei_acde... --quantization-map <0/1>,<0/1>

Docker

Build the container:docker build -t multi-impulse .

Then run:docker run --rm -it -v $PWD:/home multi-impulse --api-keys ei_0b0e...,ei_acde...

Custom deployment block

Initialize the custom block - select Deployment block and Library when prompted:edge-impulse-blocks init

Push the block:edge-impulse-blocks push

Then go your Organization and Edit the deployment block with:

CLI arguments: --api-keys ei_0b0e...,ei_acde...
Privaliged mode: Enabled

See Edge Impulse Studio -> Organizations -> Custom blocks -> Deployment blocks documentation for more details about custom deployment blocks.

Understanding the process

If you have a look at the generate.py script, it streamline the process of generating a C++ library from multiple impulses through several steps:

Library Download and Extraction:

If the script detects that the necessary projects are not already present locally, it initiates the download of C++ libraries required for edge deployment. These libraries are fetched using API keys provided by the user.
Libraries are downloaded and extracted into a temporary directory. If the user specifies a custom temporary directory, it's used; otherwise, a temporary directory is created.

Customization of Files:

For each project's library, the script performs several modifications:

At the file name level:
- It adds a project-specific suffix to certain patterns in compiled files within the tflite-model directory. This customization ensures that each project's files are unique.
- Renamed files are then copied to a target directory, mainly the first project's directory.
At the function name level:
- It edits model_variables.h functions by adding the project-specific suffix to various patterns. This step ensures that model parameters are correctly associated with each project.

Merging the projects

model_variables.h is merged into the first project's directory to consolidate model information.
The script saves the intersection of lines between trained_model_ops_define.h files for different projects, ensuring consistency.

Copying Templates:

The script copies template files from a templates directory to the target directory. The template available includes files with code structures and placeholders for customization. It's adapted from the example-standalone-inferencing example available on Github.

Generating Custom Code:

The script retrieves impulse IDs from model_variables.h for each project. Impulses are a key part of edge machine learning models.
Custom code is generated for each project, including functions to get signal data, define raw features, and run the classifier.
This custom code is inserted into the main.cpp file of each project at specific locations.

Archiving for Deployment:

Finally, the script archives the target directory, creating a zip file ready for deployment. This zip file contains all the customized files and code necessary for deploying machine learning models on edge devices.

When changing between projects and running generate.py locally:

You may need to include the --force-build option to ensure correctness of the combined library.

Compiling and running the multi-impulse library

Now to test the library generated:

Download and unzip your Edge Impulse C++ multi-impulse library into a directory
Copy a test sample's raw features into the features[] array in source/main.cpp
Enter make -j in this directory to compile the project. If you encounter any OOM memory error try make -j4 (replace 4 with the number of cores available)
Enter ./build/app to run the application
Compare the output predictions to the predictions of the test sample in the Edge Impulse Studio

Want to add your own business logic?

You can change the template you want to use in step 4 to use another compilation method, implement your custom sampling strategy and how to handle the inference results in step 5 (apply post-processing, send results somewhere else, trigger actions, etc.).

Limitations

General limitations:

The custom ML accelerator deployments are unlikely to work (TDA4VM, DRPAI, MemoryX, Brainchip).
The custom tflite kernels (ESP NN, Silabs MVP, Arc MLI) should work - but may require some additional work. I.e: for ESP32 you may need to statically allocate arena for the image model.
In general, running multiple impulses on an MCU can be challenging due to limited processing power, memory, and other hardware constraints. Make sure to thoroughly evaluate the capabilities and limitations of your specific MCU and consider the resource requirements of the impulses before attempting to run them concurrently.

Use case specific limitations:

The model_metadata.h comes from the first API Key of your project. This means some #define statement might be missing or conflicting.

Object detection: If you want to run at least one Object Detection project. Make sure to use this project API KEY first! This will set the #define EI_CLASSIFIER_OBJECT_DETECTION 1 and eventually the #define EI_HAS_FOMO 1. Note that you can overwrite them manually but it requires an extra step.
Anomaly detection: If your anomaly detection model API Key is not in the first position, the model-parameter/anomaly_metadata.h file will not be included.
Visual anomaly detection AND time-series anomaly detection (K-Means or GMM): It is currently not possible to combine two different anomaly detection models. The #define EI_CLASSIFIER_HAS_ANOMALY statement expect ONLY one of the following argument:
```
#define EI_ANOMALY_TYPE_UNKNOWN                   0
#define EI_ANOMALY_TYPE_KMEANS                    1
#define EI_ANOMALY_TYPE_GMM                       2
#define EI_ANOMALY_TYPE_VISUAL_GMM                3
```

Troubleshooting

Segmentation fault

If you see the following segmentation fault, make sure to subtract and merge the trained_model_ops_define.h or tflite_resolver.h

./build/app
run_classifier with audio impulse returned: 0
Timing: DSP 0 ms, inference 0 ms, anomaly 0 ms
Predictions:
  Background: 0.00000
  Glass_Breaking: 0.99609
zsh: segmentation fault  ./build/app

FileExistsError: [Errno 17] File exists

If you see an error like the following, you probably used twice the same API Key:

Project ID is 517331
Export ZIP saved in: temp/517331/cubes-visual-ad-v12.zip (6218297 Bytes)
Project ID is 517331
Traceback (most recent call last):
  File "generate.py", line 49, in <module>
    os.makedirs(download_path)
  File "/Users/luisomoreau/.pyenv/versions/3.8.10/lib/python3.8/os.py", line 223, in makedirs
    mkdir(name, mode)
FileExistsError: [Errno 17] File exists: 'temp/517331'

Make sure you use distinct projects.

Manual procedure

When we first wrote this tutorial, we explained how to merge two impulses manually; This process is now deprecated due to recent changes in our C++ SDK, some files and functions may have been renamed.

See the legacy steps

Some files and function names have changed

The general concepts remain valid but due to recent changes in our C++ inferencing SDK, some files and function names may have changed.

Download the impulses from your projects

Head to your projects' deployment pages and download the C++ libraries:

Make sure to select the same model versions (EON-Compiled enabled/disabled and int8/float32) for your projects.

Extract the two archive in a directory (multi-impulse for example).

Rename the tflite model files

Rename the tflite model files:

Go to the tflite-model directory in your extracted archives and rename the following files by post-fixing them with the name of the project:

for EON compiled projects: tflite_learn_[block-id]_compiled.cpp/tflite_learn_[block-id]_compiled.h.
for non-EON-compiled projects: tflite_learn_[block-id].cpp/tflite_learn_[block-id].h.

Original structure:

>  multi-impulse % tree -L 3
.
├── audio
│   ├── CMakeLists.txt
│   ├── README.txt
│   ├── edge-impulse-sdk
│   │   ├── CMSIS
│   │   ├── LICENSE
│   │   ├── LICENSE-apache-2.0.txt
│   │   ├── README.md
│   │   ├── classifier
│   │   ├── cmake
│   │   ├── dsp
│   │   ├── porting
│   │   ├── sources.txt
│   │   ├── tensorflow
│   │   └── third_party
│   ├── model-parameters
│   │   ├── model_metadata.h
│   │   └── model_variables.h
│   └── tflite-model
│       ├── tflite_learn_5_compiled.cpp
│       ├── tflite_learn_5_compiled.h
│       └── trained_model_ops_define.h
└── image
    ├── CMakeLists.txt
    ├── README.txt
    ├── edge-impulse-sdk
    │   ├── CMSIS
    │   ├── LICENSE
    │   ├── LICENSE-apache-2.0.txt
    │   ├── README.md
    │   ├── classifier
    │   ├── cmake
    │   ├── dsp
    │   ├── porting
    │   ├── sources.txt
    │   ├── tensorflow
    │   └── third_party
    ├── model-parameters
    │   ├── model_metadata.h
    │   └── model_variables.h
    └── tflite-model
        ├── tflite_learn_5_compiled.cpp
        ├── tflite_learn_5_compiled.h
        └── trained_model_ops_define.h

22 directories, 22 files

New structure after renaming the files:

>multi-impulse % tree -L 3
.
├── audio
│   ├── CMakeLists.txt
│   ├── README.txt
│   ├── edge-impulse-sdk
│   │   ├── CMSIS
│   │   ├── LICENSE
│   │   ├── LICENSE-apache-2.0.txt
│   │   ├── README.md
│   │   ├── classifier
│   │   ├── cmake
│   │   ├── dsp
│   │   ├── porting
│   │   ├── sources.txt
│   │   ├── tensorflow
│   │   └── third_party
│   ├── model-parameters
│   │   ├── model_metadata.h
│   │   └── model_variables.h
│   └── tflite-model
│       ├── trained_model_compiled_audio.cpp
│       ├── trained_model_compiled_audio.h
│       └── trained_model_ops_define.h
└── image
    ├── CMakeLists.txt
    ├── README.txt
    ├── edge-impulse-sdk
    │   ├── CMSIS
    │   ├── LICENSE
    │   ├── LICENSE-apache-2.0.txt
    │   ├── README.md
    │   ├── classifier
    │   ├── cmake
    │   ├── dsp
    │   ├── porting
    │   ├── sources.txt
    │   ├── tensorflow
    │   └── third_party
    ├── model-parameters
    │   ├── model_metadata.h
    │   └── model_variables.h
    └── tflite-model
        ├── trained_model_compiled_image.cpp
        ├── trained_model_compiled_image.h
        └── trained_model_ops_define.h

22 directories, 22 files

Rename the variables in the tflite-model directory

Rename the variables (EON model functions, such as trained_model_input etc. or tflite model array names) by post-fixing them with the name of the project.

e.g: Change the trained_model_compiled_audio.h from:

#ifndef tflite_learn_5_GEN_H
#define tflite_learn_5_GEN_H

#include "edge-impulse-sdk/tensorflow/lite/c/common.h"

// Sets up the model with init and prepare steps.
TfLiteStatus tflite_learn_5_init( void*(*alloc_fnc)(size_t,size_t) );
// Returns the input tensor with the given index.
TfLiteStatus tflite_learn_5_input(int index, TfLiteTensor* tensor);
// Returns the output tensor with the given index.
TfLiteStatus tflite_learn_5_output(int index, TfLiteTensor* tensor);
// Runs inference for the model.
TfLiteStatus tflite_learn_5_invoke();
//Frees memory allocated
TfLiteStatus tflite_learn_5_reset( void (*free)(void* ptr) );


// Returns the number of input tensors.
inline size_t tflite_learn_5_inputs() {
  return 1;
}
// Returns the number of output tensors.
inline size_t tflite_learn_5_outputs() {
  return 1;
}

#endif

to:

#include "edge-impulse-sdk/tensorflow/lite/c/common.h"

// Sets up the model with init and prepare steps.
TfLiteStatus tflite_learn_audio_init( void*(*alloc_fnc)(size_t,size_t) );
// Returns the input tensor with the given index.
TfLiteStatus tflite_learn_audio_input(int index, TfLiteTensor* tensor);
// Returns the output tensor with the given index.
TfLiteStatus tflite_learn_audio_output(int index, TfLiteTensor* tensor);
// Runs inference for the model.
TfLiteStatus tflite_learn_audio_invoke();
//Frees memory allocated
TfLiteStatus tflite_learn_audio_reset( void (*free)(void* ptr) );


// Returns the number of input tensors.
inline size_t tflite_learn_audio_inputs() {
  return 1;
}
// Returns the number of output tensors.
inline size_t tflite_learn_audio_outputs() {
  return 1;
}

#endif

Tip: Use an IDE to use the "Find and replace feature.

Here is a list of the files that need to be modified (the names may change if not compiled with the EON compiler) in folders for both projects:

tflite-model/tflite_learn_[block-id]_compiled.h
tflite-model/tflite_learn_[block-id]_compiled.cpp

Rename the variables and structs in model-parameters/model_variables.h

Be careful here when using the "find and replace" from your IDE, NOT all variables looking like _model_ need to be replaced.

Example for the audio project:

#ifndef _EI_CLASSIFIER_MODEL_VARIABLES_H_
#define _EI_CLASSIFIER_MODEL_VARIABLES_H_

#include <stdint.h>
#include "model_metadata.h"

#include "tflite-model/trained_model_compiled_audio.h"
#include "edge-impulse-sdk/classifier/ei_model_types.h"
#include "edge-impulse-sdk/classifier/inferencing_engines/engines.h"

const char* ei_classifier_inferencing_categories_audio[] = { "Background", "Glass_Breaking" };

uint8_t ei_dsp_config_3_axes_audio[] = { 0 };
const uint32_t ei_dsp_config_3_axes_size_audio = 1;
ei_dsp_config_mfe_t ei_dsp_config_3_audio = {
    3, // uint32_t blockId
    3, // int implementationVersion
    1, // int length of axes
    0.02f, // float frame_length
    0.01f, // float frame_stride
    40, // int num_filters
    256, // int fft_length
    300, // int low_frequency
    0, // int high_frequency
    101, // int win_size
    -52 // int noise_floor_db
};

const size_t ei_dsp_blocks_size_audio = 1;
ei_model_dsp_t ei_dsp_blocks_audio[ei_dsp_blocks_size_audio] = {
    { // DSP block 3
        3960,
        &extract_mfe_features,
        (void*)&ei_dsp_config_3_audio,
        ei_dsp_config_3_axes_audio,
        ei_dsp_config_3_axes_size_audio
    }
};

const ei_config_tflite_eon_graph_t ei_config_tflite_graph_audio_0 = {
    .implementation_version = 1,
    .model_init = &trained_model_audio_init,
    .model_invoke = &trained_model_audio_invoke,
    .model_reset = &trained_model_audio_reset,
    .model_input = &trained_model_audio_input,
    .model_output = &trained_model_audio_output,
};

const ei_learning_block_config_tflite_graph_t ei_learning_block_config_audio_0 = {
    .implementation_version = 1,
    .block_id = 0,
    .object_detection = 0,
    .object_detection_last_layer = EI_CLASSIFIER_LAST_LAYER_UNKNOWN,
    .output_data_tensor = 0,
    .output_labels_tensor = 1,
    .output_score_tensor = 2,
    .graph_config = (void*)&ei_config_tflite_graph_audio_0
};

const size_t ei_learning_blocks_size_audio = 1;
const ei_learning_block_t ei_learning_blocks_audio[ei_learning_blocks_size_audio] = {
    {
        &run_nn_inference,
        (void*)&ei_learning_block_config_audio_0,
    },
};

const ei_model_performance_calibration_t ei_calibration_audio = {
    1, /* integer version number */
    false, /* has configured performance calibration */
    (int32_t)(EI_CLASSIFIER_RAW_SAMPLE_COUNT / ((EI_CLASSIFIER_FREQUENCY > 0) ? EI_CLASSIFIER_FREQUENCY : 1)) * 1000, /* Model window */
    0.8f, /* Default threshold */
    (int32_t)(EI_CLASSIFIER_RAW_SAMPLE_COUNT / ((EI_CLASSIFIER_FREQUENCY > 0) ? EI_CLASSIFIER_FREQUENCY : 1)) * 500, /* Half of model window */
    0   /* Don't use flags */
};


const ei_impulse_t impulse_233502_3 = {
    .project_id = 233502,
    .project_owner = "Edge Impulse Inc.",
    .project_name = "Glass breaking - audio classification",
    .deploy_version = 3,

    .nn_input_frame_size = 3960,
    .raw_sample_count = 16000,
    .raw_samples_per_frame = 1,
    .dsp_input_frame_size = 16000 * 1,
    .input_width = 0,
    .input_height = 0,
    .input_frames = 0,
    .interval_ms = 0.0625,
    .frequency = 16000,
    .dsp_blocks_size = ei_dsp_blocks_size_audio,
    .dsp_blocks = ei_dsp_blocks_audio,

    .object_detection = 0,
    .object_detection_count = 0,
    .object_detection_threshold = 0,
    .object_detection_last_layer = EI_CLASSIFIER_LAST_LAYER_UNKNOWN,
    .fomo_output_size = 0,

    .tflite_output_features_count = 2,
    .learning_blocks_size = ei_learning_blocks_size_audio,
    .learning_blocks = ei_learning_blocks_audio,

    .inferencing_engine = EI_CLASSIFIER_TFLITE,

    .quantized = 1,

    .compiled = 1,

    .sensor = EI_CLASSIFIER_SENSOR_MICROPHONE,
    .fusion_string = "audio",
    .slice_size = (16000/4),
    .slices_per_model_window = 4,

    .has_anomaly = 0,
    .label_count = 2,
    .calibration = ei_calibration_audio,
    .categories = ei_classifier_inferencing_categories_audio
};

const ei_impulse_t ei_default_impulse = impulse_233502_3;

#endif // _EI_CLASSIFIER_MODEL_METADATA_H_

Example for the image project:

#ifndef _EI_CLASSIFIER_MODEL_VARIABLES_H_
#define _EI_CLASSIFIER_MODEL_VARIABLES_H_

#include <stdint.h>
#include "model_metadata.h"

#include "tflite-model/trained_model_compiled_image.h"
#include "edge-impulse-sdk/classifier/ei_model_types.h"
#include "edge-impulse-sdk/classifier/inferencing_engines/engines.h"

const char* ei_classifier_inferencing_categories_image[] = { "person", "unknown" };

uint8_t ei_dsp_config_3_axes_image[] = { 0 };
const uint32_t ei_dsp_config_3_axes_size_image = 1;
ei_dsp_config_image_t ei_dsp_config_3_image = {
    3, // uint32_t blockId
    1, // int implementationVersion
    1, // int length of axes
    "RGB" // select channels
};

const size_t ei_dsp_blocks_size_image = 1;
ei_model_dsp_t ei_dsp_blocks_image[ei_dsp_blocks_size_image] = {
    { // DSP block 3
        27648,
        &extract_image_features,
        (void*)&ei_dsp_config_3_image,
        ei_dsp_config_3_axes_image,
        ei_dsp_config_3_axes_size_image
    }
};

const ei_config_tflite_eon_graph_t ei_config_tflite_graph_image_0 = {
    .implementation_version = 1,
    .model_init = &trained_model_image_init,
    .model_invoke = &trained_model_image_invoke,
    .model_reset = &trained_model_image_reset,
    .model_input = &trained_model_image_input,
    .model_output = &trained_model_image_output,
};

const ei_learning_block_config_tflite_graph_t ei_learning_block_config_image_0 = {
    .implementation_version = 1,
    .block_id = 0,
    .object_detection = 0,
    .object_detection_last_layer = EI_CLASSIFIER_LAST_LAYER_UNKNOWN,
    .output_data_tensor = 0,
    .output_labels_tensor = 1,
    .output_score_tensor = 2,
    .graph_config = (void*)&ei_config_tflite_graph_image_0
};

const size_t ei_learning_blocks_size_image = 1;
const ei_learning_block_t ei_learning_blocks_image[ei_learning_blocks_size_image] = {
    {
        &run_nn_inference,
        (void*)&ei_learning_block_config_image_0,
    },
};

const ei_model_performance_calibration_t ei_calibration_image = {
    1, /* integer version number */
    false, /* has configured performance calibration */
    (int32_t)(EI_CLASSIFIER_RAW_SAMPLE_COUNT / ((EI_CLASSIFIER_FREQUENCY > 0) ? EI_CLASSIFIER_FREQUENCY : 1)) * 1000, /* Model window */
    0.8f, /* Default threshold */
    (int32_t)(EI_CLASSIFIER_RAW_SAMPLE_COUNT / ((EI_CLASSIFIER_FREQUENCY > 0) ? EI_CLASSIFIER_FREQUENCY : 1)) * 500, /* Half of model window */
    0   /* Don't use flags */
};


const ei_impulse_t impulse_233515_5 = {
    .project_id = 233515,
    .project_owner = "Edge Impulse Inc.",
    .project_name = "Person vs unknown - image classification",
    .deploy_version = 5,

    .nn_input_frame_size = 27648,
    .raw_sample_count = 9216,
    .raw_samples_per_frame = 1,
    .dsp_input_frame_size = 9216 * 1,
    .input_width = 96,
    .input_height = 96,
    .input_frames = 1,
    .interval_ms = 1,
    .frequency = 0,
    .dsp_blocks_size = ei_dsp_blocks_size_image,
    .dsp_blocks = ei_dsp_blocks_image,

    .object_detection = 0,
    .object_detection_count = 0,
    .object_detection_threshold = 0,
    .object_detection_last_layer = EI_CLASSIFIER_LAST_LAYER_UNKNOWN,
    .fomo_output_size = 0,

    .tflite_output_features_count = 2,
    .learning_blocks_size = ei_learning_blocks_size_image,
    .learning_blocks = ei_learning_blocks_image,

    .inferencing_engine = EI_CLASSIFIER_TFLITE,

    .quantized = 1,

    .compiled = 1,

    .sensor = EI_CLASSIFIER_SENSOR_CAMERA,
    .fusion_string = "image",
    .slice_size = (9216/4),
    .slices_per_model_window = 4,

    .has_anomaly = 0,
    .label_count = 2,
    .calibration = ei_calibration_image,
    .categories = ei_classifier_inferencing_categories_image
};

const ei_impulse_t ei_default_impulse = impulse_233515_5;

#endif // _EI_CLASSIFIER_MODEL_METADATA_H_

Merge the files

Create a new directory (merged-impulse for example). Copy the content of one project into this new directory (audio for example). Copy the content of the tflite-model directory from the other project (image) inside the newly created merged-impulse/tflite-model.

The structure of this new directory should look like the following:

> merged-impulse % tree -L 2
.
├── CMakeLists.txt
├── README.txt
├── edge-impulse-sdk
│   ├── CMSIS
│   ├── LICENSE
│   ├── LICENSE-apache-2.0.txt
│   ├── README.md
│   ├── classifier
│   ├── cmake
│   ├── dsp
│   ├── porting
│   ├── sources.txt
│   ├── tensorflow
│   └── third_party
├── model-parameters
│   ├── model_metadata.h
│   └── model_variables.h
└── tflite-model
    ├── trained_model_compiled_audio.cpp
    ├── trained_model_compiled_audio.h
    ├── trained_model_compiled_image.cpp
    ├── trained_model_compiled_image.h
    ├── trained_model_ops_define_audio.h
    └── trained_model_ops_define_image.h

10 directories, 14 files

Merge the variables and structs in model_variables.h

Copy the necessary variables and structs from previously updated image/model_metadata.h file content to the merged-impulse/model_metadata.h.

To do so, include both of these lines in the #include section:

#include "tflite-model/trained_model_compiled_audio.h"
#include "tflite-model/trained_model_compiled_image.h"

The section that should be copied is from const char* ei_classifier_inferencing_categories... to the line before const ei_impulse_t ei_default_impulse = impulse_<ProjectID>_<version>.

Make sure to leave only one const ei_impulse_t ei_default_impulse = impulse_233502_3; this will define which of your impulse is the default one.

Subtract and merge the trained_model_ops_define.h or tflite_resolver.h

Make sure the macros EI_TFLITE_DISABLE_... are a COMBINATION of the ones present in two deployments.

For EON-compiled projects:

E.g. if #define EI_TFLITE_DISABLE_SOFTMAX_IN_U8 1 is present in one deployment and absent in the other, it should be ABSENT in the combined trained_model_ops_define.h.

For non-EON-Compiled projects:

E.g. if resolver.AddFullyConnected(); is present in one deployment and absent in the other, it should be PRESENT in the combined tflite-resolver.h. Remember to change the length of the resolver array if necessary.

In this example, here are the lines to deleted:

Documentation

Getting started

Developer Plan

Enterprise Plan

Suitable for any type of edge AI application

API Documentation

Community

Public projects

For beginners

Why Edge Impulse, for beginners?

Getting started in a few steps

1. Sign up

2. Create a project

3. Collect/import data

4. Label your data

5. Pre-process your data and train your model

6. Run the inference on a device

7. Go further

Tutorials and resources for beginners

Join the Edge Impulse Community

For ML practitioners

Why Edge Impulse, for ML practitioners?

Getting started in a few steps

Run the inference on a device

Tutorials and resources, for ML practitioners

End-to-end tutorials

Edge Impulse Python SDK tutorials

Other useful resources

Integrations

For embedded engineers

Why Edge Impulse, for embedded engineers?

Getting started in a few steps

1. Sign Up

2. Create a Project

3. Data Collection and Labeling

4. Pre-process your data and train your model

5. Run the inference on a device

6. Go further

Tutorials and resources, for embedded engineers

End-to-end tutorials

Advanced inferencing tutorials

Other useful resources

Join the Edge Impulse Community

Frequently asked questions (FAQ)

Data

Processing

Learning

Deployment

Other

Tutorials

End-to-end tutorials

Movement classification and anomaly detection

Sound classification

Image classification

Object detection

Classification using several sensors

HR/HRV features

Computer vision

Image classification

1. Prerequisites

2. Building a dataset

3. Designing an impulse

Configuring the processing block

Configuring the transfer learning model

4. Validating your model

5. Running the model on your device

Flashing the device

Running the model on the device

Object detection

Bounding Boxes

Centroid

Visual anomaly detection

Visual regression

Audio

Time-series

Environmental (Sensor fusion)

1. Prerequisites

2. Building a dataset

4. Design an Impulse

5. Configure the Flatten block