This section provides detailed end-to-end tutorials to help you get started with Edge Impulse:

Movement classification and anomaly detection

• Continuous motion recognition

Sound classification

• Responding to your voice

• Recognize sounds from audio

Image classification

• Adding sight to your sensors

Object detection

• Detect objects using MobileNet SSD (bounding boxes)

• Object detection using FOMO (centroids)

Classification using several sensors

• Sensor fusion

When you are classifying audio - for example to detect keywords - you want to make sure that every piece of information is both captured and analyzed, to avoid missing events. This means that your device need to capture audio samples and analyze them at the same time. In this tutorial you'll learn how to continuously capture audio data, and then use the continuous inferencing mode in the Edge Impulse SDK to classify the data.

This tutorial assumes that you've completed the tutorial, and have your impulse running on your device.

Continuous Inferencing

In the normal (non-continuous) inference mode when classifying data you sample data until you have a full window of data (e.g. 1 second for a keyword spotting model, see the Create impulse tab in the studio), you then classify this window (using the run_classifier function), and a prediction is returned. Then you empty the buffer, sample new data, and run the inferencing again. Naturally this has some caveats when deploying your model in the real world: 1) you have a delay between windows, as classifying the window takes some time and you're not sampling then, making it possible to miss events. 2) there's no overlap between windows, thus if an event is at the very end of the window, not the full event might be captured - leading to a wrong classification.

To mitigate this we have added several new features to the Edge Impulse SDK.

1. Model slices

Using continuous inferencing, smaller sampling buffers (slices) are used and passed to the inferencing process. In the inferencing process, the buffers are time sequentially placed in a FIFO (First In First Out) buffer that matches the model size. After each iteration, the oldest slice is removed at the end of the buffer and a new slice is inserted at the beginning. On each slice now, the inference is run multiple times (depending on the number of slices used for a model). For example, a 1-second keyword model with 4 slices (each 250 ms), will infer each slice 4 times. So if now the keyword is on 2 edges of the slice buffers, they're glued back together in the FIFO buffer and the keyword will be classified correctly.

2. Averaging

Another advantage of this technique is that it filters out false positives. Take for instance a yes-no keyword spotting model. The word 'yesterday' should not be classified as a yes (or no). But if the 'yes-' is sampled in the first buffer and '-terday' in the next, there is a big chance that the inference step will classify the first buffer as a yes.

By running inference multiple times over the slices, continuous inferencing will filter out this false positive. When the 'yes' buffer enters the FIFO it will surely classify as a 'yes'. But as the rest of the word enters, the classified value for 'yes' will drop quickly. We just have to make sure that we don't react on peak values. Therefore a moving average filter averages the classified output and so flattens the peaks. To have a valid 'yes', we now need multiple high-rated classifications.

Continuous audio sampling

In the standard way of running the impulse, the steps of collecting data and running the inference are run sequentially. First, the audio is sampled, filling a block the size of the model. This block is sent to the inferencing part, where first the features are extracted and then the inference is run. Finally, the classified output is used in your application (by default the output will be printed over the serial connection).

In the continuous sampling method, audio is sampled in parallel with the inferencing and output steps. So while inference is running, audio sampling continues on a background process.

Implementing continuous audio sampling

Prerequisites

The embedded target needs to support running of multiple processes in parallel. This can either be achieved by an operating system; 1 low priority thread will run inferencing and 1 high priority thread will collect sample data. Or the processor should support processor offloading. This is usually done by the audio peripheral or DMA (Direct Memory Access). Here audio samples are collected in a buffer without involvement of the processor.

Double buffering

How do we know when new sample data is available? For this we use a double buffering mechanism. Hereby 2 sample buffers are used:

• 1 buffer for the audio sampling process, filling the buffer with new sample data

• 1 buffer for the inference process, get sample data out the buffer, extract the features and run inference

At start, the sampling process starts filling a buffer with audio samples. Meanwhile, the inference process waits until the buffer is full. When that happens, the sampling process passes the buffer to the inference process and starts sampling on the second buffer. Each iteration, the buffers will be switched so that there is always an empty buffer for sampling and a full buffer of samples for inferencing.

Timing and memory is everything

There are 2 constraints in this story: timing and memory. When switching the buffers there must be a 100% guarantee that the inference process is finished when the sampling process passes a full buffer. If not, the sampling process overruns the buffer and sampled data will get lost. When that happens on the ST B-L475E-IOT01A or the Arduino Nano 33 BLE Sense target, running the impulse is aborted and the following error is returned:

Error sample buffer overrun. Decrease the number of slices per model window (EI_CLASSIFIER_SLICES_PER_MODEL_WINDOW)

The EI_CLASSIFIER_SLICES_PER_MODEL_WINDOW macro is used to set the number of slices to fill the complete model window. The more slices per model, the smaller the slice size, thereby the more inference cycles on the sampled data. Leading to more accurate results. The sampling process uses this macro for the buffer size. Where following rule applies: the bigger the buffer, the longer the sampling cycle. So on targets with lower processing capabilities, we can increase this macro to meet the timing constraint.

Increasing the slice size, increases the volatile memory uses times 2 (since we use double buffering). On a target with limited volatile memory this could be a problem. In this case you want the slice size to be small.

Double buffering in action

On both the ST B-L475E-IOT01A and Arduino Nano 33 BLE Sense targets the audio sampling process calls the audio_buffer_inference_callback() function when there is data. Here the number of samples (inference.n_samples) are stored in one of the buffers. When the buffer is full, the buffers are switched by toggling inference.buf_select. The inference process is signaled by setting the flag inference.buf_ready.

static void audio_buffer_inference_callback(uint32_t n_bytes, uint32_t offset)
{
    for (uint32_t i = 0; i< (n_bytes >>  1); i++) {
        inference.buffers[inference.buf_select][inference.buf_count++] = sampleBuffer[offset + i];

        if (inference.buf_count >= inference.n_samples) {
            inference.buf_select ^= 1;
            inference.buf_count = 0;
            inference.buf_ready = 1;
        }
    }
}

The inferencing process then sets the callback function on the signal_t structure to reference the selected buffer:

int ei_microphone_audio_signal_get_data(size_t offset, size_t length, float *out_ptr)
{
    numpy::int16_to_float(&inference.buffers[inference.buf_select ^ 1][offset], out_ptr, length);
    return 0;
}

Welcome to Edge Impulse! Whether you are a machine learning engineer, MLOps engineer, data scientist, or researcher, we have developed professional tools to help you build and optimize models to run efficiently on any edge device.

In this guide, we'll explore how Edge Impulse empowers you to bring your expertise and your own models to the world of edge AI using either the Edge Impulse Studio, our visual interface, and the Edge Impulse Python SDK, available as a pip package.

Why Edge Impulse, for ML practitioners?

Flexibility: You can choose to work with the tools they are already familiar with and import your models, architecture, and feature processing algorithms into the platform. This means that you can leverage your existing knowledge and workflows seamlessly. Or, for those who prefer an all-in-one solution, Edge Impulse provides enterprise-grade tools for your entire machine-learning pipeline.

Optimized for edge devices: Edge Impulse is designed specifically for deploying machine learning models on edge devices, which are typically resource-constrained, from low-power MCUs up to powerful edge GPUs. We provide tools to optimize your models for edge deployment, ensuring efficient resource usage and peak performance. Focus on developing the best models, we will provide feedback on whether they can run on your hardware target!

Data pipelines: We developed a strong expertise in complex data pipelines (including clinical data) while working with our customers. We support data coming from multiple sources, in any format, and provide tools to perform data alignment and validation checks. All of this in customizable multi-stage pipelines. This means you can build gold-standard labeled datasets that can then be imported into your project to train your models.

Getting started in a few steps

In this getting started guide, we'll walk you through the two different approaches to bringing your expertise to edge devices. Either starting from your dataset or from an existing model.

python -m pip install edgeimpulse

import edgeimpulse as ei
ei.API_KEY = "ei_dae27..."
profile = ei.model.profile(model=model, device='cortex-m4f-80mhz')
print(profile.summary())

ei.model.deploy(model=model, 
                model_output_type=ei.model.output_type.Classification(),
                deploy_target='zip')

Run the inference on a device

If you target MCU-based devices, you can generate ready-to-flash binaries for all the officially supported hardware targets. This method will let you test your model on real hardware very quickly.

In both cases, we will provide profiling information about your models so you can make sure your model will fit your edge device constraints.

Tutorials and resources, for ML practitioners

End-to-end tutorials

Edge Impulse Python SDK tutorials

Other useful resources

Integrations

Welcome to Edge Impulse! We enable professional developers and researchers to create the next generation of intelligent products with Edge AI. In this documentation, you'll find user guides, tutorials, and API documentation. If at any point you have questions, visit our .

If you are a beginner, an advanced embedded engineer, an ML engineer, or a data scientist, you may want to use Edge Impulse differently. We have tailored Edge Impulse to suit your needs. Check out the following getting-started guides for a smooth start:

Enterprise Plan

Professional Plan

For professionals who want additional compute time, more private projects, and more flexibility in usage, we also offer a professional tier version of our platform.

Suitable for any type of edge AI application

API Documentation

Community

Public projects

Think your model is awesome, and want to share it with the world? Go to Dashboard and click Make this project public. This will make your whole project - including all data, machine learning models, and visualizations - available, and can be viewed and cloned by anyone with the URL.

Welcome to Edge Impulse! When we started Edge Impulse, we initially focused on developing a suite of engineering tools designed to empower embedded engineers to harness the power of machine learning on edge devices. As we grew, we also started to develop advanced tools for ML practitioners to ease the collaboration between teams in organizations.

In this getting started guide, we'll walk you through the essential steps to dive into Edge Impulse and leverage it for your embedded projects.

Why Edge Impulse, for embedded engineers?

Embedded systems are becoming increasingly intelligent, and Edge Impulse is here to streamline the integration of machine learning into your hardware projects. Here's why embedded engineers are turning to Edge Impulse:

• Extend hardware capabilities: Edge Impulse can extend hardware capabilities by enabling the integration of machine learning models, allowing edge devices to process complex tasks, recognize patterns, and make intelligent decisions that are complex to develop using rule-based algorithms.

• Open-source export formats: Exported models and libraries contain both digital signal processing code and machine learning models, giving you full explainability of the code.

• Powerful integrations: Edge Impulse provides complete and documented integrations with various hardware platforms, allowing you to focus on the application logic rather than the intricacies of machine learning.

• Support for diverse sensors: Whether you're working with accelerometers, microphones, cameras, or custom sensors, Edge Impulse accommodates a wide range of data sources for your projects.

• Predict on-device performances: Models trained in Edge Impulse run directly on your edge devices, ensuring real-time decision-making with minimal latency. We provide tools to ensure the DSP and models developed with Edge Impulse can fit your device constraints.

• Device-aware optimization: You have full control over model optimization, enabling you to tailor your machine-learning models to the specific requirements and constraints of your embedded systems. Our EON tuner can help you select the best model by training many different variants of models only from an existing dataset and your device constraints!

Getting started in a few steps

Ready to embark on your journey with Edge Impulse? Follow these essential steps to get started:

1. Sign Up

Start by creating your Edge Impulse account. Registration is straightforward, granting you immediate access to the comprehensive suite of tools and resources.

2. Create a Project

Upon logging in, initiate your first project. Select a name that resonates with your project's objectives. If you already which hardware target or system architecture you will be using, you can set it up directly in the dashboard's project info section. This will help you to make sure your model fits your device constraints.

3. Data Collection and Labeling

We offer various methods to collect data from your sensors or to import datasets (see Data acquisition for all methods). For the officially supported hardware targets, we provide binaries or simple steps to attach your device to Edge Impulse Studio and collect data from the Studio. However, as an embedded engineer, you might want to collect data from sensors that are not necessarily available on these devices. To do so, you can use the Data forwarder and print out your sensor values over serial (up to 8kHz) or use our C Ingestion SDK, a portable header-only library (designed to reliably store sampled data from sensors at a high frequency in very little memory).

4. Pre-process your data and train your model

Edge Impulse offers an intuitive model training process through processing blocks and learning blocks. You don't need to write Python code to train your model; the platform guides you through feature extraction, model creation, and training. Customize and fine-tune your blocks for optimal performance on your hardware. Each block will provide on-device performance information showing you the estimated RAM, flash, and latency.

5. Run the inference on a device

This is where the fun start, you can easily export your model as ready-to-flash binaries for all the officially supported hardware targets. This method will let you test your model on real hardware very quickly.

In addition, we also provide a wide variety of export methods to easily integrate your model with your application logic. See C++ library to run your model on any device that supports C++ or our guides for Arduino library, Cube.MX CMSIS-PACK, DRP-AI library, OpenMV library, Ethos-U library, Meta TF model, Simplicity Studio Component, Tensai Flow library, TensorRT library, TIDL-RT library, etc...

6. Go further

Building Edge AI solutions is an iterative process. Feel free to try our organization hub to automate your machine-learning pipelines, collaborate with your colleagues, and create custom blocks.

Tutorials and resources, for embedded engineers

End-to-end tutorials

If you want to get familiar with the full end-to-end flow, please have a look at our end-to-end tutorials on continuous motion recognition, responding to your voice, recognizing sounds from audio, adding sight to your sensors, or object detection.

Advanced inferencing tutorials

In the advanced inferencing tutorials section, you will discover useful techniques to leverage our inferencing libraries or how you can use the inference results in your application logic:

• Continuous audio sampling

• Multi-impulse

• Count objects using FOMO

Other useful resources

• EON compiler

• Inference performance metrics

Join the Edge Impulse Community

Edge Impulse offers a thriving community of embedded engineers, developers, and experts. Connect with like-minded professionals, share your knowledge, and collaborate to enhance your embedded machine-learning projects.

Now that you have a roadmap, it's time to explore Edge Impulse and discover the exciting possibilities of embedded machine learning. Let's get started!

How can I share my Edge Impulse project?

The enterprise version of Edge Impulse offers team collaboration in organizations. Try it out with our enterprise free trial. To collaboration on your projects, go to Dashboard, find the Collaborators section, and click the '+' icon.

You can also create a public version of your Edge Impulse project. This makes your project available to the whole world - including your data, your impulse design, your models, and all intermediate information - and can easily be cloned by anyone in the community. To do so, go to Dashboard, and click Make this project public.

What are the minimum hardware requirements to run the Edge Impulse inferencing library on my embedded device?

The minimum hardware requirements for the embedded device depends on the use case, anything from a Cortex-M0+ for vibration analysis to Cortex-M4F for audio, Cortex-M7 for image classification to Cortex-A for object detection in video, view our inference performance metrics for more details.

What frameworks does Edge Impulse use to train the machine learning models?

We use a wide variety of tools, depending on the machine learning model. For neural networks we typically use TensorFlow and Keras, for object detection models we use TensorFlow with Google's Object Detection API, and for 'classic' non-neural network machine learning algorithms we mainly use sklearn. For neural networks you can see (and modify) the Keras code by clicking ⋮, and selecting Switch to expert mode.

Another big part of Edge Impulse are the processing blocks, as they clean up the data, and already extract important features from your data before passing it to a machine learning model. The source code for these processing blocks can be found on GitHub: edgeimpulse/processing-blocks (and you can build your own processing blocks as well).

What engine does Edge Impulse use to compile the Impulse?

It depends on the hardware.

For general-purpose MCUs we typically use EON Compiler with TFLite Micro kernels (including hardware optimization, e.g. via CMSIS-NN, ESP-NN).

On Linux, if you run the Impulse on CPU, we use TensorFlow Lite.

For accelerators we use a wide variety of other runtimes, e.g. hardcoded network in silicon for Syntiant, custom SNN-based inference engine for Brainchip Akida, DRP-AI for Renesas RZV2L, etc...

Is there a downside to enabling the EON Compiler?

The EON Compiler compiles your neural networks to C++ source code, which then gets compiled into your application. This is great if you need the lowest RAM and ROM possible (EON typically uses 30-50% less memory than TensorFlow Lite) but you also lose some flexibility to update your neural networks in the field - as it is now part of your firmware.

By disabling EON we place the full neural network (architecture and weights) into ROM, and load it on demand. This increases memory usage, but you could just update this section of the ROM (or place the neural network in external flash, or on an SD card) to make it easier to update.

Can I use a model that has been trained elsewhere in Edge Impulse?

Yes you can! Check out our documentation on Bringing your own model (BYOM) into your Edge Impulse project, and using the Edge Impulse Python SDK!

How does the feature explorer visualize data that has more that 3 dimensions?

Edge Impulse uses UMAP (a dimensionality reduction algorithm) to project high dimensionality input data into a 3 dimensional space. This even works for extremely high dimensionality data such as images.

Does Edge impulse integrate with other cloud services?

Yes. The enterprise version of Edge Impulse can integrate directly with your cloud service to access and transform data.

What is the typical power consumption of the Edge Impulse machine learning processes on my device?

Simple answer: To get an indication of time per inference we show performance metrics in every DSP and ML block in the Studio. Multiply this by the active power consumption of your MCU to get an indication of power cost per inference.

More complicated answer: It depends. Normal techniques to conserve power still apply to ML, so try to do as little as possible (do you need to classify every second, or can you do it once a minute?), be smart about when to run inference (can there be an external trigger like a motion sensor before you run inference on a camera?), and collect data in a lower power mode (don't run at full speed when sampling low-resolution data, and see if your sensor can use an interrupt to wake your MCU - rather than polling).

Also see Analyse Power Consumption in Embedded ML Solutions.

What is the .eim model format for Edge Impulse for Linux?

See .eim models? on the Edge Impulse for Linux pages.

How is the labeling of the data performed?

Using the Edge Impulse Studio data acquisition tools (like the serial daemon or data forwarder), you can collect data samples manually with a pre-defined label. If you have a dataset that was collected outside of Edge Impulse, you can upload your dataset using the Edge Impulse CLI, data ingestion API, web uploader, enterprise data storage bucket tools or enterprise upload portals. You can then utilize the Edge Impulse Studio to split up your data into labeled chunks, crop your data samples, and more to create high quality machine learning datasets.

Can I use an unsupported development board or a custom PCB (with a different microcontroller or microprocessor) with Edge Impulse?

Yes! A "supported board" simply means that there is an official or community-supported firmware that has been developed specifically for that board that helps you collect data and run impulses. Edge Impulse is designed to be extensible to computers, smartphones, and a nearly endless array of microcontroller build systems.

You can collect data and upload it to Edge Impulse in a variety of ways. For example:

• Transmitting data to the Data forwarder

• Using the Edge Impulse for Linux SDK

• By uploading files directly (e.g. CBOR, JSON, CSV, WAV, JPG, PNG)

Your trained model can be deployed as part a C++ library. It requires some effort, but most build systems will work with our C++ library, as long as that build system has a C++ compiler and there is enough flash/RAM on your device to run the library (which includes the DSP block and model).

Welcome to Edge Impulse! If you're new to the world of edge machine learning, you've come to the right place. This guide will walk you through the essential steps to get started with Edge Impulse, a suite of engineering tools for building, training, and deploying machine learning models on edge devices.

Check out our Introduction to Edge AI course to learn more about edge computing, machine learning, and edge MLOps.

Why Edge Impulse, for beginners?

Edge Impulse empowers you to bring intelligence to your embedded projects by enabling devices to understand and respond to their environment. Whether you want to recognize sounds, identify objects, or detect motion, Edge Impulse makes it accessible and straightforward. Here's why beginners like you are diving into Edge Impulse:

• No Coding Required: You don't need to be a coding expert to use Edge Impulse. Our platform provides a user-friendly interface that guides you through the process - this includes many optimized preprocessing and learning blocks, various neural network architectures, and pre-trained models and can generate ready-to-flash binaries to test your models on real devices.

• Edge Computing: Your machine learning models are optimized to run directly on your edge devices, ensuring low latency and real-time processing.

• Support for Various Sensors: Edge Impulse supports a wide range of sensors, from accelerometers and microphones to cameras, making it versatile for different projects.

• Community and Resources: You're not alone on this journey. Edge Impulse offers a supportive community and extensive documentation to help you succeed.

Getting started in a few steps

Ready to begin? Follow these simple steps to embark on your Edge Impulse journey:

1. Sign up

Start by creating an Edge Impulse account. It's free to get started, and you'll gain access to all the tools and resources you need.

2. Create a project

Once you're logged in, create your first project. Give it a name that reflects your project's goal, whether it's recognizing sounds, detecting objects, or something entirely unique.

3. Collect/import data

To teach your device, you need data. Edge Impulse provides user-friendly tools for collecting data from your sensors, such as recording audio, capturing images, or reading sensor values. We recommend using a hardware target from this list or your smartphone to start collecting data when you begin with Edge Impulse.

You can also import existing datasets or clone a public project to get familiar with the platform.

4. Label your data

Organize your data by labeling it. For example, if you're working on sound recognition, label audio clips with descriptions like "dog barking" or "car horn." You can label your data as you collect it or add labels later, our data explorer is also particularly useful to understand your data.

5. Pre-process your data and train your model

This is where the magic happens. Edge Impulse offers an intuitive model training process through processing blocks and learning blocks. You don't need to write complex code; the platform guides you through feature extraction, model creation, and training.

6. Run the inference on a device

After training your model, you can easily export your model to run in a web browser or on your smartphone, but you can also run it on a wide variety of edge devices, whether it's a Raspberry Pi, Arduino, or other compatible hardware. We also provide ready-to-flash binaries for all the officially supported hardware targets. You don't even need to write embedded code to test your model on real devices!

If you have a device that is not supported, no problem, you can export your model as a C++ library that runs on any embedded device. See Running your impulse locally for more information.

7. Go further

Building Edge AI solutions is an iterative process. Feel free to try our organization hub to automate your machine-learning pipelines, collaborate with your colleagues, and create custom blocks.

Tutorials and resources for beginners

The end-to-end tutorials are perfect for learning how to use Edge Impulse Studio. Try the tutorials:

• continuous motion recognition,

• responding to your voice,

• recognizing sounds from audio,

• adding sight to your sensors,

• detecting objects' location and size (using bounding boxes),

• detection objects' location (using centroids)

These will let you build machine-learning models that detect things in your home or office.

Join the Edge Impulse Community

Remember, you're not alone on your journey. Join the Edge Impulse community to connect with other beginners, experts, and enthusiasts. Share your experiences, ask questions, and learn from others who are passionate about embedded machine learning.

Now that you have a roadmap, it's time to explore Edge Impulse and discover the exciting possibilities of embedded machine learning. Let's get started!

In this tutorial, you'll use machine learning to build a gesture recognition system that runs on a microcontroller. This is a hard task to solve using rule-based programming, as people don't perform gestures in the exact same way every time. But machine learning can handle these variations with ease. You'll learn how to collect high-frequency data from real sensors, use signal processing to clean up data, build a neural network classifier, and how to deploy your model back to a device. At the end of this tutorial, you'll have a firm understanding of applying machine learning in embedded devices using Edge Impulse.

There is also a video version of this tutorial:

You can view the finished project, including all data, signal processing and machine learning blocks here: Tutorial: continuous motion recognition.

1. Prerequisites

For this tutorial, you'll need a supported device.

Alternatively, use the either Data forwarder or Edge Impulse for Linux SDK to collect data from any other development board, or your mobile phone.

If your device is connected (green dot) under Devices in the studio you can proceed:

2. Collecting your first data

With your device connected, we can collect some data. In the studio go to the Data acquisition tab. This is the place where all your raw data is stored, and - if your device is connected to the remote management API - where you can start sampling new data.

Under Record new data, select your device, set the label to updown, the sample length to 10000, the sensor to Built-in accelerometer and the frequency to 62.5Hz. This indicates that you want to record data for 10 seconds, and label the recorded data as updown. You can later edit these labels if needed.

After you click Start sampling move your device up and down in a continuous motion. In about twelve seconds the device should complete sampling and upload the file back to Edge Impulse. You see a new line appear under 'Collected data' in the studio. When you click it you now see the raw data graphed out. As the accelerometer on the development board has three axes you'll notice three different lines, one for each axis.

Machine learning works best with lots of data, so a single sample won't cut it. Now is the time to start building your own dataset. For example, use the following four classes, and record around 3 minutes of data per class:

• Idle - just sitting on your desk while you're working.

• Snake - moving the device over your desk as a snake.

• Wave - waving the device from left to right.

• Updown - moving the device up and down.

Prebuilt dataset

Alternatively, you can load an example test set that has about ten minutes of data in these classes (but how much fun is that?). See the Continuous gestures dataset for more information.

3. Designing an impulse

With the training set in place, you can design an impulse. An impulse takes the raw data, slices it up in smaller windows, uses signal processing blocks to extract features, and then uses a learning block to classify new data. Signal processing blocks always return the same values for the same input and are used to make raw data easier to process, while learning blocks learn from past experiences.

For this tutorial we'll use the 'Spectral analysis' signal processing block. This block applies a filter, performs spectral analysis on the signal, and extracts frequency and spectral power data. Then we'll use a 'Neural Network' learning block, that takes these spectral features and learns to distinguish between the four (idle, snake, wave, updown) classes.

In the studio go to Create impulse, set the window size to 2000 (you can click on the 2000 ms. text to enter an exact value), the window increase to 80, and add the 'Spectral Analysis' and 'Classification (Keras)' blocks. Then click Save impulse.

Configuring the spectral analysis block

To configure your signal processing block, click Spectral features in the menu on the left. This will show you the raw data on top of the screen (you can select other files via the drop down menu), and the results of the signal processing through graphs on the right. For the spectral features block you'll see the following graphs:

• Filter response - If you have chosen a filter (with non zero order), this will show you the response across frequencies. That is, it will show you how much each frequency will be attenuated.

• After filter - the signal after applying the filter. This will remove noise.

• Spectral power - the frequencies at which the signal is repeating (e.g. making one wave movement per second will show a peak at 1 Hz).

A good signal processing block will yield similar results for similar data. If you move the sliding window (on the raw data graph) around, the graphs should remain similar. Also, when you switch to another file with the same label, you should see similar graphs, even if the orientation of the device was different.

Set the filter to low pass with the following parameters:

Once you're happy with the result, click Save parameters. This will send you to the 'Feature generation' screen. In here you'll:

1. Split all raw data up in windows (based on the window size and the window increase).

2. Apply the spectral features block on all these windows.

3. Calculate feature importance. We will use this later to set up the anomaly detection.

Click Generate features to start the process.

Afterward the 'Feature explorer' will load. This is a plot of all the extracted features against all the generated windows. You can use this graph to compare your complete data set. A good rule of thumb is that if you can visually identify some clusters by classes, then the machine learning model will be able to do so as well.

Configuring the neural network

With all data processed it's time to start training a neural network. Neural networks are a set of algorithms, modeled loosely after the human brain, that are designed to recognize patterns. The network that we're training here will take the signal processing data as an input, and try to map this to one of the four classes.

So how does a neural network know what to predict? A neural network consists of layers of neurons, all interconnected, and each connection has a weight. One such neuron in the input layer would be the height of the first peak of the X-axis (from the signal processing block); and one such neuron in the output layer would be wave (one the classes). When defining the neural network all these connections are initialized randomly, and thus the neural network will make random predictions. During training, we then take all the raw data, ask the network to make a prediction, and then make tiny alterations to the weights depending on the outcome (this is why labeling raw data is important).

This way, after a lot of iterations, the neural network learns; and will eventually become much better at predicting new data. Let's try this out by clicking on NN Classifier in the menu.

Set 'Number of training cycles' to 1. This will limit training to a single iteration. And then click Start training.

Now change 'Number of training cycles' to 2 and you'll see performance go up. Finally, change 'Number of training cycles' to 30 and let the training finish.

You've just trained your first neural networks!

4. Classifying new data

From the statistics in the previous step we know that the model works against our training data, but how well would the network perform on new data? Click on Live classification in the menu to find out. Your device should (just like in step 2) show as online under 'Classify new data'. Set the 'Sample length' to 10000 (10 seconds), click Start sampling and start doing movements. Afterward, you'll get a full report on what the network thought you did.

If the network performed great, fantastic! But what if it performed poorly? There could be a variety of reasons, but the most common ones are:

1. There is not enough data. Neural networks need to learn patterns in data sets, and the more data the better.

2. The data does not look like other data the network has seen before. This is common when someone uses the device in a way that you didn't add to the test set. You can add the current file to the test set by clicking ⋮, then selecting Move to training set. Make sure to update the label under 'Data acquisition' before training.

3. The model has not been trained enough. Up the number of epochs to 200 and see if performance increases (the classified file is stored, and you can load it through 'Classify existing validation sample').

4. The model is overfitting and thus performs poorly on new data. Try reducing the learning rate or add more data.

5. The neural network architecture is not a great fit for your data. Play with the number of layers and neurons and see if performance improves.

As you see there is still a lot of trial and error when building neural networks, but we hope the visualizations help a lot. You can also run the network against the complete validation set through 'Model validation'. Think of the model validation page as a set of unit tests for your model!

With a working model in place, we can look at places where our current impulse performs poorly.

5. Anomaly detection

Neural networks are great, but they have one big flaw. They're terrible at dealing with data they have never seen before (like a new gesture). Neural networks cannot judge this, as they are only aware of the training data. If you give it something unlike anything it has seen before it'll still classify as one of the four classes.

Let's look at how this works in practice. Go to 'Live classification' and record some new data, but now vividly shake your device. Take a look and see how the network will predict something regardless.

So, how can we do better? If you look at the feature explorer, you should be able to visually separate the classified data from the training data. We can use this to our advantage by training a new (second) network that creates clusters around data that we have seen before, and compares incoming data against these clusters. If the distance from a cluster is too large you can flag the sample as an anomaly, and not trust the neural network.

To add this block go to Create impulse, click Add learning block, and select 'Anomaly Detection (K-Means)'. Then click Save impulse.

To configure the clustering model click on Anomaly detection in the menu. Here we need to specify:

• The number of clusters. Here use 32.

• The axes that we want to select during clustering. Click on the Select suggested axes button to harness the results of the feature importance output. Alternatively, the data separates well on the accX RMS, accY RMS and accZ RMS axes, you can also include these axes.

Click Start training to generate the clusters. You can load existing validation samples into the anomaly explorer with the dropdown menu.

If you now go back to 'Live classification' and load your last sample, it should now have tagged everything as anomaly. This is a great example where signal processing (to extract features), neural networks (for classification) and clustering algorithms (for anomaly detection) can work together.

6. Deploying back to device

With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the signal processing code, neural network weights, and classification code - up in a single C++ library that you can include in your embedded software.

To export your model, click on Deployment in the menu. Then under 'Build firmware' select your development board, and click Build. This will export the impulse, and build a binary that will run on your development board in a single step. After building is completed you'll get prompted to download a binary. Save this on your computer.

Flashing the device

When you click the Build button, you'll see a pop-up with text and video instructions on how to deploy the binary to your particular device. Follow these instructions. Once you are done, we are ready to test your impulse out.

Running the model on the device

We can connect to the board's newly flashed firmware over serial. Open a terminal and run:

$ edge-impulse-run-impulse

This will sample data from the sensor, run the signal processing code, and then classify the data:

Starting inferencing in 2 seconds...
Sampling... Storing in file name: /fs/device-classification261
Tensor shape: 4
Predictions (DSP: 17 ms., Classification: 1 ms., Anomaly: 0 ms.):
    idle: 0.00004
    snake: 0.00012
    updown: 0.00009
    wave: 0.99976
    anomaly score: 0.032
Finished inferencing, raw data is stored in '/fs/device-classification261'. Use AT+UPLOADFILE to send back to Edge Impulse.

$ edge-impulse-run-impulse --continuous

Victory! You've now built your first on-device machine learning model.

7. Conclusion

Congratulations! You have used Edge Impulse to train a machine learning model capable of recognizing your gestures and understand how you can build models that classify sensor data or find anomalies. Now that you've trained your model you can integrate your impulse in the firmware of your own embedded device, see Running your impulse locally. There are examples for Mbed OS, Arduino, STM32CubeIDE, and any other target that supports a C++ compiler.

Or if you're interested in more, see our tutorials on Recognize sounds from audio or Adding sight to your sensors. If you have a great idea for a different project, that's fine too. Edge Impulse lets you capture data from any sensor, build custom processing blocks to extract features, and you have full flexibility in your Machine Learning pipeline with the learning blocks.

We can't wait to see what you'll build! 🚀

In this tutorial, you'll use machine learning to build a system that can recognize audible events, particularly your voice through audio classification. The system you create will work similarly to "Hey Siri" or "OK, Google" and is able to recognize keywords or other audible events, even in the presence of other background noise or background chatter.

You'll learn how to collect audio data from microphones, use signal processing to extract the most important information, and train a deep neural network that can tell you whether your keyword was heard in a given clip of audio. Finally, you'll deploy the system to an embedded device and evaluate how well it works.

At the end of this tutorial, you'll have a firm understanding of how to classify audio using Edge Impulse.

There is also a video version of this tutorial:

You can view the finished project, including all data, signal processing and machine learning blocks here: Tutorial: responding to your voice.

1. Prerequisites

For this tutorial, you'll need a supported device.

If your device is connected under Devices in the studio you can proceed:

2. Collecting your first data

In this tutorial we want to build a system that recognizes keywords, so your first job is to think of a great one. It can be your name, an action, or even a growl - it's your party. Do keep in mind that some keywords are harder to distinguish from others, and especially keywords with only one syllable (like 'One') might lead to false-positives (e.g. when you say 'Gone'). This is the reason that Apple, Google and Amazon all use at least three-syllable keywords ('Hey Siri', 'OK, Google', 'Alexa'). A good one would be "Hello world".

To collect your first data, go to Data acquisition, set your keyword as the label, set your sample length to 10s., your sensor to 'microphone' and your frequency to 16KHz. Then click Start sampling and start saying your keyword over and over again (with some pause in between).

Note: Data collection from a development board might be slow, you can use your Mobile phone as a sensor to make this much faster.

Afterwards you have a file like this, clearly showing your keywords, separated by some noise.

This data is not suitable for Machine Learning yet though. You will need to cut out the parts where you say your keyword. This is important because you only want the actual keyword to be labeled as such, and not accidentally label noise, or incomplete sentences (e.g. only "Hello"). Fortunately the Edge Impulse Studio can do this for you. Click ⋮ next to your sample, and select Split sample.

If you have a short keyword, enable Shift samples to randomly shift the sample around in the window, and then click Split. You now have individual 1s. long samples in your dataset. Perfect!

3. Building your dataset

Now that you know how to collect data we can consider other data we need to collect. In addition to your keyword we'll also need audio that is not your keyword. Like background noise, the TV playing ('noise' class), and humans saying other words ('unknown' class). This is required because a machine learning model has no idea about right and wrong (unless those are your keywords), but only learns from the data you feed into it. The more varied your data is, the better your model will work.

For each of these three classes ('your keyword', 'noise', and 'unknown') you want to capture an even amount of data (balanced datasets work better) - and for a decent keyword spotting model you'll want at least 10 minutes in each class (but, the more the better).

Thus, collect 10 minutes of samples for your keyword - do this in the same manner as above. The fastest way is probably through your mobile phone, collecting 1 minute clips, then automatically splitting this data. Make sure to capture wide variations of the keyword: leverage your family and your colleagues to help you collect the data, make sure you cover high and low pitches, and slow and fast speakers.

For the noise and unknown datasets you can either collect this yourself, or make your life a bit easier by using dataset of both 'noise' (all kinds of background noise) and 'unknown' (random words) data that we built for you here: Pre-built datasets > Keyword spotting.

To import this data, go to Data acquisition, click the Upload icon, and select a number of 'noise' or 'unknown' samples (there's 25 minutes of each class, but you can select less files if you want), and clicking Begin upload. The data is automatically labeled and added to your project.

Rebalancing your dataset

If you've collected all your training data through the 'Record new data' widget you'll have all your keywords in the 'Training' dataset. This is not great, because you want to keep 20% of your data separate to validate the machine learning model. To mitigate this you can go to Dashboard and select Perform train/test split. This will automatically split your data between a training class (80%) and a testing class (20%). Afterwards you should see something like this:

4. Designing your impulse

With the data set in place you can design an impulse. An impulse takes the raw data, slices it up in smaller windows, uses signal processing blocks to extract features, and then uses a learning block to classify new data. Signal processing blocks always return the same values for the same input and are used to make raw data easier to process, while learning blocks learn from past experiences.

For this tutorial we'll use the "MFCC" signal processing block. MFCC stands for Mel Frequency Cepstral Coefficients. This sounds scary, but it's basically just a way of turning raw audio—which contains a large amount of redundant information—into simplified form. Edge Impulse has many other processing blocks for audio, including "MFE" and the "Spectrogram" blocks for non-voice audio, but the "MFCC" block is great for dealing with human speech.

We'll then pass this simplified audio data into a Neural Network block, which will learn to distinguish between the three classes of audio.

In the Studio, go to the Create impulse tab, add a Time series data, an Audio (MFCC) and a Classification (Keras) block. Leave the window size to 1 second (as that's the length of our audio samples in the dataset) and click Save Impulse.

5. Configure the MFCC block

Now that we've assembled the building blocks of our Impulse, we can configure each individual part. Click on the MFCC tab in the left hand navigation menu. You'll see a page that looks like this:

This page allows you to configure the MFCC block, and lets you preview how the data will be transformed. The right of the page shows a visualization of the MFCC's output for a piece of audio, which is known as a spectrogram. An MFCC spectrogram is a specially tuned spectrogram which highlights frequencies which are common in human speech (Edge Impulse also has normal spectrograms if that's more your thing).

In the spectrogram the vertical axis represents the frequencies (the number of frequency bands is controlled by 'Number of coefficients' parameter, try it out!), and the horizontal axis represents time (controlled by 'frame stride' and 'frame length'). The patterns visible in a spectrogram contain information about what type of sound it represents. For example, the spectrogram in this image shows "Hello world":

And the spectrogram in this image shows "On":

These differences are not necessarily easy for a person to describe, but fortunately they are enough for a neural network to learn to identify.

It's interesting to explore your data and look at the types of spectrograms it results in. You can use the dropdown box near the top right of the page to choose between different audio samples to visualize, or play with the parameters to see how the spectrogram changes.

In addition, you can see the performance of the MFCC block on your microcontroller below the spectrogram. This is the complete time that it takes on a low-power microcontroller (Cortex-M4F @ 80MHz) to analyze 1 second of data.

You might think based on this number that we can only classify 2 or 3 windows per second, but we continuously build up the spectrogram (as it has a time component), which takes less time, and we can thus continuously listen for events 5-6x a second, even on an 40MHz processor. This is already implemented on all fully supported development boards, and easy to implement on your own device.

Feature explorer

The spectrograms generated by the MFCC block will be passed into a neural network architecture that is particularly good at learning to recognize patterns in this type of tabular data. Before training our neural network, we'll need to generate MFCC blocks for all of our windows of audio. To do this, click the Generate features button at the top of the page, then click the green Generate features button. This will take a minute or so to complete.

Afterwards you're presented with one of the most useful features in Edge Impulse: the feature explorer. This is a 3D representation showing your complete dataset, with each data-item color-coded to its respective label. You can zoom in to every item, find anomalies (an item that's in a wrong cluster), and click on items to listen to the sample. This is a great way to check whether your dataset contains wrong items, and to validate whether your dataset is suitable for ML (it should separate nicely).

6. Configure the neural network

With all data processed it's time to start training a neural network. Neural networks are algorithms, modeled loosely after the human brain, that can learn to recognize patterns that appear in their training data. The network that we're training here will take the MFCC as an input, and try to map this to one of three classes—your keyword, noise or unknown.

Click on NN Classifier in the left hand menu. You'll see the following page:

A neural network is composed of layers of virtual "neurons", which you can see represented on the left hand side of the NN Classifier page. An input—in our case, an MFCC spectrogram—is fed into the first layer of neurons, which filters and transforms it based on each neuron's unique internal state. The first layer's output is then fed into the second layer, and so on, gradually transforming the original input into something radically different. In this case, the spectrogram input is transformed over four intermediate layers into just two numbers: the probability that the input represents your keyword, and the probability that the input represents 'noise' or 'unknown'.

During training, the internal state of the neurons is gradually tweaked and refined so that the network transforms its input in just the right ways to produce the correct output. This is done by feeding in a sample of training data, checking how far the network's output is from the correct answer, and adjusting the neurons' internal state to make it more likely that a correct answer is produced next time. When done thousands of times, this results in a trained network.

A particular arrangement of layers is referred to as an architecture, and different architectures are useful for different tasks. The default neural network architecture provided by Edge Impulse will work well for our current project, but you can also define your own architectures. You can even import custom neural network code from tools used by data scientists, such as TensorFlow and Keras (click the three dots at the top of the page).

Before you begin training, you should change some values in the configuration. Change the Minimum confidence rating to 0.6. This means that when the neural network makes a prediction (for example, that there is 0.8 probability that some audio contains "hello world") Edge Impulse will disregard it unless it is above the threshold of 0.6.

Next, enable 'Data augmentation'. When enabled your data is randomly mutated during training. For example, by adding noise, masking time or frequency bands, or warping your time axis. This is a very quick way to make your dataset work better in real life (with unpredictable sounds coming in), and prevents your neural network from overfitting (as the data samples are changed every training cycle).

With everything in place, click Start training. You'll see a lot of text flying past in the Training output panel, which you can ignore for now. Training will take a few minutes. When it's complete, you'll see the Last training performance panel appear at the bottom of the page:

Congratulations, you've trained a neural network with Edge Impulse! But what do all these numbers mean?

At the start of training, 20% of the training data is set aside for validation. This means that instead of being used to train the model, it is used to evaluate how the model is performing. The Last training performance panel displays the results of this validation, providing some vital information about your model and how well it is working. Bear in mind that your exact numbers may differ from the ones in this tutorial.

On the left hand side of the panel, Accuracy refers to the percentage of windows of audio that were correctly classified. The higher number the better, although an accuracy approaching 100% is unlikely, and is often a sign that your model has overfit the training data. You will find out whether this is true in the next stage, during model testing. For many applications, an accuracy above 85% can be considered very good.

The Confusion matrix is a table showing the balance of correctly versus incorrectly classified windows. To understand it, compare the values in each row. For example, in the above screenshot, 96 of the helloworld audio windows were classified as helloworld, while 10 of them were incorrectly classified as unknown or noise. This appears to be a great result.

The On-device performance region shows statistics about how the model is likely to run on-device. Inferencing time is an estimate of how long the model will take to analyze one second of data on a typical microcontroller (an Arm Cortex-M4F running at 80MHz). Peak RAM usage gives an idea of how much RAM will be required to run the model on-device.

7. Classifying new data

The performance numbers in the previous step show that our model is working well on its training data, but it's extremely important that we test the model on new, unseen data before deploying it in the real world. This will help us ensure the model has not learned to overfit the training data, which is a common occurrence.

Fortunately we've put aside 20% of our data already in the 'Test set' (see Data acquisition). This is data that the model has never seen before, and we can use this to validate whether our model actually works on unseen data. To run your model against the test set, head to Model testing, select all items and click Classify selected.

To drill down into a misclassified sample, click the three dots (⋮) next to a sample and select Show classification. You're then transported to the classification view, which lets you inspect the sample, and compare the sample to your training data. This way you can inspect whether this was actually a classification failure, or whether your data was incorrectly labeled. From here you can either update the label (when the label was wrong), or move the item to the training set to refine your model.

8. Deploying to your device

With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the MFCC algorithm, neural network weights, and classification code - in a single C++ library that you can include in your embedded software.

To export your model, click on Deployment in the menu. Then under 'Build firmware' select your development board, and click Build. This will export the impulse, and build a binary that will run on your development board in a single step. After building is completed you'll get prompted to download a binary. Save this on your computer.

Flashing the device

When you click the Build button, you'll see a pop-up with text and video instructions on how to deploy the binary to your particular device. Follow these instructions. Once you are done, we are ready to test your impulse out.

Running the model on the device

We can connect to the board's newly flashed firmware over serial. Open a terminal and run:

$ edge-impulse-run-impulse --continuous

This will capture audio from the microphone, run the MFCC code, and then classify the spectrogram:

Edge Impulse impulse runner v1.9.1
[SER] Connecting to /dev/tty.usbmodem0004401658161
Predictions (DSP: 143 ms., Classification: 13 ms., Anomaly: 0 ms.):
    helloworld: 	0.98828
    noise: 	        0.0039
    unknown: 	    0.00781

Great work! You've captured data, trained a model, and deployed it to an embedded device. You can now control LEDs, activate actuators, or send a message to the cloud whenever you say a keyword!

Poor performance due to unbalanced dataset?

Is your model working properly in the Studio, but does not recognize your keyword when running in continuous mode on your device? Then this is probably due to dataset imbalance (a lot more unknown / noise data compared to your keyword) in combination with our moving average code to reduce false positives.

When running in continuous mode we run a moving average over the predictions to prevent false positives. E.g. if we do 3 classifications per second you’ll see your keyword potentially classified three times (once at the start of the audio file, once in the middle, once at the end). However, if your dataset is unbalanced (there’s a lot more noise / unknown than in your dataset) the ML model typically manages to only find your keyword in the 'center' window, and thus we filter it out as a false positive.

You can fix this by either:

1. Adding more data :-)

2. Or, by disabling the moving average filter by going into ei_run_classifier.h (in the edge-impulse-sdk directory) and removing:

   Copy

       for (size_t ix = 0; ix < EI_CLASSIFIER_LABEL_COUNT; ix++) {
           result->classification[ix].value =
               run_moving_average_filter(&classifier_maf[ix], result->classification[ix].value);
       }

Note that this might increase the number of false positives the model detects.

9. Conclusion

Congratulations! you've used Edge Impulse to train a neural network model capable of recognizing audible events. There are endless applications for this type of model, from monitoring industrial machinery to recognizing voice commands. Now that you've trained your model you can integrate your impulse in the firmware of your own embedded device, see Running your impulse locally. There are examples for Mbed OS, Arduino, STM32CubeIDE, Zephyr, and any other target that supports a C++ compiler.

Or if you're interested in more, see our tutorials on Continuous motion recognition or Adding sight to your sensors. If you have a great idea for a different project, that's fine too. Edge Impulse lets you capture data from any sensor, build custom processing blocks to extract features, and you have full flexibility in your Machine Learning pipeline with the learning blocks.

We can't wait to see what you'll build! 🚀

In this tutorial, you'll use machine learning to build a system that can recognize objects in your house through a camera - a task known as image classification - connected to a microcontroller. Adding sight to your embedded devices can make them see the difference between poachers and elephants, do quality control on factory lines, or let your RC cars drive themselves. In this tutorial you'll learn how to collect images for a well-balanced dataset, how to apply transfer learning to train a neural network, and deploy the system to an embedded device.

At the end of this tutorial, you'll have a firm understanding of how to classify images using Edge Impulse.

There is also a video version of this tutorial:

You can view the finished project, including all data, signal processing and machine learning blocks here: Tutorial: adding sight to your sensors.

1. Prerequisites

For this tutorial, you'll need a supported device.

If you don't have any of these devices, you can also upload an existing dataset through the Uploader. After this tutorial you can then deploy your trained machine learning model as a C++ library and run it on your device.

2. Building a dataset

In this tutorial we'll build a model that can distinguish between two objects in your house - we've used a plant and a lamp, but feel free to pick two other objects. To make your machine learning model see it's important that you capture a lot of example images of these objects. When training the model these example images are used to let the model distinguish between them. Because there are (hopefully) a lot more objects in your house than just lamps or plants, you also need to capture images that are neither a lamp or a plant to make the model work well.

Capture the following amount of data - make sure you capture a wide variety of angles and zoom levels:

• 50 images of a lamp.

• 50 images of a plant.

• 50 images of neither a plant nor a lamp - make sure to capture a wide variation of random objects in the same room as your lamp or plant.

You can collect data from the following devices:

• Collecting image data with the OpenMV Cam H7 Plus

• Collecting image data from the Studio - for all other officially supported boards with camera sensors.

• Collecting image data with your mobile phone

Or you can capture your images using another camera, and then upload them by going to Data acquisition and clicking the 'Upload' icon.

Afterwards you should have a well-balanced dataset listed under Data acquisition in your Edge Impulse project. You can switch between your training and testing data with the two buttons above the 'Data collected' widget.

3. Designing an impulse

With the training set in place you can design an impulse. An impulse takes the raw data, adjusts the image size, uses a preprocessing block to manipulate the image, and then uses a learning block to classify new data. Preprocessing blocks always return the same values for the same input (e.g. convert a color image into a grayscale one), while learning blocks learn from past experiences.

For this tutorial we'll use the 'Images' preprocessing block. This block takes in the color image, optionally makes the image grayscale, and then turns the data into a features array. If you want to do more interesting preprocessing steps - like finding faces in a photo before feeding the image into the network -, see the Building custom processing blocks tutorial. Then we'll use a 'Transfer Learning' learning block, which takes all the images in and learns to distinguish between the three ('plant', 'lamp', 'unknown') classes.

In the studio go to Create impulse, set the image width and image height to 96, and add the 'Images' and 'Transfer Learning (Images)' blocks. Then click Save impulse.

Configuring the processing block

To configure your processing block, click Images in the menu on the left. This will show you the raw data on top of the screen (you can select other files via the drop down menu), and the results of the processing step on the right. You can use the options to switch between 'RGB' and 'Grayscale' mode, but for now leave the color depth on 'RGB' and click Save parameters.

This will send you to the 'Feature generation' screen. In here you'll:

• Resize all the data.

• Apply the processing block on all this data.

• Create a 3D visualization of your complete dataset.

Click Generate features to start the process.

Afterwards the 'Feature explorer' will load. This is a plot of all the data in your dataset. Because images have a lot of dimensions (here: 96x96x3=27,648 features) we run a process called 'dimensionality reduction' on the dataset before visualizing this. Here the 27,648 features are compressed down to just 3, and then clustered based on similarity. Even though we have little data you can already see some clusters forming (lamp images are all on the right), and can click on the dots to see which image belongs to which dot.

Configuring the transfer learning model

With all data processed it's time to start training a neural network. Neural networks are a set of algorithms, modeled loosely after the human brain, that are designed to recognize patterns. The network that we're training here will take the image data as an input, and try to map this to one of the three classes.

It's very hard to build a good working computer vision model from scratch, as you need a wide variety of input data to make the model generalize well, and training such models can take days on a GPU. To make this easier and faster we are using transfer learning. This lets you piggyback on a well-trained model, only retraining the upper layers of a neural network, leading to much more reliable models that train in a fraction of the time and work with substantially smaller datasets.

To configure the transfer learning model, click Transfer learning in the menu on the left. Here you can select the base model (the one selected by default will work, but you can change this based on your size requirements), optionally enable data augmentation (images are randomly manipulated to make the model perform better in the real world), and the rate at which the network learns.

Set:

• Number of training cycles to 20.

• Learning rate to 0.0005.

• Data augmentation: enabled.

• Minimum confidence rating: 0.7.

Important: If you're using a development board with less memory, like the Arduino Nano 33 BLE Sense click Choose a different model and select MobileNetV1 96x96 0.25. This is a smaller transfer learning model.

And click Start training. After the model is done you'll see accuracy numbers, a confusion matrix and some predicted on-device performance on the bottom. You have now trained your model!

4. Validating your model

With the model trained let's try it out on some test data. When collecting the data we split the data up between a training and a testing dataset. The model was trained only on the training data, and thus we can use the data in the testing dataset to validate how well the model will work in the real world. This will help us ensure the model has not learned to overfit the training data, which is a common occurrence.

To validate your model, go to Model testing, select the checkbox next to 'Sample name' and click Classify selected. Here we hit 89% accuracy, which is great for a model with so little data.

To see a classification in detail, click the three dots next to an item, and select Show classification. This brings you to the Live classification screen with much more details on the file (if you collected data with your mobile phone you can also capture new testing data directly from here). This screen can help you determine why items were misclassified.

5. Running the model on your device

With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the preprocessing steps, neural network weights, and classification code - in a single C++ library that you can include in your embedded software.

To run your impulse on either the OpenMV camera or your phone, follow these steps:

• OpenMV Cam H7 Plus: Running your impulse on your OpenMV camera

• Mobile phone: just click Switch to classification mode at the bottom of your phone screen.

For other boards: click on Deployment in the menu. Then under 'Build firmware' select your development board, and click Build. This will export the impulse, and build a binary that will run on your development board in a single step. After building is completed you'll get prompted to download a binary. Save this on your computer.

Flashing the device

When you click the Build button, you'll see a pop-up with text and video instructions on how to deploy the binary to your particular device. Follow these instructions. Once you are done, we are ready to test your impulse out.

Running the model on the device

We can connect to the board's newly flashed firmware over serial. Open a terminal and run:

$ edge-impulse-run-impulse

To also see a preview of the camera, run:

$ edge-impulse-run-impulse --debug

To run continuous (without a pause every 2 seconds), but without the preview, run:

$ edge-impulse-run-impulse --continuous

Congratulations! You've added sight to your sensors. Now that you've trained your model you can integrate your impulse in the firmware of your own embedded device, see Running your impulse locally. There are examples for Mbed OS, Arduino, STM32CubeIDE, and any other target that supports a C++ compiler. Note that the model we trained in this tutorial is relatively big, but you can choose a smaller transfer learning model.

Or if you're interested in more, see our tutorials on Continuous motion recognition or Recognize sounds from audio. If you have a great idea for a different project, that's fine too. Edge Impulse lets you capture data from any sensor, build custom processing blocks to extract features, and you have full flexibility in your Machine Learning pipeline with the learning blocks.

We can't wait to see what you'll build! 🚀

This page is part of Adding sight to your sensors and describes how you can use your mobile phone to import image data into Edge Impulse.

To add your phone to your project, go to the Devices page, select Connect a new device and select Use your mobile phone. A QR code will pop up. Scan this code with your phone and your phone will pop up on the devices screen.

1. Collecting images

With your phone connected to your project, it's time to start capturing some images and build our dataset. We have a special UI for collecting images quickly, on your phone choose Collecting images?.

On your phone a permission prompt will show up, and then the viewfinder will be displayed. Set the label (in the top corner) to 'lamp', point your camera at your lamp and press Capture.

Afterwards the photo shows up in the studio on the Data acquisition page.

Do this until you have captured 30 images per class from a variety of angles. Also make sure to vary the things you capture for the unknown class.

2. Alternative: upload data directly

Alternatively you can also capture your dataset directly through a different app, and then upload the data directly to Edge Impulse There are both options to do this visually (click the 'Upload' icon on the data acquisition screen), or via the CLI. You can find instructions here: Uploader. In this case it's highly recommended to you use square images, as the transfer learning model expects these; and you probably want to resize these images before uploading them to make sure training remains fast.

This page is part of and describes how you can use development boards with an integrated camera to import image data into Edge Impulse.

First, make sure your device is connected on the Devices page in the Edge Impulse Studio. Then, head to Data acquisition, and under 'Record new data', set a label and select 'Camera' as a sensor (most devices have multiple resolutions). This shows you a nice preview of the camera. Then click Start sampling.

A few moments later - depending on the speed of the development board and the resolution - you'll now have an image collected!

Do this until you have captured 30 images per class from a variety of angles. Also make sure to vary the things you capture for the unknown class.

In this tutorial, you'll use machine learning to build a system that can recognize when a particular sound is happening—a task known as audio classification. The system you create will be able to recognize the sound of water running from a faucet, even in the presence of other background noise.

You'll learn how to collect audio data from microphones, use signal processing to extract the most important information, and train a deep neural network that can tell you whether the sound of running water can be heard in a given clip of audio. Finally, you'll deploy the system to an embedded device and evaluate how well it works.

At the end of this tutorial, you'll have a firm understanding of how to classify audio using Edge Impulse.

There is also a video version of this tutorial:

1. Prerequisites

If your device is connected under Devices in the studio you can proceed:

2. Collecting your first data

To build this project, you'll need to collect some audio data that will be used to train the machine learning model. Since the goal is to detect the sound of a running faucet, you'll need to collect some examples of that. You'll also need some examples of typical background noise that doesn't contain the sound of a faucet, so the model can learn to discriminate between the two. These two types of examples represent the two classes we'll be training our model to detect: background noise, or running faucet.

You can use your device to collect some data. In the studio, go to the Data acquisition tab. This is the place where all your raw data is stored, and - if your device is connected to the remote management API - where you can start sampling new data.

Let's start by recording an example of background noise that doesn't contain the sound of a running faucet. Under Record new data, select your device, set the label to noise, the sample length to 1000, and the sensor to Built-in microphone. This indicates that you want to record 1 second of audio, and label the recorded data as noise. You can later edit these labels if needed.

After you click Start sampling, the device will capture a second of audio and transmit it to Edge Impulse. The LED will light while recording is in progress, then light again during transmission.

When the data has been uploaded, you will see a new line appear under 'Collected data'. You will also see the waveform of the audio in the 'RAW DATA' box. You can use the controls underneath to listen to the audio that was captured.

3. Build a dataset

Since you now know how to capture audio with Edge Impulse, it's time to start building a dataset. For a simple audio classification model like this one, we should aim to capture around 10 minutes of data. We have two classes, and it's ideal if our data is balanced equally between each of them. This means we should aim to capture the following data:

• 5 minutes of background noise, with the label "noise"

• 5 minutes of running faucet noise, with the label "faucet"

Real world data

In the real world, there are usually additional sounds present alongside the sounds we care about. For example, a running faucet is often accompanied by the sound of dishes being washed, teeth being brushed, or a conversation in the kitchen. Background noise might also include the sounds of television, kids playing, or cars driving past outside.

It's important that your training data contains these types of real world sounds. If your model is not exposed to them during training, it will not learn to take them into account, and it will not perform well during real-world usage.

For this tutorial, you should try to capture the following:

• Background noise

  • 2 minutes of background noise without much additional activity

  • 1 minute of background noise with a TV or music playing

  • 1 minute of background noise featuring occasional talking or conversation

  • 1 minutes of background noise with the sounds of housework

• Running faucet noise

  • 1 minute of a faucet running

  • 1 minute of a different faucet running

  • 1 minute of a faucet running with a TV or music playing

  • 1 minute of a faucet running with occasional talking or conversation

  • 1 minute of a faucet running with the sounds of housework

It's okay if you can't get all of these, as long as you still obtain 5 minutes of data for each class. However, your model will perform better in the real world if it was trained on a representative dataset.

Data capture and transmission

The amount of audio that can be captured in one go varies depending on a device's memory. The ST B-L475E-IOT01A developer board has enough memory to capture 60 seconds of audio at a time, and the Arduino Nano 33 BLE Sense has enough memory for 16 seconds. To capture 60 seconds of audio, set the sample length to 60000. Because the board transmits data quite slowly, it will take around 7 minutes before a 60 second sample appears in Edge Impulse.

Once you've captured around 10 minutes of data, it's time to start designing an Impulse.

Prebuilt dataset

4. Design an Impulse

With the training set in place you can design an impulse. An impulse takes the raw data, slices it up in smaller windows, uses signal processing blocks to extract features, and then uses a learning block to classify new data. Signal processing blocks always return the same values for the same input and are used to make raw data easier to process, while learning blocks learn from past experiences.

For this tutorial we'll use the "MFE" signal processing block. MFE stands for Mel Frequency Energy. This sounds scary, but it's basically just a way of turning raw audio—which contains a large amount of redundant information—into simplified form.

We'll then pass this simplified audio data into a Neural Network block, which will learn to distinguish between the two classes of audio (faucet and noise).

In the studio, go to the Create impulse tab. You'll see a Raw data block, like this one.

As mentioned above, Edge Impulse slices up the raw samples into windows that are fed into the machine learning model during training. The Window size field controls how long, in milliseconds, each window of data should be. A one second audio sample will be enough to determine whether a faucet is running or not, so you should make sure Window size is set to 1000 ms. You can either drag the slider or type a new value directly.

Each raw sample is sliced into multiple windows, and the Window increase field controls the offset of each subsequent window from the first. For example, a Window increase value of 1000 ms would result in each window starting 1 second after the start of the previous one.

By setting a Window increase that is smaller than the Window size, we can create windows that overlap. This is actually a great idea. Although they may contain similar data, each overlapping window is still a unique example of audio that represents the sample's label. By using overlapping windows, we can make the most of our training data. For example, with a Window size of 1000 ms and a Window increase of 100 ms, we can extract 10 unique windows from only 2 seconds of data.

Make sure the Window increase field is set to 300 ms. The Raw data block should match the screenshot above.

Next, click Add a processing block and choose the 'MFE' block. Once you're done with that, click Add a learning block and select 'Classification (Keras)'. Finally, click Save impulse. Your impulse should now look like this:

5. Configure the MFE block

Now that we've assembled the building blocks of our Impulse, we can configure each individual part. Click on the MFE tab in the left hand navigation menu. You'll see a page that looks like this:

The MFE block transforms a window of audio into a table of data where each row represents a range of frequencies and each column represents a span of time. The value contained within each cell reflects the amplitude of its associated range of frequencies during that span of time. The spectrogram shows each cell as a colored block, the intensity which varies depends on the amplitude.

The patterns visible in a spectrogram contain information about what type of sound it represents. For example, the spectrogram in this image shows a pattern typical of background noise:

You can tell that it is slightly different from the following spectrogram, which shows a pattern typical of a running faucet:

These differences are not necessarily easy for a person to describe, but fortunately they are enough for a neural network to learn to identify.

It's interesting to explore your data and look at the types of spectrograms it results in. You can use the dropdown box near the top right of the page to choose between different audio samples to visualize, and drag the white window on the audio waveform to select different windows of data:

There are a lot of different ways to configure the MFCC block, as shown in the Parameters box:

Handily, Edge Impulse provides sensible defaults that will work well for many use cases, so we can leave these values unchanged. You can play around with the noise floor to quickly see the effect it has on the spectrogram.

The spectrograms generated by the MFE block will be passed into a neural network architecture that is particularly good at learning to recognize patterns in this type of tabular data. Before training our neural network, we'll need to generate MFE blocks for all of our windows of audio. To do this, click the Generate features button at the top of the page, then click the green Generate features button. If you have a full 10 minutes of data, the process will take a while to complete:

Next, we'll configure the neural network and begin training.

6. Configure the neural network

With all data processed it's time to start training a neural network. Neural networks are algorithms, modeled loosely after the human brain, that can learn to recognize patterns that appear in their training data. The network that we're training here will take the MFE as an input, and try to map this to one of two classes—noise, or faucet.

Click on NN Classifier in the left hand menu. You'll see the following page:

A neural network is composed of layers of virtual "neurons", which you can see represented on the left hand side of the NN Classifier page. An input—in our case, an MFE spectrogram—is fed into the first layer of neurons, which filters and transforms it based on each neuron's unique internal state. The first layer's output is then fed into the second layer, and so on, gradually transforming the original input into something radically different. In this case, the spectrogram input is transformed over four intermediate layers into just two numbers: the probability that the input represents noise, and the probability that the input represents a running faucet.

During training, the internal state of the neurons is gradually tweaked and refined so that the network transforms its input in just the right ways to produce the correct output. This is done by feeding in a sample of training data, checking how far the network's output is from the correct answer, and adjusting the neurons' internal state to make it more likely that a correct answer is produced next time. When done thousands of times, this results in a trained network.

A particular arrangement of layers is referred to as an architecture, and different architectures are useful for different tasks. The default neural network architecture provided by Edge Impulse will work well for our current project, but you can also define your own architectures. You can even import custom neural network code from tools used by data scientists, such as TensorFlow and Keras.

The default settings should work, and to begin training, click Start training. You'll see a lot of text flying past in the Training output panel, which you can ignore for now. Training will take a few minutes. When it's complete, you'll see the Model panel appear at the right side of the page:

Congratulations, you've trained a neural network with Edge Impulse! But what do all these numbers mean?

At the start of training, 20% of the training data is set aside for validation. This means that instead of being used to train the model, it is used to evaluate how the model is performing. The Last training performance panel displays the results of this validation, providing some vital information about your model and how well it is working. Bear in mind that your exact numbers may differ from the ones in this tutorial.

On the left hand side of the panel, Accuracy refers to the percentage of windows of audio that were correctly classified. The higher number the better, although an accuracy approaching 100% is unlikely, and is often a sign that your model has overfit the training data. You will find out whether this is true in the next stage, during model testing. For many applications, an accuracy above 80% can be considered very good.

The Confusion matrix is a table showing the balance of correctly versus incorrectly classified windows. To understand it, compare the values in each row. For example, in the above screenshot, all of the faucet audio windows were classified as faucet, but a few noise windows were misclassified. This appears to be a great result though.

The On-device performance region shows statistics about how the model is likely to run on-device. Inferencing time is an estimate of how long the model will take to analyze one second of data on a typical microcontroller (here: an Arm Cortex-M4F running at 80MHz). Peak memory usage gives an idea of how much RAM will be required to run the model on-device.

7. Classifying new data

The performance numbers in the previous step show that our model is working well on its training data, but it's extremely important that we test the model on new, unseen data before deploying it in the real world. This will help us ensure the model has not learned to overfit the training data, which is a common occurrence.

Edge Impulse provides some helpful tools for testing our model, including a way to capture live data from your device and immediately attempt to classify it. To try it out, click on Live classification in the left hand menu. Your device should show up in the 'Classify new data' panel. Capture 5 seconds of background noise by clicking Start sampling:

The sample will be captured, uploaded, and classified. Once this has happened, you'll see a breakdown of the results:

Once the sample is uploaded, it is split into windows–in this case, a total of 41. These windows are then classified. As you can see, our model classified all 41 windows of the captured audio as noise. This is a great result! Our model has correctly identified that the audio was background noise, even though this is new data that was not part of its training set.

Of course, it's possible some of the windows may be classified incorrectly. Since our model was 99% accurate based on its validation data, you can expect that at least 1% of windows will be classified wrongly—and likely much more than this, since our validation data doesn't represent every possible type of background or faucet noise. If your model didn't perform perfectly, don't worry. We'll get to troubleshooting later.

8. Model testing

Using the Live classification tab, you can easily try out your model and get an idea of how it performs. But to be really sure that it is working well, we need to do some more rigorous testing. That's where the Model testing tab comes in. If you open it up, you'll see the sample we just captured listed in the Test data panel:

In addition to its training data, every Edge Impulse project also has a test dataset. Samples captured in Live classification are automatically saved to the test dataset, and the Model testing tab lists all of the test data.

To use the sample we've just captured for testing, we should correctly set its expected outcome. Click the ⋮ icon and select Edit expected outcome, then enter noise. Now, select the sample using the checkbox to the left of the table and click Classify selected:

You'll see that the model's accuracy has been rated based on the test data. Right now, this doesn't give us much more information that just classifying the same sample in the Live classification tab. But if you build up a big, comprehensive set of test samples, you can use the Model testing tab to measure how your model is performing on real data.

Ideally, you'll want to collect a test set that contains a minimum of 25% the amount of data of your training set. So, if you've collected 10 minutes of training data, you should collect at least 2.5 minutes of test data. You should make sure this test data represents a wide range of possible conditions, so that it evaluates how the model performs with many different types of inputs. For example, collecting test audio for several different faucets is a good idea.

You can use the Data acquisition tab to manage your test data. Open the tab, and then click Test data at the top. Then, use the Record new data panel to capture a few minutes of test data, including audio for both background noise and faucet. Make sure the samples are labelled correctly. Once you're done, head back to the Model testing tab, select all the samples, and click Classify selected:

The screenshot shows classification results from a large number of test samples (there are more on the page than would fit in the screenshot). The panel shows that our model is performing at 85% accuracy, which is 5% less than how it performed on validation data. It's normal for a model to perform less well on entirely fresh data, so this is a successful result. Our model is working well!

For each test sample, the panel shows a breakdown of its individual performance. For example, one of the samples was classified with only 62% accuracy. Samples that contain a lot of misclassifications are valuable, since they have examples of types of audio that our model does not currently fit. It's often worth adding these to your training data, which you can do by clicking the ⋮ icon and selecting Move to training set. If you do this, you should add some new test data to make up for the loss!

Testing your model helps confirm that it works in real life, and it's something you should do after every change. However, if you often make tweaks to your model to try to improve its performance on the test dataset, your model may gradually start to overfit to the test dataset, and it will lose its value as a metric. To avoid this, continually add fresh data to your test dataset.

9. Model troubleshooting

If the network performed great, fantastic! But what if it performed poorly? There could be a variety of reasons, but the most common ones are:

1. The data does not look like other data the network has seen before. This is common when someone uses the device in a way that you didn't add to the test set. You can add the current file to the test set by adding the correct label in the 'Expected outcome' field, clicking ⋮, then selecting Move to training set.

2. The model has not been trained enough. Increase number of epochs to 200 and see if performance increases (the classified file is stored, and you can load it through 'Classify existing validation sample').

3. The model is overfitting and thus performs poorly on new data. Try reducing the number of epochs, reducing the learning rate, or adding more data.

4. The neural network architecture is not a great fit for your data. Play with the number of layers and neurons and see if performance improves.

As you see, there is still a lot of trial and error when building neural networks. Edge Impulse is continually adding features that will make it easier to train an effective model.

10. Deploying to your device

With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the MFE algorithm, neural network weights, and classification code - in a single C++ library that you can include in your embedded software.

To export your model, click on Deployment in the menu. Then under 'Build firmware' select your development board, and click Build. This will export the impulse, and build a binary that will run on your development board in a single step. After building is completed you'll get prompted to download a binary. Save this on your computer.

Flashing the device

When you click the Build button, you'll see a pop-up with text and video instructions on how to deploy the binary to your particular device. Follow these instructions. Once you are done, we are ready to test your impulse out.

Running the model on the device

We can connect to the board's newly flashed firmware over serial. Open a terminal and run:

$ edge-impulse-run-impulse

This will capture audio from the microphone, run the MFE code, and then classify the spectrogram:

Starting inferencing in 2 seconds...
Recording
Recording OK
Predictions (DSP: 399 ms., Classification: 175 ms., Anomaly: 0 ms.): 
    faucet: 0.03757
    noise: 0.96243
Starting inferencing in 2 seconds...

Great work! You've captured data, trained a model, and deployed it to an embedded device. It's time to celebrate—by pouring yourself a nice glass of water, and checking whether the sound is correctly classified by you model.

11. Conclusion

We can't wait to see what you'll build! 🚀

Object detection tasks take an image and output information about the class and number of objects, position, (and, eventually, size) in the image.

Edge Impulse provides, by default, two different model architectures to perform object detection, MobileNetV2 SSD FPN-Lite uses bounding boxes (objects location and size) and FOMO uses centroids (objects location only).

Bounding Boxes

• (bounding boxes) Can run on systems starting from Linux CPUs up to powerful GPUs

Centroid

• (centroids) Can run on high-end MCUs, Linux CPUs, and GPUs

In this tutorial, you'll use machine learning to build a system that can recognize and track multiple objects in your house through a camera - a task known as object detection. Adding sight to your embedded devices can make them see the difference between poachers and elephants, count objects, find your lego bricks, and detect dangerous situations. In this tutorial, you'll learn how to collect images for a well-balanced dataset, how to apply transfer learning to train a neural network and deploy the system to an edge device.

At the end of this tutorial, you'll have a firm understanding of how to do object detection using Edge Impulse.

There is also a video version of this tutorial:

1. Prerequisites

2. Building a dataset

In this tutorial we'll build a model that can distinguish between two objects on your desk - we've used a lamp and a coffee cup, but feel free to pick two other objects. To make your machine learning model see it's important that you capture a lot of example images of these objects. When training the model these example images are used to let the model distinguish between them.

Capturing data

Capture the following amount of data - make sure you capture a wide variety of angles and zoom level. It's fine if both images are in the same frame. We'll be cropping the images later to be square so make sure the objects are in the frame.

• 30 images of a lamp.

• 30 images of a coffee cup.

You can collect data from the following devices:

Or you can capture your images using another camera, and then upload them by going to Data acquisition and clicking the 'Upload' icon.

With the data collected we need to label this data. Go to Data acquisition, verify that you see your data, then click on the 'Labeling queue' to start labeling.

No labeling queue? Go to Dashboard, and under 'Project info > Labeling method' select 'Bounding boxes (object detection)'.

Labeling data

The labeling queue shows you all the unlabeled data in your dataset. Labeling your objects is as easy as dragging a box around the object, and entering a label. To make your life a bit easier we try to automate this process by running an object tracking algorithm in the background. If you have the same object in multiple photos we thus can move the boxes for you and you just need to confirm the new box. After dragging the boxes, click Save labels and repeat this until your whole dataset is labeled.

Afterwards you should have a well-balanced dataset listed under Data acquisition in your Edge Impulse project.

Rebalancing your dataset

To validate whether a model works well you want to keep some data (typically 20%) aside, and don't use it to build your model, but only to validate the model. This is called the 'test set'. You can switch between your training and test sets with the two buttons above the 'Data collected' widget. If you've collected data on your development board there might be no data in the testing set yet. You can fix this by going to Dashboard > Perform train/test split.

3. Designing an impulse

With the training set in place you can design an impulse. An impulse takes the raw data, adjusts the image size, uses a preprocessing block to manipulate the image, and then uses a learning block to classify new data. Preprocessing blocks always return the same values for the same input (e.g. convert a color image into a grayscale one), while learning blocks learn from past experiences.

In the studio go to Create impulse, set the image width and image height to 320, the 'resize mode' to Fit shortest axis and add the 'Images' and 'Object Detection (Images)' blocks. Then click Save impulse.

Configuring the processing block

To configure your processing block, click Images in the menu on the left. This will show you the raw data on top of the screen (you can select other files via the drop down menu), and the results of the processing step on the right. You can use the options to switch between 'RGB' and 'Grayscale' mode, but for now leave the color depth on 'RGB' and click Save parameters.

This will send you to the 'Feature generation' screen. In here you'll:

• Resize all the data.

• Apply the processing block on all this data.

• Create a 3D visualization of your complete dataset.

Click Generate features to start the process.

Afterwards the 'Feature explorer' will load. This is a plot of all the data in your dataset. Because images have a lot of dimensions (here: 320x320x3=307,200 features) we run a process called 'dimensionality reduction' on the dataset before visualizing this. Here the 307,200 features are compressed down to just 3, and then clustered based on similarity. Even though we have little data you can already see the clusters forming (lamp images are all on the left, coffee all on the right), and can click on the dots to see which image belongs to which dot.

Configuring the transfer learning model

With all data processed it's time to start training a neural network. Neural networks are a set of algorithms, modeled loosely after the human brain, that are designed to recognize patterns. The network that we're training here will take the image data as an input, and try to map this to one of the three classes.

It's very hard to build a good working computer vision model from scratch, as you need a wide variety of input data to make the model generalize well, and training such models can take days on a GPU. To make this easier and faster we are using transfer learning. This lets you piggyback on a well-trained model, only retraining the upper layers of a neural network, leading to much more reliable models that train in a fraction of the time and work with substantially smaller datasets.

To configure the transfer learning model, click Object detection in the menu on the left. Here you can select the base model (the one selected by default will work, but you can change this based on your size requirements), and set the rate at which the network learns.

Leave all settings as-is, and click Start training. After the model is done you'll see accuracy numbers below the training output. You have now trained your model!

4. Validating your model

With the model trained let's try it out on some test data. When collecting the data we split the data up between a training and a testing dataset. The model was trained only on the training data, and thus we can use the data in the testing dataset to validate how well the model will work in the real world. This will help us ensure the model has not learned to overfit the training data, which is a common occurrence.

To validate your model, go to Model testing and select Classify all. Here we hit 92.31% precision, which is great for a model with so little data.

To see a classification in detail, click the three dots next to an item, and select Show classification. This brings you to the Live classification screen with much more details on the file (you can also capture new data directly from your development board from here). This screen can help you determine why items were misclassified.

Live Classification Result

This view is particularly useful for a direct comparison between the raw image and the model's interpretation. Each object detected in the image is highlighted with a bounding box. Alongside these boxes, you'll find labels and confidence scores, indicating what the model thinks each object is and how sure it is about its prediction. This mode is ideal for understanding the model's performance in terms of object localization and classification accuracy.

Overlay Mode for the Live Classification Result

In this view, bounding boxes are drawn around the detected objects, with labels and confidence scores displayed within the image context. This approach offers a clearer view of how the bounding boxes align with the objects in the image, making it easier to assess the precision of object localization. The overlay view is particularly useful for examining the model's ability to accurately detect and outline objects within a complex visual scene.

Summary Table

Name: This field displays the name of the sample file analyzed by the model. For instance, 'sample.jpg.22l74u4f' is the file name in this case.

CATEGORY: Lists the types of objects that the model has been trained to detect. In this example, two categories are shown: 'coffee' and 'lamp'.

COUNT: Indicates the number of times each category was detected in the sample file. In this case, both 'coffee' and 'lamp' have a count of 1, meaning each object was detected once in the sample.

INFO: This column provides additional information about the model's performance. It displays the 'Precision score', which, in this example, is 95.00%. The precision score represents the model's accuracy in making correct predictions over a range of Intersection over Union (IoU) values, known as the mean Average Precision (mAP).

5. Running the model on your device

With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the preprocessing steps, neural network weights, and classification code - in a single C++ library or model file that you can include in your embedded software.

Running the impulse on your Raspberry Pi 4 or Jetson Nano

From the terminal just run edge-impulse-linux-runner. This will build and download your model, and then run it on your development board. If you're on the same network you can get a view of the camera, and the classification results directly from your dev board. You'll see a line like:

Want to see a feed of the camera and live classification in your browser? Go to http://192.168.1.19:4912

Open this URL in a browser to see your impulse running!

Running the impulse on your mobile phone

On your mobile phone just click Switch to classification mode at the bottom of your phone screen. Point it at an object and press 'Capture'.

Integrating the model in your own application

We can't wait to see what you'll build! 🚀

Neural networks are not limited to working with one type of data at a time. One of their biggest advantages is that they are incredibly flexible with the type of input data, so long as the format and ordering of that data stays the same from training to inference. As a result, we can use them to perform sensor fusion for a variety of tasks.

Sensor fusion is the process of combining data from different types of sensors or similar sensors mounted in different locations, which gives us more information to make decisions and classifications. For example, you could use temperature data with accelerometer data to get a better idea of a potential anomaly!

In this tutorial, you will learn how to use Edge Impulse to perform sensor fusion on the Arduino Nano 33 BLE Sense.

Example Project: You can find the dataset and impulse used throughout this tutorial in .

1. Prerequisites

For this tutorial, you'll need a .

2. Building a dataset

For this demo, we'll show you how to identify different environments by using a fusion of temperature, humidity, pressure, and light data. In particular, I'll have the Arduino board identify different rooms in my house as well as outside. Note that the we assume that the environment is static--if I turn out lights or the outside temperature changes, the model will not work. However, it demonstrates how we can combine different sensor data with machine learning to do classification!

As we will be collecting data from our Arduino board connected to a computer, it helps to have a laptop that you can move to different rooms.

Create a new project on the Edge Impulse studio.

Connect the Arduino Nano 33 BLE to your computer. Follow the to upload the Edge Impulse firmware to the board and connect it to your project.

Go to Data acquisition. Under Record new data, select your device and set the label to bedroom. Change Sensor to Environmental + Interactional, set the Sample length to 10000 ms and Frequency to 12.5Hz.

Stand in one of your rooms with your Arduino board (and laptop). Click Start sampling and slowly move the board around while data is collected. After sampling is complete, you should see a new data plot with a different line for each sensor.

Repeat this process to record about 3 minutes of data for the bedroom class. Try to stand in a different spot in the room while collecting data--we want a robust dataset that represents the features of each room. Head to another room and repeat data collection. Continue doing this until you have around 3 minutes of data for each of the following classes:

• Bedroom

• Hallway

• Outside

You are welcome to try other rooms or locations. For this demo, I found that my bedroom, kitchen, and living room all exhibited similar environmental and lighting properties, so the model struggled to tell them apart.

Head to Dashboard and scroll down to Danger zone. Click Perform train/test split and follow the instructions in the pop-up window to split your dataset into training and testing groups. When you're done, you can head back to Data acquisition to see that your dataset has been split. You should see about 80% of your samples in Training data and about 20% in Test data.

4. Design an Impulse

Head to Create impulse. Change the Window increase to 500 ms. Add a Flatten block. Notice that you can choose which environmental and interactional sensor data to include. Deselect proximity and gesture, as we won't need those to detect rooms. Add a Classification (Keras) learning block

Click Save impulse.

5. Configure the Flatten block

Head to Flatten. You can select different samples and move the window around to see what the DSP result will look like for each set of features to be sent to the learning block.

The Flatten block will compute the average, minimum, maximum, root-mean square, standard deviation, skewness, and kurtosis of each axis (e.g. temperature, humidity, brightness, etc.). With 7 axes and 7 features computed for each axis, that gives us 49 features for each window being sent to the learning block. You can see these computed features under Processed features.

Click Save parameters. On the next screen, select Calculate feature importance and click Generate features.

After a few moments, you should be able to explore the features of your dataset to see if your classes are easily separated into categories.

Interestingly enough, it looks like temperature and red light values were the most important features in determining the location of the Arduino board.

6. Configure the neural network

With our dataset collected and features processed, we can train our machine learning model. Click on NN Classifier. Change the Number of training cycles to 300 and click Start training. We will leave the neural network architecture as the default for this demo.

During training, parameters in the neural network's neurons are gradually updated so that the model will try to guess the class of each set of data as accurately as possible. When training is complete, you should see a Model panel appear on the right side of the page.

The Confusion matrix gives you an idea of how well the model performed at classifying the different sets of data. The top row gives the predicted label and the column on the left side gives the actual (ground-truth) label. Ideally, the model should predict the classes correctly, but that's not always the case. You want the diagonal cells from the top-left to the bottom-right to be as close to 100% as possible.

The On-device performance provides some statistics about how the model will likely run on a particular device. By default, an Arm Cortex-M4F running at 80 MHz is assumed to be your target device. The actual memory requirements and run time may vary on different platforms.

7. Model testing

Rather than simply assume that our model will work when deployed, we can run inference on our test dataset as well as on live data.

First, head to Model testing, and click Classify all. After a few moments, you should see results from your test set.

You can click on the three dots next to an item and select Show classification. This will give you a classification result screen where you can see results information in more detail.

Additionally, we can test the impulse in a real-world environment to make sure the model has not overfit the training data. To do that, head to Live classification. Make sure your device is connected to the Studio and that the Sensor, Sample length, and Frequency match what we used to initially capture data.

Click Start sampling. A new sample will be captured from your board, uploaded, and classified. Once complete, you should see the classification results.

In the example above, we sampled 10 seconds of data from the Arduino. This data is split into 1-second windows (the window moves over 0.5 seconds each time), and the data in that window is sent to the DSP block. The DSP block computes the 49 features that are then sent to the trained machine learning model, which performs a forward pass to give us our inference results.

As you can see, the inference results from all of the windows claimed that the Arduino board was in the bedroom, which was true! This is great news for our model--it seems to work even on unseen data.

8. Running the impulse on your device

Now that we have an impulse with a trained model and we've tested its functionality, we can deploy the model back to our device. This means the impulse can run locally without an internet connection to perform inference!

Edge Impulse can package up the entire impulse (preprocessing block, neural network, and classification code) into a single library that you can include in your embedded software.

Click on Deployment in the menu. Select the library that you would like to create, and click Build at the bottom of the page.

9. Conclusion

Well done! You've trained a neural network to determine the location of a development board based on a fusion of several sensors working in tandem. Note that this demo is fairly limited--as the daylight or temperature changes, the model will no longer be valid. However, it hopefully gives you some ideas about how you can mix and match sensors to achieve your machine learning goals.

We can't wait to see what you'll build! 🚀

is a brand-new approach to run object detection models on constrained devices. FOMO is a ground-breaking algorithm that brings real-time object detection, tracking and counting to microcontrollers for the first time. FOMO is 30x faster than MobileNet SSD and can run in <200K of RAM.

In this tutorial, we will explain how to count cars to estimate parking occupancy using FOMO.

View the finished project, including all data, signal processing and machine learning blocks here: .

1. Prerequisites

For this tutorial, you'll need a .

If you don't have any of these devices, you can also upload an existing dataset through the or use your to connect your device to Edge Impulse. After this tutorial, you can then deploy your trained machine learning model as a C++ library or as a WebAssembly package and run it on your device.

2. Building a dataset

Capturing data

You can collect data from the following devices:

• - for the Raspberry Pi 4 and the Jetson Nano.

• Collecting image data from any of the that have a camera.

With the data collected, we need to label this data. Go to Data acquisition, verify that you see your data, then click on the 'Labeling queue' to start labeling.

Labeling data

• Using your own trained model - Useful when you already have a trained model with classes similar to your new task.

• Using Object tracking - Useful when you have objects that are similar in size and common between images/frames.

For our case, since the 'car' object is part of the COCO dataset, we will use the YoloV5 pre-trained model to accelerate this process. To enable this feature, we will first click the Label suggestions dropdown,then select “Classify using YOLOv5.”

From the image above, the YOLOV5 model can already help us annotate more than 90% of the cars without us having to do it manually by our hands.

Rebalancing your dataset

To validate whether a model works well you want to keep some data (typically 20%) aside, and don't use it to build your model, but only to validate the model. This is called the 'test set'. You can switch between your training and test sets with the two buttons above the 'Data collected' widget. If you've collected data on your development board there might be no data in the testing set yet. You can fix this by going to Dashboard > Perform train/test split.

3. Designing an impulse

To configure this, go to Create impulse, set the image width and image height to 96, the 'resize mode' to Fit shortest axis and add the 'Images' and 'Object Detection (Images)' blocks. Then click Save Impulse.

Configuring the processing block

To configure your processing block, click Images in the menu on the left. This will show you the raw data on top of the screen (you can select other files via the drop-down menu), and the results of the processing step on the right. You can use the options to switch between RGB and Grayscale modes. Finally, click on Save parameters.

This will send you to the 'Feature generation' screen. In here you'll:

• Resize all the data.

• Apply the processing block on all this data.

• Create a 3D visualization of your complete dataset.

• Click Generate features to start the process.

Afterward, the Feature explorer will load. This is a plot of all the data in your dataset. Because images have a lot of dimensions (here: 96x96x1=9216 features for grayscale) we run a process called 'dimensionality reduction' on the dataset before visualizing this. Here the 9216 features are compressed down to 2, and then clustered based on similarity as shown in the feature explorer below.

Configuring the object detection model with FOMO

With all data processed it's time to start training our FOMO model. The model will take an image as input and output objects detected using centroids. For our case, it will show centroids of cars detected on the images.

FOMO is fully compatible with any MobileNetV2 model, and depending on where the model needs to run you can pick a model with a higher or lower alpha. Transfer learning also works (although you need to train your base models specifically with FOMO in mind). Another advantage of FOMO is that it has very few parameters to learn from compared to normal SSD networks making the network even much smaller and faster to train. Together this gives FOMO the capabilities to scale from the smallest microcontrollers to full gateways or GPUs.

To configure FOMO, head over to the ‘Object detection’ section, and select 'Choose a different model' then select one of the FOMO models as shown in the image below.

Make sure to start with a learning rate of 0.001 then click start training. After the model is done you'll see accuracy numbers below the training output. You have now trained your FOMO object detection model!

As you may have noticed from the training results above, FOMO uses F1 Score as its base evaluating metric as compared to SSD MobileNetV2 which uses Mean Average Precision (mAP). Using Mean Average Precision (mAP) as the sole evaluation metric can sometimes give limited insights into the model’s performance. This is particularly true when dealing with datasets with imbalanced classes as it only measures how accurate the predictions are without putting into account how good or bad the model is for each class. The combination between F1 score and a confusion matrix gives us both the balance between precision and recall of our model as well as how the model performs for each class.

4. Validating your model

With the model trained let's try it out on some test data. When collecting the data we split the data up between a training and a testing dataset. The model was trained only on the training data, and thus we can use the data in the testing dataset to validate how well the model will work in the real world. This will help us ensure the model has not learned to overfit the training data, which is a common occurrence. To validate our model, we will go to Model Testing and select Classify all.

Given the little training data we had and the few cycles we trained on, we got an accuracy of 84.62% which can be improved further. To see the classification in detail, we will head to Live Classification* and select one image from our test sample. Click the three dots next to an item, and select Show classification. We can also capture new data directly from your development board from here.

Live Classification Result

From the test image above, our model was able to detect 16 cars out of the actual possible 18 which is a good performance. This can be seen in side by side by default, but you can also switch to overlay mode to see the model's predictions against the actual image content.

Overlay Mode for the Live Classification Result

A display option where the original image and the model's detections overlap, providing a clear juxtaposition of the model's predictions against the actual image content.

Summary Table

The summary table for a FOMO classification result provides a concise overview of the model's performance on a specific sample file, such as 'Parking_data_2283.png.2tk8c1on'. This table is organized as follows:

CATEGORY: Metric, Object category, or class label, e.g., car. COUNT: Shows detection accuracy, frequency, e.g., car detected 7 times.

INFO: Provides performance metrics definitions, including F1 Score, Precision, and Recall, which offer insights into the model's accuracy and efficacy in detection:

Table Metrics F1 Score: (77.78%): Balances precision and recall. Precision: (100.00%): Accuracy of correct predictions. Recall: (63.64%): Proportion of actual objects detected.

Viewing Options

Bottom-right controls adjust the visibility of ground truth labels and model predictions, enhancing the analysis of the model's performance:

Prediction Controls: Customize the display of model predictions, including:

• Show All: Show all detections and confidence scores.

• Show Correct Only: Focus on accurate model predictions.

• Show incorrect only: Pinpoint undetected objects in the ground truth.

Ground Truth Controls: Toggle the visibility of original labels for direct comparison with model predictions.

• Show All: Display all ground truth labels.

• Hide All: Conceal all ground truth labels.

• Show detected only: Highlight ground truth labels detected by the model.

• Show undetected only: Identify ground truth labels missed by the model.

5. Running the model on your device

With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption. Edge Impulse can package up the complete impulse - including the preprocessing steps, neural network weights, and classification code - in a single C++ library or model file that you can include in your embedded software.

Running the impulse on a Linux-based device

From the terminal just run edge-impulse-linux-runner. This will build and download your model, and then run it on your development board. If you're on the same network you can get a view of the camera, and the classification results directly from your dev board. You'll see a line like:

Want to see a feed of the camera and live classification in your browser? Go to http://192.168.8.117:4912

Open this URL in a browser to see your impulse running!

Running the impulse on a fully supported MCU

Go to the Deployment tab, on Build firmware section and select the board-compatible firmware to download it.

Follow the instruction provided to flash the firmware to your board and head over to your terminal and run the edge-impulse-run-impulse --debug command:

You'll also see a URL you can use to view the image stream and results in your browser:

Want to see a feed of the camera and live classification in your browser? Go to http://192.168.8.117:4912

Running the impulse using a generated Arduino Library

Integrating the model in your own application

We can't wait to see what you'll build! 🚀

Sensor fusion is about combining data from various sensors to gain a more comprehensive understanding of your environment. In this tutorial, we will demonstrate sensor fusion by bringing together high-dimensional audio or image data with time-series sensor data. This combination allows you to extract deeper insights from your sensor data.

Concepts

Also, see this video (starting min 13):

The Challenge of Handling Multiple Data Types

When you have data coming from multiple sources, such as a microphone capturing audio, a camera capturing images, and sensors collecting time-series data. Integrating these diverse data types can be tricky and conventional methods fall short.

The Limitations of Default Workflows

With the standard workflow, if you have data streams from various sources, you might want to create separate DSP blocks for each data type. For instance, if you're dealing with audio data from microphones, image data from cameras, and time-series sensor data from accelerometers, you could create separate DSP blocks for each. For example:

• A spectrogram-based DSP block for audio data

• An image DSP block for image data

• A spectral analysis block for time-series sensor data

This approach initially seems logical but comes with limitations:

When using separate DSP blocks, you're constrained in your choice of neural networks. The features extracted from each data type are fundamentally different. For example, a pixel in an image or an image's spectrogram and a data point from an accelerometer's time-series data have distinct characteristics. This incompatibility makes it challenging to use a convolutional neural network (CNN) that is typically effective for image data or spectrogram. As a result, fully connected networks may be your only option, which are not ideal for audio or image data.

The Role of Embeddings

To bypass the limitation stated above, you may consider using neural networks embeddings. In essence, embeddings are compact, meaningful representations of your data, learned by a neural network.

While training the neural network, the model try to find the mathematical formula that best maps the input to the output. This is done by tweaking each neuron (each neuron is a parameter in our formula). The interesting part is that each layer of the neural network will start acting like a feature extracting step but highly tuned for your specific data.

Finally, instead of having a classifier for last layer (usually a softmax layer), we cut the neural network somewhere at the end and we obtained the embeddings.

Thus, we can consider the embeddings as learnt features and we will pass these "features" to the final Impulse:

Sensor Fusion Workflow using Embeddings

Here's how we approach advanced sensor fusion with Edge Impulse.

In this workflow, we will show how to perform sensor fusion using both audio data and accelerometer data to classify different stages of a grinding coffee machine (grind, idle, pump and extract). First, we are going to use a spectrogram DSP block and a NN classifier using two dense network. This first impulse will then be used to generate the embeddings and will be made available in a custom DSP block. Finally, we are going to train a fully connected layer using features coming from both the generated embeddings and a spectral feature DSP block.

We have develop two Edge Impulse public projects, one publicly available dataset and a Github repository containing the source code to help you follow the steps:

Please note that with a few changes, you will be able to change the sensor type (audio to images) or the first pre-processing method (spectrogram to MFE/MFCC).

1. Prepare your dataset

The first step is to have input data samples that contain both sensors. In Edge Impulse studio, you can easily visualize time-series data, like audio and accelerometer data.

Note: it is not trivial to group together images and time-series. Our core-engineering team is working on improving this workflow. In the meantime, as a workaround, you can encode your image as time-series with one axis per channel (red, green, blue) plus the sensor:

2. Training Edge Impulse projects for each high-dimensional sensor data

Train separate projects for high dimensional data (audio or image data). Each project contains both a DSP block and a trained neural network.

3. Generate the Embeddings using the exported C++ inferencing SDK

Clone this repository:

git clone https://github.com/edgeimpulse/example-sensor-fusion-using-embeddings.git

Download the generated Impulse to extract the embeddings, which encapsulate distilled knowledge about their respective data types.

Download Test Data: From the same dashboard, download the test or train data NPY file (input.npy). Place this numpy array file under the /input repository. This will allow us to generate a quantized version of the tflite embeddings. Ideally choose the test data if you have some data available.

Generate the embeddings:

python saved-model-to-embeddings.py --input input/ --output dsp-blocks/features-from-audio-embeddings

This will cut off the last layer of the neural network and convert it to TensorFlow Lite (TFLite) format. You can follow the process outlined in the saved_model_to_embeddings.py script for this conversion for a better understanding.

4. Encapsulate the Embeddings in a Custom DSP Block

To make sensor fusion work seamlessly, Edge Impulse enables you to create custom DSP blocks. These blocks combine the necessary spectrogram/image-processing and neural network components for each data type.

Custom DSP Block Configuration: In the DSP block, perform two key operations as specified in the dsp.py script:

• Run the DSP step with fixed parameters.

• Run the neural network.

Replace this following lines in dsp-blocks/features-from-audio-embeddings/dsp.py to match your DSP configuration:

def generate_features(implementation_version, draw_graphs, raw_data, axes, sampling_freq):
    frame_length = 0.032
    frame_stride = 0.024
    fft_length = 128
    noise_floor_db = -85
    ...

Return Neural Network Embeddings: The DSP block should be configured to return the neural network embeddings, as opposed to the final classification result.

Implement get_tflite_implementation: Ensure that the get_tflite_implementation function returns the TFLite model. Note that the on-device implementation will not be correct initially when generating the C++ library, as only the neural network part is compiled. We will fix this in the final exported C++ Library.

Now publish your new custom DSP block.

cd dsp-blocks/features-from-audio-embeddings
edge-impulse-block init

Fill the necessary information and push your block:

edge-impulse-block push

5. Create a New Impulse

Multiple DSP Blocks: Create a new impulse with three DSP blocks and a classifier block. The routing should be as follows:

• Audio data routed through the custom block.

• Sensor data routed through spectral analysis.

Training the Model: Train the model within the new impulse, using a fully-connected network.

6. Export and Modify the C++ Library

Export as a C++ Library:

• In the Edge Impulse platform, export your project as a C++ library.

• Choose the model type that suits your target device (quantized vs. float32).

• Make sure to select EON compiler option

• Copy the exported C++ library to the example-cpp folder for easy access.

Add a Forward Declaration:

• In the model-parameters/model_variables.h file of the exported C++ library, add a forward declaration for the custom DSP block you created.

For example:

int custom_sensor_fusion_features(signal_t *signal, matrix_t *output_matrix, void *config_ptr, const float frequency);

And change &extract_tflite_eon_features into &custom_sensor_fusion_features in the ei_dsp_blocks object.

ei_model_dsp_t ei_dsp_blocks[ei_dsp_blocks_size] = {
    { // DSP block 46
        207,
        &extract_spectral_analysis_features,
        (void*)&ei_dsp_config_46,
        ei_dsp_config_46_axes,
        ei_dsp_config_46_axes_size
    },
    { // DSP block 58
        416,
        &custom_sensor_fusion_features, // <-- change is here
        (void*)&ei_dsp_config_58,
        ei_dsp_config_58_axes,
        ei_dsp_config_58_axes_size
    }
};

Implement the Custom DSP Block:

In the main.cpp file of the C++ library, implement the custom_sensor_fusion_features block. This block should:

1. Call into the Edge Impulse SDK to generate features.

2. Execute the rest of the DSP block, including neural network inference.

/**
 Custom DSP function implementation
 */
int custom_sensor_fusion_features(signal_t *signal, matrix_t *output_matrix, void *config_ptr, const float frequency) {
...
}

7. Compile and run the app

• Copy a test sample's raw features into the features[] array in source/main.cpp

• Enter make -j in this directory to compile the project. If you encounter any OOM memory error try make -j4 (replace 4 with the number of cores available)

• Enter ./build/app to run the application

• Compare the output predictions to the predictions of the test sample in the Edge Impulse Studio.

run_classifier returned: 0
Timing: DSP 4 ms, inference 0 ms, anomaly 0 ms
Predictions:
  extract: 0.01953
  grind: 0.98047
  idle: 0.00000
  pump: 0.00000

Note that if you are using the quantized version of the model, you may encounter a slight difference between the Studio Live Classification page and the above results, the float32 model however should give you the same results.

This page is part of and describes how you can use the OpenMV Cam H7 Plus to build a dataset, and import the data into Edge Impulse.

1. Setting up your environment

To set up your OpenMV camera, and collect some data:

1. Install the .

2. Follow the to clean the sensor and focus the lens.

3. Connect a micro-USB cable to the camera, and open the OpenMV IDE. The camera should automatically update to the latest firmware.

4. Verify that the camera can capture live images, by clicking on the Connect button in the bottom left corner, then pressing Play to run the application.

A live feed from your camera will be displayed in the top right corner of the IDE.

2. Collecting images

Once your camera is up and running, it's time to start capturing some images and build our dataset.

First, set up a new dataset via Tools -> Dataset Editor, select New Dataset.

This opens the 'Dataset editor' panel on the left side, and the 'dataset capture script' in the main panel of the IDE. Here, create three classes: "plant", "lamp" and "unknown". It's important to add an unknown class that contains random images which are neither lamps nor plants.

As we'll build a model that takes in square images, change the 'Dataset capture script' to read:

import sensor, image, time

sensor.reset()
sensor.set_pixformat(sensor.RGB565) # Modify as you like.
sensor.set_framesize(sensor.QVGA) # Modify as you like.
sensor.set_windowing((240, 240)) # Modify as you like.
sensor.skip_frames(time = 2000)

clock = time.clock()

while(True):
    clock.tick()
    img = sensor.snapshot()
    print(clock.fps())

Now you can capture data for the three classes.

1. Click the Play icon to run the 'dataset capture script' on your OpenMV camera.

2. Select one of the classes by clicking on the folder name in the 'Dataset editor'.

3. Take a snap by clicking the Capture data (camera icon) button.

Do this until you have captured 30 images per class from a variety of angles. Also make sure to vary the things you capture for the unknown class.

3. Sending the dataset to Edge Impulse

To import the dataset into Edge Impulse go to Tools > Dataset Editor > Export > Upload to Edge Impulse project.

Then, choose the project name, and the split between training and testing data (recommended to keep this to 80/20).

A duplicate check runs when you upload new data, so you can upload your dataset multiple times (for example, when you've added new files) without adding the same data twice.

Our dataset now appears under the Data acquisition section of our project.

The Edge Impulse API exposes programmatic access to most functionality in the studio and it is particularly useful when it comes to automating tasks. You can use the API to edit the labels of many samples at once, train models, or create new impulses. In addition, you can subscribe to events, such as when a new file is processed by the ingestion service. An Edge Impulse Python API bindings is also available as the edgeimpulse-api pip package.

See the following examples:

• Running jobs using the API

• Using the Python SDK API Bindings

• Customize the EON Tuner

• Trigger connected board data sampling

In the advanced inferencing tutorials section, you will discover useful techniques to leverage our inferencing libraries or how you can use the inference results in your application logic:

• Continuous audio sampling

• Multi-impulse

• Count objects using FOMO

The Edge Impulse API gives programmatic access to all features in the studio, and many tasks that might normally have to be performed manually can thus be automated. In this tutorial you'll create a job that deploys your model (as a ZIP file), you monitor the job status, and then finally download the deployed model. The tutorial uses Python, but you can use any environment capable of scripting HTTP requests.

To run this script you'll need:

• A recent version of Python 3.

• The requests module:

pip3 install requests

• Your project ID (can be found on Dashboard in the Edge Impulse Studio).

• An API Key (under Dashboard > Keys).

Then, create the following script build-model.py:

import requests, json, datetime, time, re

project_id = 1 # YOUR PROJECT ID
api_key = "ei_..." # YOUR API KEY
deploy_type = "zip" # CAN CHANGE TO DIFFERENT TYPE

def build_model():
    url = f"https://studio.edgeimpulse.com/v1/api/{project_id}/jobs/build-ondevice-model"
    querystring = {"type": deploy_type}
    payload = {"engine": "tflite-eon"}
    headers = {
        "x-api-key": api_key,
        "Accept": "application/json",
        "Content-Type": "application/json",
    }
    response = requests.request("POST", url, json=payload, headers=headers, params=querystring)
    body = json.loads(response.text)
    if (not body['success']):
        raise Exception(body['error'])
    return body['id']

def get_stdout(job_id, skip_line_no):
    url = f"https://studio.edgeimpulse.com/v1/api/{project_id}/jobs/{job_id}/stdout"
    headers = {
        "x-api-key": api_key,
        "Accept": "application/json",
        "Content-Type": "application/json",
    }
    response = requests.request("GET", url, headers=headers)
    body = json.loads(response.text)
    if (not body['success']):
        raise Exception(body['error'])
    stdout = body['stdout'][::-1] # reverse array so it's old -> new
    return [ x['data'] for x in stdout[skip_line_no:] ]

def wait_for_job_completion(job_id):
    skip_line_no = 0

    url = f"https://studio.edgeimpulse.com/v1/api/{project_id}/jobs/{job_id}/status"
    headers = {
        "x-api-key": api_key,
        "Accept": "application/json",
        "Content-Type": "application/json",
    }
    while True:
        response = requests.request("GET", url, headers=headers)
        body = json.loads(response.text)
        if (not body['success']):
            raise Exception(body['error'])

        stdout = get_stdout(job_id, skip_line_no)
        for l in stdout:
            print(l, end='')
        skip_line_no = skip_line_no + len(stdout)

        if (not 'finished' in body['job']):
            # print('Job', job_id, 'is not finished yet...', body['job'])
            time.sleep(1)
            continue
        if (not body['job']['finishedSuccessful']):
            raise Exception('Job failed')
        else:
            break

def download_model():
    url = f"https://studio.edgeimpulse.com/v1/api/{project_id}/deployment/download"
    querystring = {"type": deploy_type}
    headers = {
        "x-api-key": api_key,
        "Accept": "application/json",
        "Content-Type": "application/json",
    }
    response = requests.request("GET", url, headers=headers, params=querystring)

    d = response.headers['Content-Disposition']
    fname = re.findall("filename\*?=(.+)", d)[0].replace('utf-8\'\'', '')

    return fname, response.content

if __name__ == "__main__":
    job_id = build_model()
    print('Job ID is', job_id)
    wait_for_job_completion(job_id)
    print('Job', job_id, 'is finished')
    bin_filename, bin_file = download_model()
    print('Output file is', len(bin_file), 'bytes')
    with open(bin_filename, 'wb') as f:
        f.write(bin_file)
    print('Written job output to', bin_filename)

When you now run python3 download-model.py you'll see something like:

Job ID is 1486148
Writing templates OK

Scheduling job in cluster...
Compiling EON model...
Job started
Compiling EON model OK
Removing clutter...
Removing clutter OK
Copying output...
Copying output OK

Job 1486148 is finished
Output file is 4824580 bytes
Written job output to myawesomeproject-v63.zip

Introduction

This page is part of the Lifecycle Management with Edge Impulse tutorial series. If you haven't read the introduction yet, we recommend you to do so here.

In this tutorial, we'll guide you through deploying updated impulses over-the-air (OTA) using the Particle Workbench. We'll build on Particle's firmware update workflow, incorporating Edge Impulse's API to check for updates and download the latest build.

We will modify the Edge Impulse Photon 2 example to incorporate the OTA functionality and update the device with the latest build. Based on the particle workbench OTA example.

Prerequisites

• Edge Impulse Account: If you haven't got one, sign up here.

• Trained Impulse: If you're new, follow one of our end-to-end tutorials

• Knowledge of the Particle Workbench

• Installation of required software as detailed in the Photon 2 example

Preparation

Begin by setting up your device for OTA updates following the particle documentation. https://docs.particle.io/getting-started/cloud/ota-updates/

Let's get started!

Key Features of Particle Workbench OTA Updates:

• Development Devices: Specific devices can be marked for internal testing to validate firmware updates before a broader rollout.

• Firmware Upload: Developers upload the compiled binary to the console, ensuring the firmware version is included.

• Firmware Locking: Specific devices can be locked to run certain firmware versions for testing purposes.

• Release Targets: Firmware can be released to all devices or specific groups to control the rollout process.

• Intelligent Firmware Releases: Ensures timely delivery of updates without disrupting device operation, taking into account the device’s status and activity.

Particle OTA

Particle OTA is a fully-integrated over-the-air software update system that is built into the Particle IoT PaaS and Device OS. It allows customers to safely and reliably push software updates to single devices or entire fleets of devices directly from Particle’s device management console and developer tools, with no implementation work necessary.

Particle OTA allows you to update your entire IoT device (both the Particle device and any other components) by delivering three kinds of updates:

• Application OTA allows users to update the firmware application they are running on the Particle device in order to introduce new features, fix bugs, and generally improve the software application over time.

• Device OS OTA allows users to update Device OS to the latest version alongside an application update so that Device OS can be kept up to date with improvements and bug fixes while properly testing against the user-generated software application.

• Asset OTA allows users to include bundled assets in an OTA software update that can be delivered to other processors and components in the electronics system so that the Particle device can be responsible not just for updating itself but also the system that surrounds the device.

On your Edge Impulse project

Export your impulse to the Particle Workbench library:

Add the library through the Particle Workbench via:

1. Extract the .zip library.

2. Particle: Import Project and select project.properties. Examples can then be found in: yourprojectname/examples/

We created an example repository which contains a small application for your particle device.

git clone https://github.com/edgeimpulse/example-standalone-inferencing-photon2

Creating the Intial Impulse Export to Particle Library and Deployment firmware

You will need to treat this as a new project and follow the instructions in the Particle Workbench to set up your environment for local development.

Projects with external libraries

If your project includes a library that has not been registered in the Particle libraries system, you should create a new folder named /lib/<libraryname>/src under /<project dir> and add the .h, .cpp & library.properties files for your library there. Read the Firmware Libraries guide for more details on how to develop libraries. Note that all contents of the /lib folder and subfolders will also be sent to the Cloud for compilation.

Compiling your project

When you're ready to compile your project, make sure you have the correct Particle device target selected and run particle compile <platform> in the CLI or click the Compile button in the Desktop IDE. The following files in your project folder will be sent to the compile service:

• Everything in the /src folder, including your .ino application file

• The project.properties file for your project

• Any libraries stored under lib/<libraryname>/src

You should now have a compiled binary file in your project folder named firmware.bin.

Next lets look at the OTA functionality and how we can incorporate it into our project.

Particle Workbench OTA Update Workflow and example

You will need to have a device deployed with an existing firmware containing an impulse before you can roll out a new version. Then you can use the OTA workflow to update the firmware on your device to a new version. While blocking all other devices from updating to the new version.

Step 1: Testing on Development Devices

Step 2: Locking Firmware for Testing

Lock Devices: Lock specific devices to the new firmware version and monitor their behavior. Unlock Devices: Unlock the devices once testing is complete and you are satisfied with the firmware's performance.

Step 3: Rolling Out the Firmware

Release Targets: Choose to release the firmware to specific groups or the entire fleet of devices. Intelligent Firmware Release: Opt for this to have the Particle Device Cloud intelligently manage the update rollout. Important Tips: Avoid Disruptions: Utilize Device OS APIs to control OTA availability, ensuring devices aren’t disrupted during critical operations. Managing OTA Updates: Use the console or REST API to force enable OTA updates if needed. Monitoring: Keep an eye on the devices' behaviors post-update to quickly identify and address any potential issues. With Particle's OTA capabilities, developers can ensure that their IoT devices are always running the latest, most secure, and most efficient version of their firmware, enhancing the reliability and functionality of their IoT ecosystems.

Creating a Webhook for the Edge Impulse ingestion API

This guide supplements the tutorial on OTA Model Updates with Edge Impulse on Particle Workbench, focusing on configuring a Particle webhook for sending data to the Edge Impulse ingestion API.

Steps for Webhook Configuration:

1. Access Particle Console:

   • Visit Particle Console.

   • Log in with your Particle account credentials.

2. Navigate to Integrations:

   • Click on the "Integrations" tab in the left-hand menu.

   • Select "Webhooks" from the available options.

3. Create a New Webhook:

   • Click "New Integration".

   • Choose "Webhook".

4. Webhook Configuration:

   • Name: Assign a descriptive name to your webhook.

   • Event Name: Specify the event name that triggers the webhook (e.g., "edge/ingest").

   • URL: Set this to the Edge Impulse ingestion API URL, typically something like https://ingestion.edgeimpulse.com/api/training/data.

   • Request Type: Choose "POST".

   • Request Format: Select "Custom".

5. Custom Request Body:

   • Input the JSON structure required by Edge Impulse. This will vary based on your project's data schema.

6. HTTP Headers:

   • Add necessary headers:

     • x-api-key: Your Edge Impulse API key.

     • Content-Type: "application/json".

     • x-file-name: Use a dynamic data field like {{PARTICLE_EVENT_NAME}}.

7. Advanced Settings:

   • Response Topic: Create a custom topic for webhook responses, e.g., {{PARTICLE_DEVICE_ID}}/hook-response/{{PARTICLE_EVENT_NAME}}.

   • Enforce SSL: Choose "Yes" for secure transmission.

8. Save the Webhook:

   • After entering all details, click "Save".

9. Test the Webhook:

   • Use example device firmware to trigger the webhook.

   • Observe the responses in the Particle Console.

10. Debugging:

• If errors occur, review the logs for detailed information.

• Ensure payload format aligns with Edge Impulse requirements.

• Verify the accuracy of your API key and other details.

Custom Template Example:

Copy and paste the following into the Custom Template section of the webhook:

{
    "name": "edgeimpulse.com",
    "event": "edge/ingest",
    "responseTopic": "",
    "disabled": true,
    "url": "http://ingestion.edgeimpulse.com/api/training/data",
    "requestType": "POST",
    "noDefaults": true,
    "rejectUnauthorized": false,
    "headers": {
        "x-api-key": "ei_1855db...",
        "x-file-name": "{{PARTICLE_EVENT_NAME}}",
        "x-label": "coffee"
    },
    "json": "{\n  \"payload\": {\n    \"device_name\": \"0a10a...\",\n    \"device_type\": \"photon2\",\n    \"interval_ms\": 20,\n    \"sensors\": [\n      {\n        \"name\": \"volt\",\n        \"units\": \"V\"\n      },\n      {\n        \"name\": \"curr\",\n        \"units\": \"A\"\n      }\n    ],\n    \"values\": [\n{{{PARTICLE_EVENT_VALUE}}}\n    ]\n  },\n  \"protected\": {\n    \"alg\": \"none\",\n    \"ver\": \"v1\"\n  },\n  \"signature\": \"00\"\n}"
}

Closing the Loop to send data to Edge Impulse for Lifecycle Management

Below we will do a basic example of sending data to Edge Impulse for Lifecycle Management. We could further extend this example to include more advanced checking, utilizing more of Particle OTA functionality, and add checks for model performance and versioning. The code gathers data from analog sensors, processes it, and sends it to Edge Impulse for Lifecycle Management. The webhook is triggered with the event name edge/ingest/sample.

#include "Particle.h"

// Sample data array
int sampleData[2][10] = {
    {100, 110, 120, 130, 140, 150, 160, 170, 180, 190}, // Sample voltage data
    {200, 210, 220, 230, 240, 250, 260, 270, 280, 290}  // Sample current data
};

void setup() {
    Serial.begin(9600);
    Particle.connect();
}

void loop() {
    char dataBuf[512];
    JSONBufferWriter writer(dataBuf, sizeof(dataBuf) - 1);

    writer.beginArray();
    for(int i = 0; i < 10; i++) {
        writer.beginArray();
        writer.value(sampleData[0][i]);
        writer.value(sampleData[1][i]);
        writer.endArray();
    }
    writer.endArray();

    Particle.publish("edge/ingest/sample", dataBuf, PRIVATE);
    delay(10000); // Send data every 10 seconds
}

This code continuously samples voltage and current, calculates RMS current, and then sends a JSON array of the sampled data to the Edge Impulse ingestion API. The publishData function uses Particle's Particle.publish method to send the data to the specified event. This triggers the webhook configured to send data to Edge Impulse. See the Particle energy monitor code from our Imagine 2023 demo for a full example for more information.

Conclusion

This tutorial provides a basic guide for implementing OTA updates on your Particle Workbench connected device (Photon 2). It can be extended to include more advanced checking, utilizing more of espressif OTA functionality, and extending the python server to check for model performance and versioning. Happy coding!

Once you successfully trained or imported a model, you can use Edge Impulse to download a C++ library that bundles both your signal processing and your machine learning model. Until recently, we could only run one impulse on MCUs.

In this tutorial, we will see how to run multiple impulses using the downloaded C++ libraries of two different projects.

We have put together a custom deployment block that will automate all the processes and provide a C++ library that can be compiled and run as a standalone.

In this page, we will explain the high level concepts of how to merge two impulses. Feel free to look at the code to gain a deeper understanding. Alternatively, when we first wrote this tutorial, we explained how to merge two impulses manually; we will kept this process in the Manual procedure section but due to recent changes in our C++ SDK, some files and functions may have been renamed.

Also see this video (starting min 13):

Prerequisites

Make sure you have at least two impulses fully trained.

As an example, we will build an intrusion detection system. We will use a first model to detect glass-breaking sounds, if we detected this sound, we will then classify an image to see if there is a person or not in the image. In this tutorial, we will use the following public projects:

• Glass breaking - audio classification

• Person detection - image classification

Multi-impulse deployment block

The deployment block can be found here. To add it to your organization, head to this page: Edge Impulse Studio -> Organizations -> Custom blocks -> Deployment blocks.

Please note that the script works with EON compiled projects only and anomaly detection blocks have not been tested.

Modifying the generated libraries and merging them into a single library

If you have a look at the generate.py script, it streamline the process of generating a C++ library from multiple impulses through several steps:

1. Library Download and Extraction:

• If the script detects that the necessary projects are not already present locally, it initiates the download of C++ libraries required for edge deployment. These libraries are fetched using API keys provided by the user.

• Libraries are downloaded and extracted into a temporary directory. If the user specifies a custom temporary directory, it's used; otherwise, a temporary directory is created.

1. Customization of Files:

For each project's library, the script performs several modifications:

• At the file name level:

  • It adds a project-specific suffix to certain patterns in compiled files within the tflite-model directory. This customization ensures that each project's files are unique.

  • Renamed files are then copied to a target directory, mainly the first project's directory.

• At the function name level:

  • It edits model_variables.h functions by adding the project-specific suffix to various patterns. This step ensures that model parameters are correctly associated with each project.

1. Merging the projects

• model_variables.h is merged into the first project's directory to consolidate model information.

• The script saves the intersection of lines between trained_model_ops_define.h files for different projects, ensuring consistency.

1. Copying Templates:

• The script copies template files from a templates directory to the target directory. The template available includes files with code structures and placeholders for customization. It's adapted from the example-standalone-inferencing example available on Github.

1. Generating Custom Code:

• The script retrieves impulse IDs from model_variables.h for each project. Impulses are a key part of edge machine learning models.

• Custom code is generated for each project, including functions to get signal data, define raw features, and run the classifier.

• This custom code is inserted into the main.cpp file of each project at specific locations.

1. Archiving for Deployment:

• Finally, the script archives the target directory, creating a zip file ready for deployment. This zip file contains all the customized files and code necessary for deploying machine learning models on edge devices.

Compiling and running the multi-impulse library

Now to test the library generated:

• Download and unzip your Edge Impulse C++ multi-impulse library into a directory

• Copy a test sample's raw features into the features[] array in source/main.cpp

• Enter make -j in this directory to compile the project. If you encounter any OOM memory error try make -j4 (replace 4 with the number of cores available)

• Enter ./build/app to run the application

• Compare the output predictions to the predictions of the test sample in the Edge Impulse Studio

Manual procedure

Download the impulses from your projects

Head to your projects' deployment pages and download the C++ libraries:

Make sure to select the same model versions (EON-Compiled enabled/disabled and int8/float32) for your projects.

Extract the two archive in a directory (multi-impulse for example).

Rename the tflite model files

Rename the tflite model files:

Go to the tflite-model directory in your extracted archives and rename the following files by post-fixing them with the name of the project:

• for EON compiled projects: tflite_model_compiled.cpp/tflite_model_compiled.h.

• for non-EON-compiled projects: tflite-trained.cpp/tflite-trained.h.

Original structure:

>  multi-impulse % tree -L 3
.
├── audio
│   ├── CMakeLists.txt
│   ├── README.txt
│   ├── edge-impulse-sdk
│   │   ├── CMSIS
│   │   ├── LICENSE
│   │   ├── LICENSE-apache-2.0.txt
│   │   ├── README.md
│   │   ├── classifier
│   │   ├── cmake
│   │   ├── dsp
│   │   ├── porting
│   │   ├── sources.txt
│   │   ├── tensorflow
│   │   └── third_party
│   ├── model-parameters
│   │   ├── model_metadata.h
│   │   └── model_variables.h
│   └── tflite-model
│       ├── trained_model_compiled.cpp
│       ├── trained_model_compiled.h
│       └── trained_model_ops_define.h
└── image
    ├── CMakeLists.txt
    ├── README.txt
    ├── edge-impulse-sdk
    │   ├── CMSIS
    │   ├── LICENSE
    │   ├── LICENSE-apache-2.0.txt
    │   ├── README.md
    │   ├── classifier
    │   ├── cmake
    │   ├── dsp
    │   ├── porting
    │   ├── sources.txt
    │   ├── tensorflow
    │   └── third_party
    ├── model-parameters
    │   ├── model_metadata.h
    │   └── model_variables.h
    └── tflite-model
        ├── trained_model_compiled.cpp
        ├── trained_model_compiled.h
        └── trained_model_ops_define.h

22 directories, 22 files

New structure after renaming the files:

>multi-impulse % tree -L 3
.
├── audio
│   ├── CMakeLists.txt
│   ├── README.txt
│   ├── edge-impulse-sdk
│   │   ├── CMSIS
│   │   ├── LICENSE
│   │   ├── LICENSE-apache-2.0.txt
│   │   ├── README.md
│   │   ├── classifier
│   │   ├── cmake
│   │   ├── dsp
│   │   ├── porting
│   │   ├── sources.txt
│   │   ├── tensorflow
│   │   └── third_party
│   ├── model-parameters
│   │   ├── model_metadata.h
│   │   └── model_variables.h
│   └── tflite-model
│       ├── trained_model_compiled_audio.cpp
│       ├── trained_model_compiled_audio.h
│       └── trained_model_ops_define.h
└── image
    ├── CMakeLists.txt
    ├── README.txt
    ├── edge-impulse-sdk
    │   ├── CMSIS
    │   ├── LICENSE
    │   ├── LICENSE-apache-2.0.txt
    │   ├── README.md
    │   ├── classifier
    │   ├── cmake
    │   ├── dsp
    │   ├── porting
    │   ├── sources.txt
    │   ├── tensorflow
    │   └── third_party
    ├── model-parameters
    │   ├── model_metadata.h
    │   └── model_variables.h
    └── tflite-model
        ├── trained_model_compiled_image.cpp
        ├── trained_model_compiled_image.h
        └── trained_model_ops_define.h

22 directories, 22 files

Rename the variables in the tflite-model directory

Rename the variables (EON model functions, such as trained_model_input etc or tflite model array names) by post-fixing them with the name of the project.

e.g: Change the trained_model_compiled_audio.h from:

#ifndef trained_model_GEN_H
#define trained_model_GEN_H

#include "edge-impulse-sdk/tensorflow/lite/c/common.h"

// Sets up the model with init and prepare steps.
TfLiteStatus trained_model_init( void*(*alloc_fnc)(size_t,size_t) );
// Returns the input tensor with the given index.
TfLiteStatus trained_model_input(int index, TfLiteTensor* tensor);
// Returns the output tensor with the given index.
TfLiteStatus trained_model_output(int index, TfLiteTensor* tensor);
// Runs inference for the model.
TfLiteStatus trained_model_invoke();
//Frees memory allocated
TfLiteStatus trained_model_reset( void (*free)(void* ptr) );


// Returns the number of input tensors.
inline size_t trained_model_inputs() {
  return 1;
}
// Returns the number of output tensors.
inline size_t trained_model_outputs() {
  return 1;
}

#endif

to:

#include "edge-impulse-sdk/tensorflow/lite/c/common.h"

// Sets up the model with init and prepare steps.
TfLiteStatus trained_model_audio_init( void*(*alloc_fnc)(size_t,size_t) );
// Returns the input tensor with the given index.
TfLiteStatus trained_model_audio_input(int index, TfLiteTensor* tensor);
// Returns the output tensor with the given index.
TfLiteStatus trained_model_audio_output(int index, TfLiteTensor* tensor);
// Runs inference for the model.
TfLiteStatus trained_model_audio_invoke();
//Frees memory allocated
TfLiteStatus trained_model_audio_reset( void (*free)(void* ptr) );


// Returns the number of input tensors.
inline size_t trained_model_audio_inputs() {
  return 1;
}
// Returns the number of output tensors.
inline size_t trained_model_audio_outputs() {
  return 1;
}

#endif

Tip: Use an IDE to use the "Find and replace feature.

Here is a list of the files that need to be modified (the names may change if not compiled with the EON compiler):

• tflite-model/trained_model_compiled_<project1|2>.h

• tflite-model/trained_model_compiled_<project1|2>.cpp

Rename the variables and structs in model-parameter/model_variables.h

Be careful here when using the "find and replace" from your IDE, NOT all variables looking like _model_ need to be replaced.

Example for the audio project:

#ifndef _EI_CLASSIFIER_MODEL_VARIABLES_H_
#define _EI_CLASSIFIER_MODEL_VARIABLES_H_

#include <stdint.h>
#include "model_metadata.h"

#include "tflite-model/trained_model_compiled_audio.h"
#include "edge-impulse-sdk/classifier/ei_model_types.h"
#include "edge-impulse-sdk/classifier/inferencing_engines/engines.h"

const char* ei_classifier_inferencing_categories_audio[] = { "Background", "Glass_Breaking" };

uint8_t ei_dsp_config_3_axes_audio[] = { 0 };
const uint32_t ei_dsp_config_3_axes_size_audio = 1;
ei_dsp_config_mfe_t ei_dsp_config_3_audio = {
    3, // uint32_t blockId
    3, // int implementationVersion
    1, // int length of axes
    0.02f, // float frame_length
    0.01f, // float frame_stride
    40, // int num_filters
    256, // int fft_length
    300, // int low_frequency
    0, // int high_frequency
    101, // int win_size
    -52 // int noise_floor_db
};

const size_t ei_dsp_blocks_size_audio = 1;
ei_model_dsp_t ei_dsp_blocks_audio[ei_dsp_blocks_size_audio] = {
    { // DSP block 3
        3960,
        &extract_mfe_features,
        (void*)&ei_dsp_config_3_audio,
        ei_dsp_config_3_axes_audio,
        ei_dsp_config_3_axes_size_audio
    }
};

const ei_config_tflite_eon_graph_t ei_config_tflite_graph_audio_0 = {
    .implementation_version = 1,
    .model_init = &trained_model_audio_init,
    .model_invoke = &trained_model_audio_invoke,
    .model_reset = &trained_model_audio_reset,
    .model_input = &trained_model_audio_input,
    .model_output = &trained_model_audio_output,
};

const ei_learning_block_config_tflite_graph_t ei_learning_block_config_audio_0 = {
    .implementation_version = 1,
    .block_id = 0,
    .object_detection = 0,
    .object_detection_last_layer = EI_CLASSIFIER_LAST_LAYER_UNKNOWN,
    .output_data_tensor = 0,
    .output_labels_tensor = 1,
    .output_score_tensor = 2,
    .graph_config = (void*)&ei_config_tflite_graph_audio_0
};

const size_t ei_learning_blocks_size_audio = 1;
const ei_learning_block_t ei_learning_blocks_audio[ei_learning_blocks_size_audio] = {
    {
        &run_nn_inference,
        (void*)&ei_learning_block_config_audio_0,
    },
};

const ei_model_performance_calibration_t ei_calibration_audio = {
    1, /* integer version number */
    false, /* has configured performance calibration */
    (int32_t)(EI_CLASSIFIER_RAW_SAMPLE_COUNT / ((EI_CLASSIFIER_FREQUENCY > 0) ? EI_CLASSIFIER_FREQUENCY : 1)) * 1000, /* Model window */
    0.8f, /* Default threshold */
    (int32_t)(EI_CLASSIFIER_RAW_SAMPLE_COUNT / ((EI_CLASSIFIER_FREQUENCY > 0) ? EI_CLASSIFIER_FREQUENCY : 1)) * 500, /* Half of model window */
    0   /* Don't use flags */
};


const ei_impulse_t impulse_233502_3 = {
    .project_id = 233502,
    .project_owner = "Edge Impulse Inc.",
    .project_name = "Glass breaking - audio classification",
    .deploy_version = 3,

    .nn_input_frame_size = 3960,
    .raw_sample_count = 16000,
    .raw_samples_per_frame = 1,
    .dsp_input_frame_size = 16000 * 1,
    .input_width = 0,
    .input_height = 0,
    .input_frames = 0,
    .interval_ms = 0.0625,
    .frequency = 16000,
    .dsp_blocks_size = ei_dsp_blocks_size_audio,
    .dsp_blocks = ei_dsp_blocks_audio,
    
    .object_detection = 0,
    .object_detection_count = 0,
    .object_detection_threshold = 0,
    .object_detection_last_layer = EI_CLASSIFIER_LAST_LAYER_UNKNOWN,
    .fomo_output_size = 0,
    
    .tflite_output_features_count = 2,
    .learning_blocks_size = ei_learning_blocks_size_audio,
    .learning_blocks = ei_learning_blocks_audio,

    .inferencing_engine = EI_CLASSIFIER_TFLITE,
    
    .quantized = 1,
    
    .compiled = 1,

    .sensor = EI_CLASSIFIER_SENSOR_MICROPHONE,
    .fusion_string = "audio",
    .slice_size = (16000/4),
    .slices_per_model_window = 4,

    .has_anomaly = 0,
    .label_count = 2,
    .calibration = ei_calibration_audio,
    .categories = ei_classifier_inferencing_categories_audio
};

const ei_impulse_t ei_default_impulse = impulse_233502_3;

#endif // _EI_CLASSIFIER_MODEL_METADATA_H_

Example for the image project:

#ifndef _EI_CLASSIFIER_MODEL_VARIABLES_H_
#define _EI_CLASSIFIER_MODEL_VARIABLES_H_

#include <stdint.h>
#include "model_metadata.h"

#include "tflite-model/trained_model_compiled_image.h"
#include "edge-impulse-sdk/classifier/ei_model_types.h"
#include "edge-impulse-sdk/classifier/inferencing_engines/engines.h"

const char* ei_classifier_inferencing_categories_image[] = { "person", "unknown" };

uint8_t ei_dsp_config_3_axes_image[] = { 0 };
const uint32_t ei_dsp_config_3_axes_size_image = 1;
ei_dsp_config_image_t ei_dsp_config_3_image = {
    3, // uint32_t blockId
    1, // int implementationVersion
    1, // int length of axes
    "RGB" // select channels
};

const size_t ei_dsp_blocks_size_image = 1;
ei_model_dsp_t ei_dsp_blocks_image[ei_dsp_blocks_size_image] = {
    { // DSP block 3
        27648,
        &extract_image_features,
        (void*)&ei_dsp_config_3_image,
        ei_dsp_config_3_axes_image,
        ei_dsp_config_3_axes_size_image
    }
};

const ei_config_tflite_eon_graph_t ei_config_tflite_graph_image_0 = {
    .implementation_version = 1,
    .model_init = &trained_model_image_init,
    .model_invoke = &trained_model_image_invoke,
    .model_reset = &trained_model_image_reset,
    .model_input = &trained_model_image_input,
    .model_output = &trained_model_image_output,
};

const ei_learning_block_config_tflite_graph_t ei_learning_block_config_image_0 = {
    .implementation_version = 1,
    .block_id = 0,
    .object_detection = 0,
    .object_detection_last_layer = EI_CLASSIFIER_LAST_LAYER_UNKNOWN,
    .output_data_tensor = 0,
    .output_labels_tensor = 1,
    .output_score_tensor = 2,
    .graph_config = (void*)&ei_config_tflite_graph_image_0
};

const size_t ei_learning_blocks_size_image = 1;
const ei_learning_block_t ei_learning_blocks_image[ei_learning_blocks_size_image] = {
    {
        &run_nn_inference,
        (void*)&ei_learning_block_config_image_0,
    },
};

const ei_model_performance_calibration_t ei_calibration_image = {
    1, /* integer version number */
    false, /* has configured performance calibration */
    (int32_t)(EI_CLASSIFIER_RAW_SAMPLE_COUNT / ((EI_CLASSIFIER_FREQUENCY > 0) ? EI_CLASSIFIER_FREQUENCY : 1)) * 1000, /* Model window */
    0.8f, /* Default threshold */
    (int32_t)(EI_CLASSIFIER_RAW_SAMPLE_COUNT / ((EI_CLASSIFIER_FREQUENCY > 0) ? EI_CLASSIFIER_FREQUENCY : 1)) * 500, /* Half of model window */
    0   /* Don't use flags */
};


const ei_impulse_t impulse_233515_5 = {
    .project_id = 233515,
    .project_owner = "Edge Impulse Inc.",
    .project_name = "Person vs unknown - image classification",
    .deploy_version = 5,

    .nn_input_frame_size = 27648,
    .raw_sample_count = 9216,
    .raw_samples_per_frame = 1,
    .dsp_input_frame_size = 9216 * 1,
    .input_width = 96,
    .input_height = 96,
    .input_frames = 1,
    .interval_ms = 1,
    .frequency = 0,
    .dsp_blocks_size = ei_dsp_blocks_size_image,
    .dsp_blocks = ei_dsp_blocks_image,
    
    .object_detection = 0,
    .object_detection_count = 0,
    .object_detection_threshold = 0,
    .object_detection_last_layer = EI_CLASSIFIER_LAST_LAYER_UNKNOWN,
    .fomo_output_size = 0,
    
    .tflite_output_features_count = 2,
    .learning_blocks_size = ei_learning_blocks_size_image,
    .learning_blocks = ei_learning_blocks_image,

    .inferencing_engine = EI_CLASSIFIER_TFLITE,
    
    .quantized = 1,
    
    .compiled = 1,

    .sensor = EI_CLASSIFIER_SENSOR_CAMERA,
    .fusion_string = "image",
    .slice_size = (9216/4),
    .slices_per_model_window = 4,

    .has_anomaly = 0,
    .label_count = 2,
    .calibration = ei_calibration_image,
    .categories = ei_classifier_inferencing_categories_image
};

const ei_impulse_t ei_default_impulse = impulse_233515_5;

#endif // _EI_CLASSIFIER_MODEL_METADATA_H_

Merge the files

Create a new directory (merged-impulse for example). Copy the content of one project into this new directory (audio for example). Copy the content of the tflite-model directory from the other project (image) inside the newly created merged-impulse/tflite-model.

The structure of this new directory should look like the following:

> merged-impulse % tree -L 2
.
├── CMakeLists.txt
├── README.txt
├── edge-impulse-sdk
│   ├── CMSIS
│   ├── LICENSE
│   ├── LICENSE-apache-2.0.txt
│   ├── README.md
│   ├── classifier
│   ├── cmake
│   ├── dsp
│   ├── porting
│   ├── sources.txt
│   ├── tensorflow
│   └── third_party
├── model-parameters
│   ├── model_metadata.h
│   └── model_variables.h
└── tflite-model
    ├── trained_model_compiled_audio.cpp
    ├── trained_model_compiled_audio.h
    ├── trained_model_compiled_image.cpp
    ├── trained_model_compiled_image.h
    ├── trained_model_ops_define_audio.h
    └── trained_model_ops_define_image.h

10 directories, 14 files

Merge the variables and structs in model_variables.h

Copy the necessary variables and structs from previously updated image/model_metadata.h file content to the merged-impulse/model_metadata.h.

To do so, include both of these lines in the #include section:

#include "tflite-model/trained_model_compiled_audio.h"
#include "tflite-model/trained_model_compiled_image.h"

The section that should be copied is from const char* ei_classifier_inferencing_categories... to the line before const ei_impulse_t ei_default_impulse = impulse_<ProjectID>_<version>.

Make sure to leave only one const ei_impulse_t ei_default_impulse = impulse_233502_3; this will define which of your impulse is the default one.

Subtract and merge the trained_model_ops_define.h or tflite_resolver.h

Make sure the macros EI_TFLITE_DISABLE_... are a COMBINATION of the ones present in two deployments.

For EON-compiled projects:

E.g. if #define EI_TFLITE_DISABLE_SOFTMAX_IN_U8 1 is present in one deployment and absent in the other, it should be ABSENT in the combined trained_model_ops_define.h.

For non-EON-Compiled projects:

E.g. if resolver.AddFullyConnected(); is present in one deployment and absent in the other, it should be PRESENT in the combined tflite-resolver.h. Remember to change the length of the resolver array if necessary.

In this example, here are the lines to deleted:

Prepare the c++ application

Clone this repository: https://github.com/edgeimpulse/example-standalone-inferencing-multi-impulse

git clone git@github.com:edgeimpulse/example-standalone-inferencing-multi-impulse.git

Copy the content of the merged-impulse directory to example-standalone-inferencing-multi-impulse (replace the files and directory sharing the same).

Rename the variables in source/main.cpp

Edit the source/main.cpp file and replace the callback function names, the features buffers.

Note: The run_classifier takes the impulse pointer as a first argument

Copy the raw features from the studio Live Classification page.

Compile and run

Enter make -j in this directory to compile the project Enter ./build/app to run the application Compare the output predictions to the predictions of the test sample in the Edge Impulse Studio.

> example-standalone-inferencing-multi-impulse % ./build/app     
run_classifier with audio impulse returned: 0
Timing: DSP 0 ms, inference 0 ms, anomaly 0 ms
Predictions:
  Background: 0.00000
  Glass_Breaking: 0.99609
run_classifier with image impulse returned: 0
Timing: DSP 0 ms, inference 10 ms, anomaly 0 ms
Predictions:
  person: 0.99609
  unknown: 0.00000

Enter rm -f build/app && make clean to clean the project.

Congrats, you can now run multiple Impulse!!

Limitations

• The custom ML accelerator deployments are unlikely to work (TDA4VM, DRPAI, MemoryX, Brainchip).

• The custom tflite kernels (ESP NN, Silabs MVP, Arc MLI) should work - but may require some additional work. I.e: for ESP32 you may need to statically allocate arena for the image model.

• In general, running multiple impulses on an MCU can be challenging due to limited processing power, memory, and other hardware constraints. Make sure to thoroughly evaluate the capabilities and limitations of your specific MCU and consider the resource requirements of the impulses before attempting to run them concurrently.

Troubleshooting

Segmentation fault

If you see the following segmentation fault, make sure to subtract and merge the trained_model_ops_define.h or tflite_resolver.h

./build/app
run_classifier with audio impulse returned: 0
Timing: DSP 0 ms, inference 0 ms, anomaly 0 ms
Predictions:
  Background: 0.00000
  Glass_Breaking: 0.99609
zsh: segmentation fault  ./build/app

The Edge Impulse object detection model (FOMO) is effective at classifying objects and very lightweight (can run on MCUs). It does not however have any object persistence between frames. One common use of computer vision is for object counting- in order to achieve this you will need to add in some extra logic when deploying.

This notebook takes you through how to count objects using the linux deployment block (and provides some pointers for how to achieve similar logic other firmware deployment options).

Relevant links:

• Raw python files for the linux deployment example: https://github.com/edgeimpulse/object-counting-demo

• An end-to-end demo for on-device deployment of object counting: https://github.com/edgeimpulse/conveyor-counting-data-synthesis-demo

1. Download the linux deployment .eim for your project

To run your model locally, you need to deploy to a linux target in your project. First you need to enable all linux targets. Head to the deployment screen and click "Linux Boards" then in the following pop-up select "show all Linux deployment options on this page":

Then download the linux/mac target which is relevant to your machine:

Finally, follow the instructions shown as a pop-up to make your .eim file executable (for example for MacOS):

• Open a terminal window and navigate to the folder where you downloaded this model.

• Mark the model as executable: chmod +x path-to-model.eim

• Remove the quarantine flag: xattr -d com.apple.quarantine ./path-to-model.eim

2. Object Detection

Dependencies

Ensure you have these libraries installed before starting:

! pip install edge_impulse_linux
! pip install numpy
! pip install opencv-python

2.1 Run Object Counting on a video file

(see next heading for running on a webcam)

This program will run object detection on an input video file and count the objects going upwards which pass a threshold (TOP_Y). The sensitivity can be tuned with the number of columns (NUM_COLS) and the DETECT_FACTOR which is the factor of width/height of the object used to determine object permanence between frames.

Ensure you have added the relevant paths to your model file and video file:

• modelfile = '/path/to/modelfile.eim'

• videofile = '/path/to/video.mp4'

import cv2
import os
import time
import sys, getopt
import numpy as np
from edge_impulse_linux.image import ImageImpulseRunner


modelfile = '/path/to/modelfile.eim'
videofile = '/path/to/video.mp4'


runner = None
# if you don't want to see a video preview, set this to False
show_camera = True
if (sys.platform == 'linux' and not os.environ.get('DISPLAY')):
    show_camera = False
print('MODEL: ' + modelfile)

with ImageImpulseRunner(modelfile) as runner:
    try:
        model_info = runner.init()
        print('Loaded runner for "' + model_info['project']['owner'] + ' / ' + model_info['project']['name'] + '"')
        labels = model_info['model_parameters']['labels']
        count = 0
        vidcap = cv2.VideoCapture(videofile)
        sec = 0
        start_time = time.time()

        def getFrame(sec):
            vidcap.set(cv2.CAP_PROP_POS_MSEC,sec*1000)
            hasFrames,image = vidcap.read()
            if hasFrames:
                return image
            else:
                print('Failed to load frame', videofile)
                exit(1)


        img = getFrame(sec)
        
    
        # Define the top of the image and the number of columns
        TOP_Y = 30
        NUM_COLS = 5
        COL_WIDTH = int(vidcap.get(3) / NUM_COLS)
        # Define the factor of the width/height which determines the threshold
        # for detection of the object's movement between frames:
        DETECT_FACTOR = 1.5

        # Initialize variables
        count = [0] * NUM_COLS
        countsum = 0
        previous_blobs = [[] for _ in range(NUM_COLS)]
        
        while img.size != 0:
            # imread returns images in BGR format, so we need to convert to RGB
            img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)

            # get_features_from_image also takes a crop direction arguments in case you don't have square images
            features, cropped = runner.get_features_from_image(img)
            img2 = cropped
            COL_WIDTH = int(np.shape(cropped)[0]/NUM_COLS)

            # the image will be resized and cropped, save a copy of the picture here
            # so you can see what's being passed into the classifier
            cv2.imwrite('debug.jpg', cv2.cvtColor(cropped, cv2.COLOR_RGB2BGR))

            res = runner.classify(features)
            # Initialize list of current blobs
            current_blobs = [[] for _ in range(NUM_COLS)]
            
            if "bounding_boxes" in res["result"].keys():
                print('Found %d bounding boxes (%d ms.)' % (len(res["result"]["bounding_boxes"]), res['timing']['dsp'] + res['timing']['classification']))
                for bb in res["result"]["bounding_boxes"]:
                    print('\t%s (%.2f): x=%d y=%d w=%d h=%d' % (bb['label'], bb['value'], bb['x'], bb['y'], bb['width'], bb['height']))
                    img2 = cv2.rectangle(cropped, (bb['x'], bb['y']), (bb['x'] + bb['width'], bb['y'] + bb['height']), (255, 0, 0), 1)

                        # Check which column the blob is in
                    col = int(bb['x'] / COL_WIDTH)

                    # Check if blob is within DETECT_FACTOR*h of a blob detected in the previous frame and treat as the same object
                    for blob in previous_blobs[col]:
                        if abs(bb['x'] - blob[0]) < DETECT_FACTOR * (bb['width'] + blob[2]) and abs(bb['y'] - blob[1]) < DETECT_FACTOR * (bb['height'] + blob[3]):
                        # Check this blob has "moved" across the Y threshold
                            if blob[1] >= TOP_Y and bb['y'] < TOP_Y:
                                # Increment count for this column if blob has left the top of the image
                                count[col] += 1
                                countsum += 1
                    # Add current blob to list
                    current_blobs[col].append((bb['x'], bb['y'], bb['width'], bb['height']))



            # Update previous blobs
            previous_blobs = current_blobs

            if (show_camera):
                im2 = cv2.resize(img2, dsize=(800,800))
                cv2.putText(im2, f'{countsum} items passed', (15,750), cv2.FONT_HERSHEY_COMPLEX, 1, (0,255,0), 2)
                cv2.imshow('edgeimpulse', cv2.cvtColor(im2, cv2.COLOR_RGB2BGR))
                print(f'{count}')
                if cv2.waitKey(1) == ord('q'):
                    break

            sec = time.time() - start_time
            sec = round(sec, 2)
            # print("Getting frame at: %.2f sec" % sec)
            img = getFrame(sec)
    finally:
        if (runner):
            print(f'{countsum} Items Left Conveyorbelt')
            runner.stop()

2.2 Run Object Counting on a webcam stream

This program will run object detection on a webcam port and count the objects going upwards which pass a threshold (TOP_Y). The sensitivity can be tuned with the number of columns (NUM_COLS) and the DETECT_FACTOR which is the factor of width/height of the object used to determine object permanence between frames.

Ensure you have added the relevant paths to your model file and video file:

• modelfile = '/path/to/modelfile.eim'

• [OPTIONAL] camera_port = '/camera_port'

import cv2
import os
import sys, getopt
import signal
import time
from edge_impulse_linux.image import ImageImpulseRunner

modelfile = '/path/to/modelfile.eim'
# If you have multiple webcams, replace None with the camera port you desire, get_webcams() can help find this
camera_port = None


runner = None
# if you don't want to see a camera preview, set this to False
show_camera = True
if (sys.platform == 'linux' and not os.environ.get('DISPLAY')):
    show_camera = False

def now():
    return round(time.time() * 1000)

def get_webcams():
    port_ids = []
    for port in range(5):
        print("Looking for a camera in port %s:" %port)
        camera = cv2.VideoCapture(port)
        if camera.isOpened():
            ret = camera.read()[0]
            if ret:
                backendName =camera.getBackendName()
                w = camera.get(3)
                h = camera.get(4)
                print("Camera %s (%s x %s) found in port %s " %(backendName,h,w, port))
                port_ids.append(port)
            camera.release()
    return port_ids

def sigint_handler(sig, frame):
    print('Interrupted')
    if (runner):
        runner.stop()
    sys.exit(0)

signal.signal(signal.SIGINT, sigint_handler)


print('MODEL: ' + modelfile)



with ImageImpulseRunner(modelfile) as runner:
    try:
        model_info = runner.init()
        print('Loaded runner for "' + model_info['project']['owner'] + ' / ' + model_info['project']['name'] + '"')
        labels = model_info['model_parameters']['labels']
        if camera_port:
            videoCaptureDeviceId = int(args[1])
        else:
            port_ids = get_webcams()
            if len(port_ids) == 0:
                raise Exception('Cannot find any webcams')
            if len(port_ids)> 1:
                raise Exception("Multiple cameras found. Add the camera port ID as a second argument to use to this script")
            videoCaptureDeviceId = int(port_ids[0])

        camera = cv2.VideoCapture(videoCaptureDeviceId)
        ret = camera.read()[0]
        if ret:
            backendName = camera.getBackendName()
            w = camera.get(3)
            h = camera.get(4)
            print("Camera %s (%s x %s) in port %s selected." %(backendName,h,w, videoCaptureDeviceId))
            camera.release()
        else:
            raise Exception("Couldn't initialize selected camera.")

        next_frame = 0 # limit to ~10 fps here
        
        # Define the top of the image and the number of columns
        TOP_Y = 100
        NUM_COLS = 5
        COL_WIDTH = int(w / NUM_COLS)
        # Define the factor of the width/height which determines the threshold
        # for detection of the object's movement between frames:
        DETECT_FACTOR = 1.5

        # Initialize variables
        count = [0] * NUM_COLS
        countsum = 0
        previous_blobs = [[] for _ in range(NUM_COLS)]

        

        for res, img in runner.classifier(videoCaptureDeviceId):
            # Initialize list of current blobs
            current_blobs = [[] for _ in range(NUM_COLS)]
            
            if (next_frame > now()):
                time.sleep((next_frame - now()) / 1000)

            if "bounding_boxes" in res["result"].keys():
                print('Found %d bounding boxes (%d ms.)' % (len(res["result"]["bounding_boxes"]), res['timing']['dsp'] + res['timing']['classification']))
                for bb in res["result"]["bounding_boxes"]:
                    print('\t%s (%.2f): x=%d y=%d w=%d h=%d' % (bb['label'], bb['value'], bb['x'], bb['y'], bb['width'], bb['height']))
                    img = cv2.rectangle(img, (bb['x'], bb['y']), (bb['x'] + bb['width'], bb['y'] + bb['height']), (255, 0, 0), 1)

                        # Check which column the blob is in
                    col = int(bb['x'] / COL_WIDTH)
                    # Check if blob is within DETECT_FACTOR*h of a blob detected in the previous frame and treat as the same object
                    for blob in previous_blobs[col]:
                        print(abs(bb['x'] - blob[0]) < DETECT_FACTOR * (bb['width'] + blob[2]))
                        print(abs(bb['y'] - blob[1]) < DETECT_FACTOR * (bb['height'] + blob[3]))
                        if abs(bb['x'] - blob[0]) < DETECT_FACTOR * (bb['width'] + blob[2]) and abs(bb['y'] - blob[1]) < DETECT_FACTOR * (bb['height'] + blob[3]):
                        # Check this blob has "moved" across the Y threshold
                            if blob[1] >= TOP_Y and bb['y'] < TOP_Y:
                                # Increment count for this column if blob has left the top of the image
                                count[col] += 1
                                countsum += 1
                    # Add current blob to list
                    current_blobs[col].append((bb['x'], bb['y'], bb['width'], bb['height']))
                
            # Update previous blobs
            previous_blobs = current_blobs

            if (show_camera):
                im2 = cv2.resize(img, dsize=(800,800))
                cv2.putText(im2, f'{countsum} items passed', (15,750), cv2.FONT_HERSHEY_COMPLEX, 1, (0,255,0), 2)
                cv2.imshow('edgeimpulse', cv2.cvtColor(im2, cv2.COLOR_RGB2BGR))
                print('Found %d bounding boxes (%d ms.)' % (len(res["result"]["bounding_boxes"]), res['timing']['dsp'] + res['timing']['classification']))

                if cv2.waitKey(1) == ord('q'):
                    break

            next_frame = now() + 100
    finally:
        if (runner):
            runner.stop()