Sensor fusion is about combining data from various sensors to gain a more comprehensive understanding of your environment. In this tutorial, we will demonstrate sensor fusion by bringing together high-dimensional audio or image data with time-series sensor data. This combination allows you to extract deeper insights from your sensor data.
This is an advanced tutorial where you will need to parse your dataset to create multi-sensor data samples, train several Edge Impulse project in order to extract the embeddings from the tflite models, create custom DSP blocks and, finally, modify the C++ inferencing SDK.
If you are looking for a more beginner-level tutorial, please head to the Sensor Fusion tutorial.
Multi-impulse vs multi-model vs sensor fusion
Running multi-impulse refers to running two separate projects (different data, different DSP blocks and different models) on the same target. It will require modifying some files in the EI-generated SDKs.
Running multi-model refers to running two different models (same data, same DSP block but different tflite models) on the same target. See how to run a motion classifier model and an anomaly detection model on the same device in this tutorial.
Sensor fusion refers to the process of combining data from different types of sensors to give more information to the neural network. To extract meaningful information from this data, you can use the same DSP block, multiples DSP blocks, or use neural networks embeddings like we will see in this tutorial.
Also, see this video (starting min 13):
When you have data coming from multiple sources, such as a microphone capturing audio, a camera capturing images, and sensors collecting time-series data. Integrating these diverse data types can be tricky and conventional methods fall short.
With the standard workflow, if you have data streams from various sources, you might want to create separate DSP blocks for each data type. For instance, if you're dealing with audio data from microphones, image data from cameras, and time-series sensor data from accelerometers, you could create separate DSP blocks for each. For example:
A spectrogram-based DSP block for audio data
An image DSP block for image data
A spectral analysis block for time-series sensor data
This approach initially seems logical but comes with limitations:
When using separate DSP blocks, you're constrained in your choice of neural networks. The features extracted from each data type are fundamentally different. For example, a pixel in an image or an image's spectrogram and a data point from an accelerometer's time-series data have distinct characteristics. This incompatibility makes it challenging to use a convolutional neural network (CNN) that is typically effective for image data or spectrogram. As a result, fully connected networks may be your only option, which are not ideal for audio or image data.
To bypass the limitation stated above, you may consider using neural networks embeddings. In essence, embeddings are compact, meaningful representations of your data, learned by a neural network.
Embeddings are super powerful, we use them for various features of Edge Impulse, such as the Data Explorer, the Auto Labeler or in this advanced sensor fusion tutorial.
While training the neural network, the model try to find the mathematical formula that best maps the input to the output. This is done by tweaking each neuron (each neuron is a parameter in our formula). The interesting part is that each layer of the neural network will start acting like a feature extracting step but highly tuned for your specific data.
Finally, instead of having a classifier for last layer (usually a softmax layer), we cut the neural network somewhere at the end and we obtained the embeddings.
Thus, we can consider the embeddings as learnt features and we will pass these "features" to the final Impulse:
Here's how we approach advanced sensor fusion with Edge Impulse.
In this workflow, we will show how to perform sensor fusion using both audio data and accelerometer data to classify different stages of a grinding coffee machine (grind, idle, pump and extract). First, we are going to use a spectrogram DSP block and a NN classifier using two dense network. This first impulse will then be used to generate the embeddings and will be made available in a custom DSP block. Finally, we are going to train a fully connected layer using features coming from both the generated embeddings and a spectral feature DSP block.
We have develop two Edge Impulse public projects, one publicly available dataset and a Github repository containing the source code to help you follow the steps:
Dataset: Coffee Machine Stages
Edge Impulse project 1 (used to generate the embeddings): Audio Sensor Fusion - Step 1
Edge Impulse project 2 (final impulse): Audio Sensor Fusion - Step 2
Github repository containing the source code: Sensor fusion using NN Embeddings
Please note that with a few changes, you will be able to change the sensor type (audio to images) or the first pre-processing method (spectrogram to MFE/MFCC).
The first step is to have input data samples that contain both sensors. In Edge Impulse studio, you can easily visualize time-series data, like audio and accelerometer data.
Note: it is not trivial to group together images and time-series. Our core-engineering team is working on improving this workflow. In the meantime, as a workaround, you can encode your image as time-series with one axis per channel (red, green, blue) plus the sensor:
Train separate projects for high dimensional data (audio or image data). Each project contains both a DSP block and a trained neural network.
See Audio Sensor Fusion - Step 1
Clone this repository:
Download the generated Impulse to extract the embeddings, which encapsulate distilled knowledge about their respective data types.
Download Model: From the project dashboard, download the TensorFlow SavedModel (saved_model). Extract the save_model directory and place it under the /input repository.
Download Test Data: From the same dashboard, download the test or train data NPY file (input.npy). Place this numpy array file under the /input repository. This will allow us to generate a quantized version of the tflite embeddings. Ideally choose the test data if you have some data available.
Generate the embeddings:
This will cut off the last layer of the neural network and convert it to LiteRT (previously Tensorflow Lite) (TFLite) format. You can follow the process outlined in the saved_model_to_embeddings.py script for this conversion for a better understanding.
To make sensor fusion work seamlessly, Edge Impulse enables you to create custom DSP blocks. These blocks combine the necessary spectrogram/image-processing and neural network components for each data type.
Custom DSP Block Configuration: In the DSP block, perform two key operations as specified in the dsp.py script:
Run the DSP step with fixed parameters.
Run the neural network.
Replace this following lines in dsp-blocks/features-from-audio-embeddings/dsp.py to match your DSP configuration:
If you want to use another DSP block than the spectrogram one, all the source code of the available DSP code can be found in this public repository: processing-blocks
Return Neural Network Embeddings: The DSP block should be configured to return the neural network embeddings, as opposed to the final classification result.
Implement get_tflite_implementation: Ensure that the get_tflite_implementation function returns the TFLite model. Note that the on-device implementation will not be correct initially when generating the C++ library, as only the neural network part is compiled. We will fix this in the final exported C++ Library.
Now publish your new custom DSP block.
Fill the necessary information and push your block:
During development, it might be easier to host the block locally so you can make changes, see Custom processing blocks
Multiple DSP Blocks: Create a new impulse with three DSP blocks and a classifier block. The routing should be as follows:
Audio data routed through the custom block.
Sensor data routed through spectral analysis.
See Audio Sensor Fusion - Step 2
Training the Model: Train the model within the new impulse, using a fully-connected network.
Export as a C++ Library:
In the Edge Impulse platform, export your project as a C++ library.
Choose the model type that suits your target device (quantized vs. float32).
Make sure to select EON compiler option
Copy the exported C++ library to the example-cpp folder for easy access.
Add a Forward Declaration:
In the model-parameters/model_variables.h file of the exported C++ library, add a forward declaration for the custom DSP block you created.
For example:
And change &extract_tflite_eon_features into &custom_sensor_fusion_features in the ei_dsp_blocks object.
Implement the Custom DSP Block:
In the main.cpp file of the C++ library, implement the custom_sensor_fusion_features block. This block should:
Call into the Edge Impulse SDK to generate features.
Execute the rest of the DSP block, including neural network inference.
For example, see the main.cpp file in the Github repository
Copy a test sample's raw features into the features[] array in source/main.cpp
Enter make -j in this directory to compile the project. If you encounter any OOM memory error try make -j4 (replace 4 with the number of cores available)
Enter ./build/app to run the application
Compare the output predictions to the predictions of the test sample in the Edge Impulse Studio.
Note that if you are using the quantized version of the model, you may encounter a slight difference between the Studio Live Classification page and the above results, the float32 model however should give you the same results.
Congratulations on successfully completing this advanced tutorial. You have been through the complex process of integrating high-dimensional audio or image data with time-series sensor data, employing advanced techniques like custom DSP blocks, neural network embeddings, and modifications to the C++ inferencing SDK. Also, note that you can simplify this workflow using custom deployment blocks to generate the custom DSP block with the embeddings.
If you are interested in using it for an enterprise project, please sign up for our FREE Enterprise Trial and our solution engineers can work with you on the integration.
Extracting meaningful features from your data is crucial to building small and reliable machine learning models, and in Edge Impulse this is done through processing blocks. We ship a number of processing blocks for common sensor data (such as vibration and audio), but they might not be suitable for all applications. Perhaps you have a very specific sensor, want to apply custom filters, or are implementing the latest research in digital signal processing. In this tutorial you'll learn how to support these use cases by adding custom processing blocks to the studio.
There is also a complete video covering how to implement your custom DSP block:
Make sure you follow the Continuous motion recognition tutorial, and have a trained impulse.
Development flow
This tutorial shows you the development flow of building custom processing blocks, and requires you to run the processing block on your own machine or server. Enterprise customers can share processing blocks within their organization, and run these on our infrastructure. See custom processing blocks for more details.
Processing blocks take data and configuration parameters in, and return features and visualizations like graphs or images. To communicate to custom processing blocks, Edge Impulse studio will make HTTP calls to the block, and then use the response both in the UI, while generating features, or when training a machine learning model. Thus, to load a custom processing block we'll need to run a small server that responds to these HTTP calls. You can write this in any language, but we have created an example in Python. To load this example, open a terminal and run:
This creates a copy of the example project locally. Then, you can run the example either through Docker or locally via:
Docker
Locally
Then go to http://localhost:4446 and you should be shown some information about the block.
As this block is running locally the studio cannot reach the block. To resolve this we can use ngrok which can make a local port accessible from a public URL. After you've finished development you can move the processing block to a server with a publicly accessible address (or run it on our infrastructure through your enterprise account). To set up a tunnel:
Sign up for ngrok.
Install the ngrok binary for your platform.
Get a URL to access the processing block from the outside world via:
This yields a public URL for your block under Forwarding. Note down the address that includes https://.
Now that the custom processing block was created, and you've made it accessible to the outside world, you can add this block to Edge Impulse. In a project, go to Create Impulse, click Add a processing block, choose Add custom block (in the bottom left corner of the modal), and paste in the public URL of the block:
After you click Add block the block will show like any other processing block.
Add a learning bloc, then click Save impulse to store the impulse.
Processing blocks have configuration options which are rendered on the block parameter page. These could be filter configurations, scaling options, or control which visualizations are loaded. These options are defined in the parameters.json file. Let's add an option to smooth raw data. Open example-custom-processing-block-python/parameters.json and add a new section under parameters:
Then, open example-custom-processing-block-python/dsp.py and replace its contents with:
Restart the Python script, and then click Custom block in the studio (in the navigation bar). You now have a new option 'Smooth'. Every time an option changes we'll re-run the block, but as we have not written any code to respond to these changes nothing will happen.
For the full documentation on customizing parameters, and a list of all configuration options; see parameters.json.
To show the user what is happening we can also draw visuals in the processing block. Right now we support graphs (linear and logarithmic) and arbitrary images. By showing a graph of the smoothed sample we can quickly identify what effect the smooth option has on the raw signal. Open dsp.py and replace the content with the following script. It contains a very basic smoothing algorithm and draws a graph:
Restart the script, and click the Smooth toggle to observe the difference. Congratulations! You have just created your first custom processing block.
If you extract set features from the signal, like the mean, that you that return, you can also label these features. These labels will be used in the feature explorer. To do so, add a labels array that contains strings that map back to the features you return (labels and features should have the same length).
In the previous step we drew a linear graph, but you can also draw logarithmic graphs or even full images. This is done through the type parameter:
This draws a graph with a logarithmic scale:
To show an image you should return the base64 encoded image and its MIME type. Here's how you draw a small PNG image:
If you output high-dimensional data (like a spectrogram or an image) you can enable dimensionality reduction for the feature explorer. This will run UMAP over the data to compress the features into three dimensions. To do so, set:
On the info object in parameters.json.
For all options that you can return in a graph, see the Run DSP return types in the API documentation.
Your custom block behaves exactly the same as any of the built-in blocks. You can process all your data, train neural networks or anomaly blocks, and validate that your model works.
However, we cannot automatically generate optimized native code for the block, like we do for built-in processing blocks, but we try to help you write this code as much as possible.
Export as a C++ Library:
In the Edge Impulse platform, export your project as a C++ library.
Choose the model type that suits your target device (quantized vs. float32).
Forward Declaration:
You don't need to add this part, it is automatically generated!
In the model-parameters/model_variables.h file of the exported C++ library, you can see a forward declaration for the custom DSP block you created.
For example:
The name of that function comes from the cppType field in your custom DSP parameter.json. It takes your {cppType} and generates the following extract_{cppType}_features function.
Implement the Custom DSP Block:
In the main.cpp file of the C++ library, implement the extract_my_preprocessing_features block. This block should:
Call into the Edge Impulse SDK to generate features.
Execute the rest of the DSP block, including neural network inference.
For examples, have a look at our official DSP blocks implementations in our Inferencing C++ SDK
Also, please have a look at the video on the top of this page (around minute 25) where Jan explains how to implement your custom DSP block with your C++ library.
Compile and Run the App
Copy a test sample's raw features into the features[] array in source/main.cpp
Enter make -j in this directory to compile the project. If you encounter any OOM memory error try make -j4 (replace 4 with the number of cores available)
Enter ./build/app to run the application
Compare the output predictions to the predictions of the test sample in the Edge Impulse Studio.
Blog post: Utilize Custom Processing Blocks in Your Image ML Pipelines
With good feature extraction you can make your machine learning models smaller and more reliable, which are both very important when you want to deploy your model on embedded devices. With custom processing blocks you can now develop new feature extraction pipelines straight from Edge Impulse. Whether you're following the latest research, want to implement proprietary algorithms, or are just exploring data.
For inspiration we have published all our own blocks here: edgeimpulse/processing-blocks. If you've made an interesting block that you think is valuable for the community, please let us know on the forums or by opening a pull request. We'd be happy to help write efficient native code for the block, and then publish it as a standard block!
This is the format for the parameters.json file:
\
