Sensor fusion is about combining data from various sensors to gain a more comprehensive understanding of your environment. In this tutorial, we will demonstrate sensor fusion by bringing together high-dimensional audio or image data with time-series sensor data. This combination allows you to extract deeper insights from your sensor data.
This is an advanced tutorial where you will need to parse your dataset to create multi-sensor data samples, train several Edge Impulse project in order to extract the embeddings from the tflite models, create custom DSP blocks and, finally, modify the C++ inferencing SDK.
If you are looking for a more beginner-level tutorial, please head to the Sensor Fusion tutorial.
Multi-impulse vs multi-model vs sensor fusion
Running multi-impulse refers to running two separate projects (different data, different DSP blocks and different models) on the same target. It will require modifying some files in the EI-generated SDKs.
Running multi-model refers to running two different models (same data, same DSP block but different tflite models) on the same target. See how to run a motion classifier model and an anomaly detection model on the same device in this tutorial.
Sensor fusion refers to the process of combining data from different types of sensors to give more information to the neural network. To extract meaningful information from this data, you can use the same DSP block, multiples DSP blocks, or use neural networks embeddings like we will see in this tutorial.
Also, see this video (starting min 13):
When you have data coming from multiple sources, such as a microphone capturing audio, a camera capturing images, and sensors collecting time-series data. Integrating these diverse data types can be tricky and conventional methods fall short.
With the standard workflow, if you have data streams from various sources, you might want to create separate DSP blocks for each data type. For instance, if you're dealing with audio data from microphones, image data from cameras, and time-series sensor data from accelerometers, you could create separate DSP blocks for each. For example:
A spectrogram-based DSP block for audio data
An image DSP block for image data
A spectral analysis block for time-series sensor data
This approach initially seems logical but comes with limitations:
When using separate DSP blocks, you're constrained in your choice of neural networks. The features extracted from each data type are fundamentally different. For example, a pixel in an image or an image's spectrogram and a data point from an accelerometer's time-series data have distinct characteristics. This incompatibility makes it challenging to use a convolutional neural network (CNN) that is typically effective for image data or spectrogram. As a result, fully connected networks may be your only option, which are not ideal for audio or image data.
To bypass the limitation stated above, you may consider using neural networks embeddings. In essence, embeddings are compact, meaningful representations of your data, learned by a neural network.
Embeddings are super powerful, we use them for various features of Edge Impulse, such as the Data Explorer, the Auto Labeler or in this advanced sensor fusion tutorial.
While training the neural network, the model try to find the mathematical formula that best maps the input to the output. This is done by tweaking each neuron (each neuron is a parameter in our formula). The interesting part is that each layer of the neural network will start acting like a feature extracting step but highly tuned for your specific data.
Finally, instead of having a classifier for last layer (usually a softmax layer), we cut the neural network somewhere at the end and we obtained the embeddings.
Thus, we can consider the embeddings as learnt features and we will pass these "features" to the final Impulse:
Here's how we approach advanced sensor fusion with Edge Impulse.
In this workflow, we will show how to perform sensor fusion using both audio data and accelerometer data to classify different stages of a grinding coffee machine (grind, idle, pump and extract). First, we are going to use a spectrogram DSP block and a NN classifier using two dense network. This first impulse will then be used to generate the embeddings and will be made available in a custom DSP block. Finally, we are going to train a fully connected layer using features coming from both the generated embeddings and a spectral feature DSP block.
We have develop two Edge Impulse public projects, one publicly available dataset and a Github repository containing the source code to help you follow the steps:
Dataset: Coffee Machine Stages
Edge Impulse project 1 (used to generate the embeddings): Audio Sensor Fusion - Step 1
Edge Impulse project 2 (final impulse): Audio Sensor Fusion - Step 2
Github repository containing the source code: Sensor fusion using NN Embeddings
Please note that with a few changes, you will be able to change the sensor type (audio to images) or the first pre-processing method (spectrogram to MFE/MFCC).
The first step is to have input data samples that contain both sensors. In Edge Impulse studio, you can easily visualize time-series data, like audio and accelerometer data.
Note: it is not trivial to group together images and time-series. Our core-engineering team is working on improving this workflow. In the meantime, as a workaround, you can encode your image as time-series with one axis per channel (red, green, blue) plus the sensor:
Train separate projects for high dimensional data (audio or image data). Each project contains both a DSP block and a trained neural network.
See Audio Sensor Fusion - Step 1
Clone this repository:
Download the generated Impulse to extract the embeddings, which encapsulate distilled knowledge about their respective data types.
Download Model: From the project dashboard, download the TensorFlow SavedModel (saved_model). Extract the save_model directory and place it under the /input repository.
Download Test Data: From the same dashboard, download the test or train data NPY file (input.npy). Place this numpy array file under the /input repository. This will allow us to generate a quantized version of the tflite embeddings. Ideally choose the test data if you have some data available.
Generate the embeddings:
This will cut off the last layer of the neural network and convert it to TensorFlow Lite (TFLite) format. You can follow the process outlined in the saved_model_to_embeddings.py script for this conversion for a better understanding.
To make sensor fusion work seamlessly, Edge Impulse enables you to create custom DSP blocks. These blocks combine the necessary spectrogram/image-processing and neural network components for each data type.
Custom DSP Block Configuration: In the DSP block, perform two key operations as specified in the dsp.py script:
Run the DSP step with fixed parameters.
Run the neural network.
Replace this following lines in dsp-blocks/features-from-audio-embeddings/dsp.py to match your DSP configuration:
If you want to use another DSP block than the spectrogram one, all the source code of the available DSP code can be found in this public repository: processing-blocks
Return Neural Network Embeddings: The DSP block should be configured to return the neural network embeddings, as opposed to the final classification result.
Implement get_tflite_implementation: Ensure that the get_tflite_implementation function returns the TFLite model. Note that the on-device implementation will not be correct initially when generating the C++ library, as only the neural network part is compiled. We will fix this in the final exported C++ Library.
Now publish your new custom DSP block.
Fill the necessary information and push your block:
During development, it might be easier to host the block locally so you can make changes, see Custom DSP block
Multiple DSP Blocks: Create a new impulse with three DSP blocks and a classifier block. The routing should be as follows:
Audio data routed through the custom block.
Sensor data routed through spectral analysis.
See Audio Sensor Fusion - Step 2
Training the Model: Train the model within the new impulse, using a fully-connected network.
Export as a C++ Library:
In the Edge Impulse platform, export your project as a C++ library.
Choose the model type that suits your target device (quantized vs. float32).
Make sure to select EON compiler option
Copy the exported C++ library to the example-cpp folder for easy access.
Add a Forward Declaration:
In the model-parameters/model_variables.h file of the exported C++ library, add a forward declaration for the custom DSP block you created.
For example:
And change &extract_tflite_eon_features into &custom_sensor_fusion_features in the ei_dsp_blocks object.
Implement the Custom DSP Block:
In the main.cpp file of the C++ library, implement the custom_sensor_fusion_features block. This block should:
Call into the Edge Impulse SDK to generate features.
Execute the rest of the DSP block, including neural network inference.
For example, see the main.cpp file in the Github repository
Copy a test sample's raw features into the features[] array in source/main.cpp
Enter make -j in this directory to compile the project. If you encounter any OOM memory error try make -j4 (replace 4 with the number of cores available)
Enter ./build/app to run the application
Compare the output predictions to the predictions of the test sample in the Edge Impulse Studio.
Note that if you are using the quantized version of the model, you may encounter a slight difference between the Studio Live Classification page and the above results, the float32 model however should give you the same results.
Congratulations on successfully completing this advanced tutorial. You have been through the complex process of integrating high-dimensional audio or image data with time-series sensor data, employing advanced techniques like custom DSP blocks, neural network embeddings, and modifications to the C++ inferencing SDK. Also, note that you can simplify this workflow using custom deployment blocks to generate the custom DSP block with the embeddings.
If you are interested in using it for an enterprise project, please sign up for our FREE Enterprise Trial and our solution engineers can work with you on the integration.
