Motion recognition - Syntiant
In this tutorial, you'll use machine learning to build a gesture recognition system that runs on the Syntiant TinyML board. This is a hard task to solve using rule-based programming, as people don't perform gestures in the exact same way every time. But machine learning can handle these variations with ease. You'll learn how to collect high-frequency data from an IMU, build a neural network classifier, and how to deploy your model back to a device. At the end of this tutorial, you'll have a firm understanding of applying machine learning on Syntiant TinyML board using Edge Impulse.
For this tutorial you'll need the:
Follow the steps to connect your development board to Edge Impulse.
If your device is connected under Devices in the studio you can proceed:
Device connected to Edge Impulse
With your device connected, we can collect some data. In the studio go to the Data acquisition tab. This is the place where all your raw data is stored, and - if your device is connected to the remote management API - where you can start sampling new data.
Under Record new data, select your Syntiant device, set the label to
circular, the sample length to
2000, the sensor to
Inertialand the frequency to
100 Hz. This indicates that you want to record data for 2 seconds, and label the recorded data as
circular. You can later edit these labels if needed.
Record new data screen
After you click Start sampling move your device in a circular motion. In about twelve seconds the device should complete sampling and upload the file back to Edge Impulse. You see a new line appear under 'Collected data' in the studio. When you click it you now see the raw data graphed out. As the accelerometer on the development board has three axes you'll notice three different lines, one for each axis.
Circular movements recorded from the IMU
Machine learning works best with lots of data, so a single sample won't cut it. Now is the time to start building your own dataset. For example, use the following two classes, and record around 3 minutes of data per class:
- Circular - circular movements
- Z_Openset - random movements that are not circular
With the training set in place you can design an impulse. An impulse takes the raw data, slices it up in smaller windows, uses signal processing blocks to extract features, and then uses a learning block to classify new data. Signal processing blocks always return the same values for the same input and are used to make raw data easier to process, while learning blocks learn from past experiences.
For this tutorial we'll use the 'IMU Syntiant' signal processing block. This block rescales raw data to 8 bits values to match the NDP chip input requirements. Then we'll use a 'Neural Network' learning block, that takes these generated features and learns to distinguish between our different classes (circular or not).
In the studio go to Create impulse, set the window size to
1800(you can click on the
1800 ms.text to enter an exact value), the window increase to
80, and add the 'IMU Syntiant' and 'Classification (Keras)' blocks. Then click Save impulse.
Impulse with processing and learning blocks
To configure your signal processing block, click Syntiant IMU in the menu on the left. This will show you the raw data on top of the screen (you can select other files via the drop down menu), and the processed features on the right.
Scale 16 bits to 8 bitsconverts your raw data to 8 bits and normalize it to the range [-1, 1]. The circular motion public project's dataset is already rescaled so you need to disable the option in this case.
Click Save parameters. This will send you to the 'Feature generation' screen.
Click Generate features to start the process.
Afterwards the 'Feature explorer' will load. This is a plot of all the extracted features against all the generated windows. You can use this graph to compare your complete data set. A good rule of thumb is that if you can visually separate the data on a number of axes, then the machine learning model will be able to do so as well.
Examining your full dataset in the feature explorer
With all data processed it's time to start training a neural network. Neural networks are algorithms, modeled loosely after the human brain, that can learn to recognize patterns that appear in their training data. The network that we're training here will take the processing block features as an input, and try to map this to one of the two classes — 'circular' or 'z_openset'.
Click on NN Classifier in the left hand menu. You'll see the following page:
Syntiant neural network configuration
With everything in place, click Start training. You'll see a lot of text flying past in the Training output panel, which you can ignore for now. Training will take a few minutes. When it's complete, you'll see the Last training performance panel appear at the bottom of the page:
Congratulations, you've trained a neural network with Edge Impulse and ready to deploy on the Syntiant TinyML Board! But what do all these numbers mean?
At the start of training, 20% of the training data is set aside for validation. This means that instead of being used to train the model, it is used to evaluate how the model is performing. The Last training performance panel displays the results of this validation, providing some vital information about your model and how well it is working. Bear in mind that your exact numbers may differ from the ones in this tutorial.
On the left hand side of the panel, Accuracy refers to the percentage of windows of audio that were correctly classified. The higher number the better, although an accuracy approaching 100% is unlikely, and is often a sign that your model has overfit the training data. You will find out whether this is true in the next stage, during model testing. For many applications, an accuracy above 85% can be considered very good.
The Confusion matrix is a table showing the balance of correctly versus incorrectly classified windows. To understand it, compare the values in each row. For example, in the above screenshot, 100% of the circular motion samples were classified correctly, and 99.6% for the openset samples.
From the statistics in the previous step we know that the model works against our training data, but how well would the network perform on new data? Click on Live classification in the menu to find out. Your device should (just like in step 2) show as online under 'Classify new data'. Set the 'Sample length' to
2000(5 seconds), click Start sampling and start doing movements. Afterward, you'll get a full report on what the network thought that you did.
Classification result. Showing the conclusions, the raw data and processed features in one overview
If the network performed great, fantastic! But what if it performed poorly? There could be a variety of reasons, but the most common ones are:
- 1.There is not enough data. Neural networks need to learn patterns in data sets, and the more data the better.
- 2.The data does not look like other data the network has seen before. This is common when someone uses the device in a way that you didn't add to the test set. You can add the current file to the test set by clicking
⋮, then selecting Move to training set. Make sure to update the label under 'Data acquisition' before training.
- 3.The model has not been trained enough. Up the number of epochs to
50and see if performance increases (the classified file is stored, and you can load it through 'Classify existing validation sample').
- 4.The model is overfitting and thus performs poorly on new data. Try reducing the learning rate or add more data.
- 5.The neural network architecture is not a great fit for your data. Play with the number of layers and neurons and see if performance improves.
As you see there is still a lot of trial and error when building neural networks, but we hope the visualizations help a lot. You can also run the network against the complete validation set through 'Model validation'. Think of the model validation page as a set of unit tests for your model!
With a working model in place, we can look at places where our current impulse performs poorly.
With the impulse designed, trained and verified you can deploy this model back to your device. This makes the model run without an internet connection, minimizes latency, and runs with minimum power consumption.
To export your model, click on Deployment in the menu. Then under 'Build firmware' select the Syntiant development board,
The final step before building the firmware is to configure the posterior handler parameters of the Syntiant chip.
Optimizing posterior parameters
Those parameters are used to tune the precision and recall of the neural network activations, to minimize False Rejection Rate and False Activation Rate. You can manually edit those parameters in JSON format or use the Find posterior parameters to search for the best values:
- Select the classes you want to detect (the z_openset class should be omitted except for testing purpose)
- Select a calibration method: either no calibration (fastest), or FAR optimized (FAR is optimized for an FRR target < 0.2).
Once optimized parameters have been found, you can click Build. This will build a Syntiant package that will run on your development board. After building is completed you'll get prompted to download a zipfile. Save this on your computer. A pop-up video will show how the download process works.
After unzipping the downloaded file, run the appropriate flashing script for your platform (Linux, Mac, or Win 10) to flash the Syntiant TinyML Board with the Syntiant Circular Motion model and associated firmware. You might see a Microsoft Defender screen pop up when the script is run on Windows 10. It's safe to proceed so select 'More info' and continue.
We can connect to the board's newly flashed firmware over serial. Open a terminal and run:
This will sample data from the sensor, run the signal processing code, and then classify the data:
[SER] Started inferencing, press CTRL+C to stop...
Interval: 10.0000 ms.
Frame size: 1080
Sample length: 11 ms.
No. of classes: 2
Starting inferencing, press 'b' to break
Victory! You've now built your first on-device machine learning model.
We can't wait to see what you'll build! 🚀